cloud-init Troubleshooting: 5 Silent Problems Hiding in Your Cloud VMs

Part 5 of “cloud-init Under the Hood”: ghost logs, lost seconds, and a password hiding in plain sight.

You probably think that once cloud-init reports status: done, your VM booted cleanly.

Ours did too. Every module ran, zero failures. Yet when we read the logs line by line, we found five quiet problems. None of them broke the boot. All of them could have cost us time, money, or worse.

In Part 4, we opened /var/lib/cloud and saw how cloud-init remembers its work. This time, we look at what it remembered that it shouldn’t have, and what it did that nobody noticed.

By the end, you’ll have a short list of things to check on your own images before they bite you.

---

Pitfall 1: Ghost logs from the image build

When I opened /var/log/cloud-init.log on our new VM, the first lines were dated ten months earlier. They mentioned a hostname I’d never seen.

cloud-init analyze show explained it. It listed four boot records. Only the last one belonged to our VM. The other three came from the machines that built the golden image.

The image pipeline never ran cloud-init clean --logs before the snapshot was taken. So the build history shipped inside every VM made from that image.

It’s harmless until you’re troubleshooting at 2 a.m. and reading the wrong boot.

How to avoid it:

See also: Mastering the Linux Command Line — Your Complete Linux Training Course

• Add cloud-init clean --logs as the last step of your image build. Some teams also add --machine-id.
• Until then, find the real start of your boot with:

sudo grep -n "Cloud-init v." /var/log/cloud-init.log

The last match is yours.

---

Pitfall 2: 49 seconds lost to a network probe

Our first stage, init-local, took almost 50 seconds. On other platforms, this stage usually finishes in two or three.

The log pointed straight at it:

performance.py: Resolving URL took 49.157 seconds

cloud-init probes two metadata addresses at once: an IPv6 link-local address and the familiar 169.254.169.254. Our hardened image had IPv6 disabled, and probing that IPv6 address stalled. Every new VM paid that toll once, on its very first boot.

How to avoid it: Tell cloud-init to use only the IPv4 address. A drop-in file in the image like this should do it:

# /etc/cloud/cloud.cfg.d/99-metadata-ipv4.cfg
datasource:
  OpenStack:
    metadata_urls: ['http://169.254.169.254']

To be honest: we haven’t rolled this out yet. Test it on one image before trusting it fleet-wide.

---

Pitfall 3: A password in the user data

This is the one that matters most.

Our bootstrap script joined each VM to a directory domain. To do that, it needed a service account password. And that password sat in the user data, in plain text.

You might think it was protected. The user data was gzip-compressed and base64-encoded by Terraform. On the VM, cloud-init’s copy in /var/lib/cloud was readable only by root.

None of that helped.

• Compression isn’t encryption. zcat reverses gzip in a second.
• The metadata service has no login. On OpenStack, any process on the VM can run curl http://169.254.169.254/openstack/latest/user_data. No root needed.
• It lives elsewhere too. The same value sits in Terraform state and in any log export someone saved to a laptop.

Benjamin Franklin said “An ounce of prevention is worth a pound of cure.” Rotating a leaked domain password across a fleet is a very heavy pound.

How to avoid it:

• Keep secrets out of user data entirely. Fetch them at boot from a secrets manager using a short-lived token.
• If a secret has already shipped this way, rotate it.

---

Pitfall 4: One wrong key, and the status turns “degraded”

After we fixed metadata access on another VM, cloud-init status --long still looked worrying:

extended_status: degraded done
WARNING:
  - cloud-config failed schema validation!

Every module had run successfully. The cause was a single line in our users block:

create_home: true

That key doesn’t exist in cloud-init’s schema. The valid one is no_create_home, and home directories are created by default anyway. cloud-init ignored the line, but flagged the whole run as degraded.

That matters if your monitoring alerts on “degraded”. You’ll chase a problem that isn’t there.

How to avoid it: Validate your config before it ever reaches a VM:

cloud-init schema --config-file my-user-data.yaml

On a running VM, sudo cloud-init schema --system tells you exactly which key is wrong.

---

Pitfall 5: Logs that vanish after a reboot

Our runcmd script began with these lines:

exec 1>/tmp/init_log
exec 2>&1

Every command’s output went to /tmp/init_log. Each step also wrote a small status file into tmp.

There were two problems:

• Nothing reached cloud-init-output.log, the first place most people look.
• tmp may be cleared on reboot. The one record of what your bootstrap actually did can disappear exactly when you need it.

How to avoid it:

• Write bootstrap logs under log, for example /var/log/bootstrap.log.
• Add a timestamp to each step, so you can see where the minutes go.

---

A quick checklist

Before your next image or template change, run through these:

1. Does the image build end with cloud-init clean --logs?
2. Does first boot spend more than a few seconds in init-local?
3. Is there any secret in your user data?
4. Does cloud-init schema pass on your config?
5. Will your bootstrap logs survive a reboot?

None of these showed up as an error. That’s exactly why they’re worth checking.

---

Coming up in Part 6

We’ve covered how cloud-init boots, hands off the network, remembers its work, and where it can trip you up. In the final part, I’ll pull it all together into a one-page debugging playbook, with the commands I now run on every new VM.

Which of these five have you run into? Or did you find a sixth? Tell me in the comments.