Part 4 of “cloud-init Under the Hood”: the cache, the markers, and the files that make reboots safe.
You probably think cloud-init keeps its state in memory and starts fresh on every boot.
It doesn’t. It keeps a small, very organized notebook on disk. Every time your VM boots, cloud-init opens that notebook first, reads what it did last time, and decides what’s left to do.
In Part 3, we saw cloud-init write network files and hand the interface to NetworkManager. But it also writes something for itself. It lives under /var/lib/cloud, and almost nobody looks inside.
By the end of this article, you’ll know what each file in that directory means, which stage created it, and how cloud-init uses it to tell a reboot from a brand-new machine.
Table of Contents
The notebook on disk
The Roman orator Cicero called memory “the treasury and guardian of all things.” For cloud-init, that treasury is one directory.
Here’s what /var/lib/cloud/instance/ looked like on our VM after its first boot:
-rw-r--r-- 63 11:24 datasource
-rw------- 320 11:24 network-config.json
-r-------- 30660 11:24 obj.pkl
-rw------- 5440 11:24 user-data.txt
-rw------- 16515 11:24 user-data.txt.i
-rw------- 0 11:24 vendor-data.txt
-rw------- 16713 11:24 cloud-config.txt
drwxr-xr-x 11:24 handlers/
drwxr-xr-x 11:24 scripts/
drwxr-xr-x 11:29 sem/
-rw-r--r-- 57 11:29 boot-finishedLook at the timestamps. Almost everything appeared at 11:24, during the first two stages. Only sem/ and boot-finished changed at 11:29, when the final stage ended. The files tell the story of the boot in order.
One more detail: instance is a symlink. It points to /var/lib/cloud/instances/<instance-id>/. Each instance ID gets its own folder, which matters a lot later.
What came from the cloud
These files are cloud-init’s copy of what it received:
See also: Mastering the Linux Command Line — Your Complete Linux Training Course
datasourcenames the source that answered, for exampleDataSourceOpenStackLocal.user-data.txtis the raw user data, exactly as delivered.user-data.txt.iis the processed version after cloud-init unpacks the MIME parts. It’s three times larger than the raw file. Our Terraform module likely compresses user data before sending it, which would explain the gap.vendor-data.txtholds anything the cloud provider injects. Ours was empty.network-config.jsonis the network description cloud-init used to render the files we met in Part 3.
Notice the permissions. Most of these are readable only by root. That’s deliberate. User data often contains things you’d rather not share.
What cloud-init built from it
Two files are cloud-init’s own work:
cloud-config.txt is the merged, final configuration. cloud-init combines system defaults, vendor data, and your user data into this one file. If you ever wonder what cloud-init actually saw, read this instead of your Terraform template.
scripts/runcmd is your runcmd list, written out as a real shell script. As we saw in Part 2, the runcmd module only writes this file. A different module runs it later.
The two files that make it smart
Now for the part that makes reboots safe.
obj.pkl is a serialized snapshot of the entire datasource, including all the metadata it fetched. On a normal reboot, cloud-init loads this file instead of calling the metadata service again. That’s why later boots are fast and work even if the metadata service is briefly down.
sem/ is a folder of markers, one per module. When a once-per-instance module finishes, cloud-init drops a file there, such as config_scripts_user. On the next boot, it sees the marker and skips the module.
Think of sem/ as a checklist taped to the wall. The work only happens once. After that, cloud-init just reads the ticks.
Reboot or new machine?
Here’s the clever part. Before trusting its notebook, cloud-init checks it’s still on the same machine.
At the very start of init-local, it loads obj.pkl and compares the cached instance ID with the hardware UUID in /sys/class/dmi/id/product_uuid. This check is local and needs no network.
On a plain reboot of our VM, they matched:
init-local/check-cache: attempting to read from cache [check]
restored from checked cache: DataSourceOpenStackLocalOn the first boot of our production VM, built from a golden image, they didn’t:
init-local/check-cache: SUCCESS: cache invalid in datasource
previous iid found to be <image-build-instance>
init will now be targeting instance id: <new-instance>. new=TrueThe image still carried the build machine’s notebook. cloud-init noticed the UUID was different, set the old notebook aside, and opened a new folder under instances/. A new folder means an empty sem/, so every one-time module ran again.
A few related files live in /var/lib/cloud/data/:
instance-idstores the current instance ID.previous-instance-idstores the one before it.set-hostnamerecords the hostname cloud-init last applied.
The last one explains a log line we saw in Part 1: No hostname changes. Skipping set_hostname. cloud-init compared the target name with this file and found nothing to do.
The finish line
boot-finished is written last, by the final_message module. It records when cloud-init finished and how long the system had been up.
Many automation tools simply wait for this file to appear. When it exists, initialization is done.
Try it yourself
You don’t need to memorize every file. These commands cover most of what you’ll want:
ls -l /var/lib/cloud/instance/
ls /var/lib/cloud/instance/sem/
cat /var/lib/cloud/data/instance-id
sudo cat /var/lib/cloud/instance/cloud-config.txtAnd if you ever need cloud-init to forget everything, sudo cloud-init clean --logs --reboot wipes this directory. The next boot then looks brand new.
Coming up in Part 5
Not everything in that notebook was helpful. Our image carried old logs from its build, our first boot lost almost a minute to a network probe, and our user data held something it really shouldn’t have.
In Part 5, we’ll walk through those traps one by one.
What’s the strangest thing you’ve found in /var/lib/cloud? Tell me in the comments.
