Why your bootstrap script doesn’t run twice, and the small set of things that do.
You probably think cloud-init is a first-boot tool. It runs once, sets up your server, and then disappears forever.
That’s half true. And the other half is where the surprises live.
I found this out while reading the boot history of one of our database VMs. cloud-init analyze show listed all separate boot records. cloud-init had started every single time. Yet on most of those boots, it finished in well under a second and changed almost nothing.
By the end of this article, you’ll know exactly what cloud-init repeats on every boot, what it never repeats, and how it tells the difference. You’ll also know how to make it run again on purpose, which comes in handy more often than you’d think.
Table of Contents
The short answer
cloud-init’s four systemd services start on every single boot:
cloud-init-local.servicecloud-init.servicecloud-config.servicecloud-final.service
So in one sense, yes, cloud-init runs every time.
But each module inside those services has a frequency. That frequency decides whether the module actually does anything on this boot. Think of cloud-init as a night-shift guard with a checklist. He walks the whole building every night. But most items on the list say “only do this once,” and he just ticks them off as done.
Here is what the same server looked like on two different boots.
On the first boot of a new instance:
See also: Mastering the Linux Command Line — Your Complete Linux Training Course
|`->config-users_groups ran successfully @03.66100s +00.09800s
|`->config-ssh ran successfully @03.75900s +00.41800s
Total Time: 3.98500 secondsOn a plain reboot of the same instance:
|`->config-users_groups previously ran @00.76800s +00.00000s
|`->config-ssh previously ran @00.76800s +00.00000s
Total Time: 0.67300 secondsSame services. Same modules. Almost no work on the second boot.
The three frequencies
Every cloud-init module declares one of three frequencies:
once-per-instance runs once for each new instance. This is the default for most of the important work: creating users, setting the hostname, generating SSH host keys, and running your runcmd script. Clone the VM into a new instance and these run again.
always runs on every boot. A small group of modules live here, and you can see why:
growpartandresizefsgrow the root partition, in case someone enlarged the disk while the machine was off.update_hostnamekeeps/etc/hostnamein sync.scripts_per_bootruns anything you placed in/var/lib/cloud/scripts/per-boot/.final_messagelogs that boot is finished.
once runs once for the whole life of the machine, even across new instance IDs.
How cloud-init remembers
You might wonder how cloud-init knows it already did something. The answer sits in two places on disk.
The first is a semaphore directory: /var/lib/cloud/instances/<instance-id>/sem/. When a once-per-instance module finishes, cloud-init drops a marker file there, such as config_scripts_user. On the next boot, it sees the marker and skips the module.
The second is a cache check at the start of every boot. cloud-init loads its cached datasource from obj.pkl. On OpenStack, it then compares the cached instance ID with the hardware UUID in /sys/class/dmi/id/product_uuid. This check is local and needs no network.
If the two match, the log says:
restored from checked cache: DataSourceOpenStackLocalThis is a reboot of the same machine, so cloud-init doesn’t even call the metadata service.
If they don’t match, you see this instead:
cache invalid in datasource: DataSourceOpenStackLocal
init will now be targeting instance id: <new-id>. new=Truenew=True means a new instance, so cloud-init creates an empty sem/ directory. Every once-per-instance module now runs again.
This is exactly why a VM built from a golden image still initializes correctly. The image contains a cache from the build machine, but the UUID doesn’t match, so cloud-init throws it away.
When you actually want it to run again
Sometimes you need a full re-run. In our case, a VM booted while the metadata service was unreachable. cloud-init fell back to an empty datasource and skipped everything.
A plain reboot didn’t help, because the markers were already written. The fix was:
sudo cloud-init clean --logs --rebootThis deletes the cache and the semaphores, so the next boot looks brand new.
If you only need one module, you can re-run just that one:
sudo cloud-init single --name set_hostname --frequency alwaysBe careful with scripts_user, though. It reruns your entire runcmd. If that script formats disks or appends to /etc/fstab, running it twice can hurt.
A common trap
If you’re thinking “I’ll just put my config in runcmd and it’ll stay in sync,” that’s where it bites.
runcmd is tied to once-per-instance. Change your user data later and reboot, and nothing happens. cloud-init sees the marker and moves on. For work that must happen on every boot, use bootcmd or drop a script into per-boot/. For ongoing configuration, use a proper configuration management tool.
How to check your own machines
These three commands answer “what ran on this boot?”:
sudo cloud-init analyze show
ls /var/lib/cloud/instance/sem/
sudo grep -nE 'check-cache|previous iid|new=(True|False)' /var/log/cloud-init.logLook for ran successfully versus previously ran. That one word tells you whether a module did real work or just checked its list.
As the old Unix saying goes, “Do one thing, and do it well.” cloud-init’s one thing is turning a new instance into your instance. After that, it mostly stands guard and stays out of the way.
Have you ever been surprised by something cloud-init did, or didn’t do, after a reboot? Share it in the comments. I’d love to hear your stories.
