Part 2 of a series on what cloud-init really does when your VM wakes up.
You probably think cloud-init reads your user data from top to bottom and runs it like a shell script.
It doesn’t. Your instructions are split up, sorted, and handed to four different systemd services that start at four different moments in the boot. Some of your lines run before the network even exists. Others wait until almost everything else is done.
In Part 1, our database VM booted with the wrong name because cloud-init couldn’t reach the metadata service. Once we fixed that, cloud-init analyze show raised a new question. Why did our setup script start more than a minute after boot, and then run for over five minutes?
By the end of this article, you’ll know what each of the four stages does, why your commands run last, and how to read a boot timeline at a glance.
Table of Contents
The theater before opening night
Shakespeare wrote that “All the world’s a stage.” For cloud-init, the boot is literally four of them.
Picture a theater on opening night. The stage crew arrives first, in the dark. The lighting team comes next. Then the costumes and props. The actors walk on last, once everything else is in place.
cloud-init follows the same order:
| Stage | systemd unit | Its job |
|---|---|---|
| init-local | cloud-init-local.service | Find the cloud and get the instructions, before networking starts |
| init-network | cloud-init.service | Prepare the machine once the network is up |
| modules-config | cloud-config.service | Apply system configuration |
| modules-final | cloud-final.service | Run your scripts, last of all |
Here is how long each stage took on that VM’s first boot:
init-local 49.99s
init-network 1.11s
modules-config 0.02s
modules-final 319.73sThe gaps between stages aren’t cloud-init’s fault. That’s systemd waiting for other services, like the network coming online.
See also: Mastering the Linux Command Line — Your Complete Linux Training Course
Act 1: init-local, the stage crew
This stage runs before the operating system sets up its network. Its only goal is to find out where the machine is and what it’s supposed to become.
In Part 1 we saw it grab a temporary DHCP address, download the metadata and user data, set the hostname, and write a network config file. Then it removes the temporary network and leaves.
On our VM, almost all of those 50 seconds went to one thing: probing an IPv6 metadata address on an image where IPv6 was disabled. That’s a story for a later part.
| # | Time | Action | Details |
|---|---|---|---|
| 1 | 11:23:22.856 | check-cache | Result: cache invalid. The cache belonged to the image-build instance, so this boot is treated as a new instance (boot-new-instance) |
| 2 | 11:23:22.859 | Platform detection | OpenStack identified via DMI; DataSourceOpenStackLocal selected. eth0 has no carrier yet |
| 3 | 11:23:22.873 | Ephemeral DHCP | dhclient -1 eth0 obtained 10.0.0.20/25 and added two temporary routes:169.254.169.254/32 via 10.0.0.2 and default via 10.0.0.1 |
| 4 | 11:23:22 → 11:24:12 | URL resolution took 49.157s | cloud-init probes two metadata URLs in parallel: http://[fe80::a9fe:a9fe%eth0]/openstack (IPv6 link-local) and http://169.254.169.254/openstack. IPv6 is disabled in the image, and resolving the IPv6 URL stalled, adding about 50 seconds to first boot |
| 5 | 11:24:12.405 | Read metadata | API version 2018-08-27 selected. Read meta_data.json (1035B), user_data (5440B), vendor_data.json, vendor_data2.json, network_data.json (397B), plus the EC2-compatible tree (hostname, instance-id, block-device-mapping ebs0–ebs4, local-ipv4, security-groups, etc.) |
| 6 | 11:24:12.644 | Tear down ephemeral network | Temporary routes and IP removed; ip link set eth0 down |
| 7 | 11:24:12.665 | Load datasource | Loaded datasource DataSourceOpenStackLocal. Previous instance ID was <instance-uuid-03> (image build VM); this instance is <instance-uuid-04> |
| 8 | 11:24:12.716 | set_hostname | hostnamectl set-hostname db-node-01, taken from hostname/fqdn in user_data |
| 9 | 11:24:12.804 | Write network config | Rendered from network_data.json: NIC fa:16:3e:xx:xx:xx, dhcp4, MTU 1500, written with bringup=False for NetworkManager to activate later in boot |
Act 2: init-network, the lighting team
Now the real network is up, so cloud-init can do the heavier preparation. On our VM, nine modules ran with zero failures in just over a second:
- growpart and resizefs grew the root disk from about 18.8 GiB to about 138.8 GiB, matching the size we asked for in Terraform.
- set_hostname confirmed the name from Act 1.
- users_groups created two admin users and locked their passwords.
- ssh generated brand-new SSH host keys, so no two VMs from the same image share a key.
This stage also parses your user data and writes the merged result to /var/lib/cloud/instance/cloud-config.txt. If you want to see what cloud-init actually received, that file is the place to look.
| Step / module | Frequency | Duration | What it did |
|---|---|---|---|
| check-cache | – | 0.003s | Reused the init-local cache (trust mode). Network config was already written in init-local and is not reapplied |
| consume-user-data | – | 0.033s | Parsed the MIME multipart (one text/cloud-config part) and wrote the merged result to /var/lib/cloud/instance/cloud-config.txt |
| consume-vendor-data / vendor-data2 | – | 0s | vendor_data contained nothing executable |
| schema validation | – | – | 11:24:13.446 cloud-config failed schema validation!; see 6.2 |
| seed_random | once-per-instance | 0.003s | Seeded the kernel entropy pool with random_seed from metadata |
| growpart | always | 0.441s | Root partition /dev/sda4 grown from 20,189,265,408 B (~18.8 GiB) to 149,038,284,288 B (~138.8 GiB), matching the 140 GB root volume in Terraform |
| resizefs | always | 0.032s | xfs_growfs / to grow the root filesystem to the new partition size |
| mounts | once-per-instance | 0.069s | No mounts in user_data; no ephemeral0 or swap device found, so fstab was unchanged (data volumes and swap are handled later by runcmd) |
| set_hostname | once-per-instance | 0.003s | Hostname already set in init-local; confirmed only |
| update_hostname | always | 0.003s | Checked and updated /etc/hostname |
| users_groups | once-per-instance | 0.192s | groupadd sudo (RHEL has no such group by default)useradd administrator --groups sudo --shell /bin/bash -m + passwd -l (lock password)useradd devops --groups sudo --shell /bin/bash -m + passwd -lsudoers rule: ALL=(ALL) NOPASSWD:ALL |
| ssh | once-per-instance | 0.154s | Removed the image’s host keys and regenerated rsa / ecdsa / ed25519 SSH host keys (fingerprints in cloud-init-output.log, comment root@db-node-01) |
| set_passwords | once-per-instance | 0.003s | No password config in user_data; nothing done |
Act 3: modules-config, the prop table
This act lasted two hundredths of a second. You might expect your commands to run here, because runcmd is listed under this stage in /etc/cloud/cloud.cfg.
You’d be surprised. Here’s the log line:
config-runcmd ran successfully and took 0.003 secondsThree milliseconds. The runcmd module doesn’t execute anything. It only writes your commands into a script file at /var/lib/cloud/instance/scripts/runcmd, like a prop placed on a table for later.
| Module | Frequency | Duration | What it did |
|---|---|---|---|
| locale | once-per-instance | 0.009s | Set the system locale (default) |
| runcmd | once-per-instance | 0.003s | Only wrote the runcmd list to a script at /var/lib/cloud/instance/scripts/runcmd. It does not execute it. Execution happens in the final stage via scripts_user |
Act 4: modules-final, the actors
This is where your script finally runs. A module with a less obvious name, scripts_user, picks up that file and executes it:
Running command ['/var/lib/cloud/instance/scripts/runcmd']
config-scripts_user ran successfully and took 319.643 secondsOn our VM, those five minutes covered mounting data disks, installing packages, creating a swap file, and joining a directory domain. Everything we cared about as database engineers happened in this last act.
Why so late? Because your script usually needs everything else to be ready first: the network, the users, the resized disk. Running last is a feature, not a delay.
| Module | Frequency | Status | Duration | What it did |
|---|---|---|---|---|
| reset_rmc | once-per-instance | ran | 0.004s | IBM RMC related; no-op on OpenStack/KVM |
| scripts_vendor | once-per-instance | ran | 0.002s | No vendor scripts |
| scripts_per_once | once | previously ran | 0s | Frequency “once” (once per machine lifetime); already ran during the image build, so skipped |
| scripts_per_boot | always | ran | 0s | /var/lib/cloud/scripts/per-boot/ is empty |
| scripts_per_instance | once-per-instance | ran | 0.002s | /var/lib/cloud/scripts/per-instance/ is empty |
| scripts_user | once-per-instance | ran | 319.643s | Executed /var/lib/cloud/instance/scripts/runcmd (11:24:27 → 11:29:46); |
| ssh_authkey_fingerprints | once-per-instance | ran | 0.014s | Printed fingerprints of users’ authorized_keys |
| keys_to_console | once-per-instance | ran | 0.037s | Wrote SSH host key fingerprints to the console (for first-login verification) |
| install_hotplug | once-per-instance | ran | 0.003s | NIC hotplug events not enabled; udev rule not installed |
| final_message | always | ran | 0.006s | Printed Cloud-init v. 24.4-7.el9 finished … Up 389.19 seconds and wrote /var/lib/cloud/instance/boot-finished |
The modules that never showed up
If you’ve read cloud.cfg, you may have noticed many modules that never appear in analyze show. Puppet, Chef, Ansible, package updates, and others are simply missing.
The log explains it:
Skipping modules 'package_update_upgrade_install,puppet,chef,ansible,...'
because no applicable config is provided.Modern cloud-init only activates a module when your config contains its key. No puppet: key, no Puppet module. It isn’t an error. It’s cloud-init skipping work you never asked for.
How to read your own timeline
Next time a VM feels slow to come up, try this:
sudo cloud-init analyze show
sudo cloud-init analyze blameThe first shows every stage and module in order. The second sorts modules by time, so the slow ones jump out. If one stage dominates, you know exactly where to start looking. On our VM, that was scripts_user, and from there the custom script itself.
You don’t need to memorize every module name. Just remember the order: find the cloud, prepare the machine, configure it, then run your script.
Coming up in Part 3
There’s one handoff I skipped on purpose. In Act 1, cloud-init wrote a network config file and then deliberately left the network down. So who actually brought up the real IP address?
In Part 3, we’ll follow the relay between cloud-init and NetworkManager, and see why neither could do the job alone.
Which stage has caused you the most trouble on your own machines? Tell me in the comments.
