cloud-init and NetworkManager: Who Really Configures Your VM’s Network?

Part 3 of “cloud-init Under the Hood”: the handoff that gives your instance its IP address.

You probably think cloud-init sets your VM’s IP address.

It doesn’t. On our VM, cloud-init held a working IP for about a minute, then deliberately threw it away. The address your server actually uses was brought up by NetworkManager, a few seconds after cloud-init had already stepped aside.

In Part 2, we walked through cloud-init’s four boot stages. I skipped one detail on purpose: in the very first stage, cloud-init writes a network config file and leaves the network down. So who turns it on?

By the end of this article, you’ll know exactly where cloud-init stops and NetworkManager starts, which files sit between them, and how to tell in seconds which side broke when a new VM has no network.

---

The chicken-and-egg problem

Every new cloud VM faces a small paradox at boot.

To learn its network settings, the VM must ask the metadata service at 169.254.169.254. But to reach that address, it already needs a working network. And it can’t have a proper network until it knows its settings.

cloud-init solves this like a relay race. It runs the first leg on a borrowed track, then hands the baton to a runner built for the long distance.

---

Leg 1: cloud-init borrows a network

The first stage, init-local, starts before the operating system configures any interfaces. Here is what our log shows, with addresses anonymized:

dhcp.py: Performing a dhcp discovery on eth0
ephemeral.py: Received dhcp lease on eth0 for 10.0.0.20/25
subp.py: ip -4 route append 169.254.169.254/32 via 10.0.0.2
subp.py: ip -4 route append 0.0.0.0/0 via 10.0.0.1

cloud-init runs a one-shot DHCP client, gets a temporary lease, and adds a route to the metadata service. It downloads network_data.json along with the rest of the metadata.

See also: Mastering the Linux Command Line — Your Complete Linux Training Course

Then it tears everything down:

subp.py: ip -4 route del 169.254.169.254/32 via 10.0.0.2
subp.py: ip link set dev eth0 down
subp.py: ip addr del 10.0.0.20/25 dev eth0

If you’re wondering why it would destroy a working network, the answer is simple. cloud-init isn’t a network daemon. It won’t renew leases, react to link changes, or manage DNS for the next six months. Its job in this leg is reconnaissance, not ownership.

---

The baton: config files on disk

Before leaving, cloud-init translates the cloud’s network description into files your OS understands. The component that does this is called a renderer:

distros: Selected renderer 'sysconfig' from priority list:
         ['sysconfig', 'eni', 'netplan', 'network-manager', 'networkd']
util.py: Writing to /etc/sysconfig/network-scripts/ifcfg-eth0 - 176 bytes
util.py: Writing to /etc/resolv.conf - 222 bytes
util.py: Writing to /etc/NetworkManager/conf.d/99-cloud-init.conf - 72 bytes
util.py: Writing to /etc/udev/rules.d/70-persistent-net.rules - 96 bytes
stages.py: Applying network configuration from ds bringup=False

That last line is the key. bringup=False means write the files, but don’t activate anything.

Each file has a clear purpose:

• ifcfg-eth0 describes the interface: DHCP, MTU 1500, matched to the NIC’s MAC address.
• resolv.conf holds the DNS settings from the cloud.
• 99-cloud-init.conf is a small drop-in that asks NetworkManager not to overwrite that resolv.conf.
• 70-persistent-net.rules pins the name eth0 to this MAC, so the name stays stable across reboots.

Which files you get depends on the renderer. cloud-init walks its priority list and picks the first one available on the system. On our RHEL 9 image that was sysconfig. Other systems may end up with a NetworkManager keyfile or a Netplan YAML file instead. The idea is the same.

---

Leg 2: NetworkManager takes over

Once cloud-init-local.service finishes, systemd starts NetworkManager. It reads ifcfg-eth0, runs DHCP on eth0, and brings up the real address.

Here’s a detail that confused me at first: the “real” IP was exactly the same as the temporary one. That’s because the cloud’s DHCP server binds a fixed address to the VM’s MAC. Both legs asked the same question and got the same answer.

When the interface is up, systemd reaches network-online.target, and only then does the second cloud-init stage begin. Its first message about networking is telling:

stages.py: No network config applied.
           Neither a new instance nor datasource network update allowed

cloud-init doesn’t touch the network again. By default, it only writes network config when it detects a brand-new instance. On later reboots, NetworkManager simply reuses the files that are already there.

---

When the runners bump elbows

The two tools share one sensitive area: DNS.

cloud-init writes resolv.conf and asks NetworkManager to leave it alone. Our own bootstrap script later reversed that:

printf '[main]\ndns = default\n' > /etc/NetworkManager/conf.d/9A-override-99-cloud-init.conf
systemctl restart NetworkManager

NetworkManager reads drop-in files in alphabetical order, and later files win. 9A-... sorts after 99-..., so this override hands DNS back to NetworkManager without changing the IP setup.

If you ever see DNS settings flip after boot, look in /etc/NetworkManager/conf.d/ first.

---

A two-question checklist

Doug McIlroy, one of the people behind Unix, summed up its philosophy as “Write programs that do one thing and do it well.” That split is also your fastest troubleshooting shortcut.

When a new VM has no network, ask two questions:

1. Did cloud-init leave the config file? Check ls /etc/sysconfig/network-scripts/ /etc/NetworkManager/system-connections/. If nothing is there, the first leg failed. Run cloud-init status --long and look for DataSourceNone.
2. Is the file there but the interface still down? Then cloud-init did its part. Look at NetworkManager with journalctl -b -u NetworkManager and nmcli device show eth0.

---

Coming up in Part 4

We’ve now seen cloud-init write files all over the system. But it also keeps a private stash under /var/lib/cloud: a cache, a set of markers, and a copy of everything it received. That’s how it knows whether it already did its job.

In Part 4, we’ll open that directory and see how cloud-init remembers.

Have you ever lost an afternoon to DNS changing after boot? Share what you found in the comments.