OpenNebula Cloud Compute
A study building a minimal self-hosted IaaS from open-source, cloud-native components — OpenNebula over nested KVM — covering the control plane, shared-storage live migration, VXLAN overlay and multi-tenant VPC routing, end to end.
Why
A cloud orchestrator hides a lot behind a simple API. This research builds a minimal but complete IaaS from open-source, cloud-native components to understand — hands-on — every block a real cloud needs, from the control plane down to the edge router: control plane → compute → shared storage → live migration → overlay network → internet egress.
The focus is open source and self-hosted throughout: OpenNebula as the IaaS orchestrator, KVM/QEMU for compute, NFS for shared storage, VXLAN for the overlay network, and VyOS for edge routing. The aim is to see exactly which primitives an orchestrator drives underneath, and to experiment with networking and lifecycle freely rather than treating the stack as a black box.
The constraints shape the design. With no spare bare metal, the cluster runs nested KVM on an existing hypervisor — the KVM nodes are themselves VMs. The lab network is isolated, reachable only through a VPN and a jump host. Every result is verified on real infrastructure rather than assumed.
Scope
Stand up and verify — on real infrastructure, not on paper — a minimal IaaS with every block a real cloud needs:
- Control plane — OpenNebula front-end (oned + Sunstone/FireEdge + XML-RPC).
- Compute — two KVM nodes, balancing and live-migrating VMs between them.
- Storage — NFS shared datastore, the precondition for live migration.
- Network — VXLAN overlay for VMs, a VyOS edge router doing NAT to the internet, and a private VPC layer with OpenNebula vRouters for multi-tenancy.
- Images — marketplace OS images the team can instantiate.
Billing, true multi-zone HA, and distributed storage (Ceph) are out of scope.
Problems & Solutions
Live migration has three easily-missed preconditions
Live migration fails unless the disk is on shared storage, the host is not named
localhost, and libvirt can resolve the destination hostname. Each missing
condition produces a very different error message. NFS supplies the shared
datastore, and the hosts are registered by resolvable name — making two-way
migration complete in ~2.5s with no VM reboot.
VXLAN across vendors
VyOS 1.4+ defaults to UDP 4789 while OpenNebula uses 8472; mismatched ports leave the link up but carrying no traffic, so the ports are aligned explicitly. Because the overlay MAC lives inside the UDP payload, VXLAN over a VMware port group needs neither promiscuous mode nor forged-transmits, unlike bridged L2.
Non-interactive vRouter provisioning
The OpenNebula vRouter is a VNF appliance that configures NAT during
contextualization, so it reaches the internet only 1–2 minutes after RUNNING.
Its default template carries USER_INPUTS, so a no-input template plus the
VROUTER=YES flag is used to instantiate it non-interactively.
Persisting a stateless edge router
The VyOS stream image ships without open-vm-tools and runs a squashfs/tmpfs root,
so configuration is persisted through /config and a boot script — keeping the
edge router alive across reboot and live-migration.
What the lab proved
Every block was verified on real infrastructure, not assumed:
- Two-way live migration in ~2.5s with no VM reboot — RAM copied over the underlay while the disk stays on NFS, so the VM keeps its IP and gateway.
- VXLAN overlay connectivity across both hosts — VMs ping each other over VNI 100 with 0% loss, once the UDP ports are aligned.
- Internet egress through the VyOS edge router — a VM on the public overlay
reaches
8.8.8.8and exits at a single public IP via VyOS NAT. - Per-tenant VPC routing with isolation — two VPCs each reach the internet through their own vRouter NAT, while cross-VPC traffic is fully blocked (100% loss on a cross-VPC ping).
- The VyOS edge router survives a reboot — gateway and NAT restore automatically, so overlay egress recovers without manual intervention.
Architecture
The lab exercises the core primitives any IaaS orchestrator exposes — instances
(onevm), networks (onevnet / VXLAN), routers (onevrouter) and volumes —
driven by hand through the oneadmin CLI, so the full control surface is visible
rather than hidden behind an API. Every network stage is validated by measuring
the real egress IP, confirming packets reach the public internet rather than
only the gateway.