← RESEARCH
LAB / 2026.10 CLOUD RESEARCH

OpenNebula Cloud Compute

A study building a minimal self-hosted IaaS from open-source, cloud-native components — OpenNebula over nested KVM — covering the control plane, shared-storage live migration, VXLAN overlay and multi-tenant VPC routing, end to end.

OpenNebulaKVMNFSVXLANVyOS

Why

A cloud orchestrator hides a lot behind a simple API. This research builds a minimal but complete IaaS from open-source, cloud-native components to understand — hands-on — every block a real cloud needs, from the control plane down to the edge router: control plane → compute → shared storage → live migration → overlay network → internet egress.

The focus is open source and self-hosted throughout: OpenNebula as the IaaS orchestrator, KVM/QEMU for compute, NFS for shared storage, VXLAN for the overlay network, and VyOS for edge routing. The aim is to see exactly which primitives an orchestrator drives underneath, and to experiment with networking and lifecycle freely rather than treating the stack as a black box.

The constraints shape the design. With no spare bare metal, the cluster runs nested KVM on an existing hypervisor — the KVM nodes are themselves VMs. The lab network is isolated, reachable only through a VPN and a jump host. Every result is verified on real infrastructure rather than assumed.

Scope

Stand up and verify — on real infrastructure, not on paper — a minimal IaaS with every block a real cloud needs:

  • Control plane — OpenNebula front-end (oned + Sunstone/FireEdge + XML-RPC).
  • Compute — two KVM nodes, balancing and live-migrating VMs between them.
  • Storage — NFS shared datastore, the precondition for live migration.
  • Network — VXLAN overlay for VMs, a VyOS edge router doing NAT to the internet, and a private VPC layer with OpenNebula vRouters for multi-tenancy.
  • Images — marketplace OS images the team can instantiate.

Billing, true multi-zone HA, and distributed storage (Ceph) are out of scope.

Problems & Solutions

Live migration has three easily-missed preconditions

Live migration fails unless the disk is on shared storage, the host is not named localhost, and libvirt can resolve the destination hostname. Each missing condition produces a very different error message. NFS supplies the shared datastore, and the hosts are registered by resolvable name — making two-way migration complete in ~2.5s with no VM reboot.

VXLAN across vendors

VyOS 1.4+ defaults to UDP 4789 while OpenNebula uses 8472; mismatched ports leave the link up but carrying no traffic, so the ports are aligned explicitly. Because the overlay MAC lives inside the UDP payload, VXLAN over a VMware port group needs neither promiscuous mode nor forged-transmits, unlike bridged L2.

Non-interactive vRouter provisioning

The OpenNebula vRouter is a VNF appliance that configures NAT during contextualization, so it reaches the internet only 1–2 minutes after RUNNING. Its default template carries USER_INPUTS, so a no-input template plus the VROUTER=YES flag is used to instantiate it non-interactively.

Persisting a stateless edge router

The VyOS stream image ships without open-vm-tools and runs a squashfs/tmpfs root, so configuration is persisted through /config and a boot script — keeping the edge router alive across reboot and live-migration.

What the lab proved

Every block was verified on real infrastructure, not assumed:

  • Two-way live migration in ~2.5s with no VM reboot — RAM copied over the underlay while the disk stays on NFS, so the VM keeps its IP and gateway.
  • VXLAN overlay connectivity across both hosts — VMs ping each other over VNI 100 with 0% loss, once the UDP ports are aligned.
  • Internet egress through the VyOS edge router — a VM on the public overlay reaches 8.8.8.8 and exits at a single public IP via VyOS NAT.
  • Per-tenant VPC routing with isolation — two VPCs each reach the internet through their own vRouter NAT, while cross-VPC traffic is fully blocked (100% loss on a cross-VPC ping).
  • The VyOS edge router survives a reboot — gateway and NAT restore automatically, so overlay egress recovers without manual intervention.

Architecture

G frontend Control plane OpenNebula front-end (oned · Sunstone/FireEdge · XML-RPC) kvm1 KVM node 1 frontend->kvm1 kvm2 KVM node 2 frontend->kvm2 kvm1->kvm2 live migration (~2.5s) nfs NFS shared datastore (both nodes) kvm1->nfs vxlan VXLAN public overlay (VNI 100) kvm1->vxlan kvm2->nfs kvm2->vxlan vpc Private VPCs (per-tenant VNI) vrouter OpenNebula vRouters (per-VPC NAT) vpc->vrouter SNAT vrouter->vxlan vyos VyOS edge router — NAT vxlan->vyos internet Internet vyos->internet

The lab exercises the core primitives any IaaS orchestrator exposes — instances (onevm), networks (onevnet / VXLAN), routers (onevrouter) and volumes — driven by hand through the oneadmin CLI, so the full control surface is visible rather than hidden behind an API. Every network stage is validated by measuring the real egress IP, confirming packets reach the public internet rather than only the gateway.