Native CDN with Varnish
Self-hosted CDN research: GeoDNS/Anycast → Edge Varnish → Shield Varnish → Origin, with cache control, invalidation and multi-tenant concerns.
Why
Managed CDN providers — Cloudflare, Fastly, Akamai — deliver edge caching with almost no operational effort, but they charge per request and per gigabyte, abstract away the caching logic behind limited configuration, and keep the infrastructure opaque. At high bandwidth, the per-request and egress costs dominate, and compliance-sensitive workloads inherit data-residency constraints the provider controls rather than the operator.
Varnish Cache compiles VCL to C and runs it as native machine code, giving full control over caching behavior, routing, and invalidation. This research asks whether a self-hosted CDN built on Varnish can match a managed CDN on the cases that matter — edge caching, geo routing, origin protection — while keeping the caching logic, cost structure, and data residency in-house.
Scope
- Two-tier topology: edge PoPs close to users, a shield tier that protects the origin, and the request path from GeoDNS/Anycast through to the origin server.
- VCL for cache-key normalization, grace/keep semantics, and backend health probes.
- Cache invalidation: PURGE, ban, and surrogate-key (xkey) tagging, plus purge fan-out across PoPs.
- Zero-downtime config delivery: hot VCL swapping, validation, canary rollout, and instant rollback.
- Multi-tenancy: cache-namespace isolation, per-tenant routing, purge, rate limiting, and metering.
- Hardware sizing on Dell PowerEdge (R740 through R660): memory, CPU, NIC, and OS tuning.
Problems & Solutions
Thundering herd on cold cache
N edge nodes each fetch from the origin independently on a miss, overloading the backend. A shield tier sits between edge and origin and collapses concurrent misses into a single origin request. Request coalescing within each Varnish instance collapses duplicate in-flight fetches, and grace mode serves stale content while a background fetch revalidates — keeping the edge available even when the origin is down.
Cross-tenant cache leaks
On shared edge infrastructure, tenant A must never receive tenant B’s cached
object. The tenant ID enters the cache hash key alongside the URL, so the key
becomes SHA256(url + tenant_id + protocol). /index.html for one tenant and
the same path for another resolve to distinct cache objects that cannot collide.
Hitch selects the per-tenant TLS certificate by SNI, and vsthrottle enforces
layered rate limits — global, per-plan, per-IP — so one tenant’s traffic spike
or DDoS does not degrade the others.
Deploying config to N PoPs without downtime
Varnish loads multiple compiled VCL programs at once and switches between them
atomically with vcl.use: new requests use the new VCL instantly, in-flight
requests finish on the old one, and the old VCL stays loaded for millisecond
rollback. Changes flow from Git through a CI gate (varnishd -C compilation plus
varnishtest VTC cases), roll out canary-first to one PoP, and auto-roll-back
when a Prometheus rule detects a 5xx spike after a VCL change.
Precise, fast invalidation
URL-level PURGE handles single objects; ban expressions handle regex patterns; surrogate keys (xkey) tag objects so a single call invalidates every object carrying a tag regardless of URL. A central purge service fans the request out to every edge and shield node over HTTP or a message queue.
Architecture
The control plane holds the authoritative state: Git and CI own VCL, Terraform
provisions infrastructure, Ansible or Consul delivers config, and Vault issues
TLS certificates. Each edge PoP runs Hitch for TLS, Varnish for caching, and
varnishncsa for per-tenant JSON access logs that feed analytics and billing.