We use nested virtualization only for Testing, Development, and POC. But the real value of this setup comes when you move to production-ready Kubernetes clusters — especially when deploying Rook-Ceph in complex environments.

In real-world production setups, networking issues can be just as tricky as in labs. Ceph daemons timing out, RGW endpoints not reachable, OSD pods flapping — we’ve seen all of this at scale.

Here’s how we diagnosed and solved these issues — and how you can too.

---

1. The Problem: Ceph Services Couldn’t Reach Each Other

We were running a full Kubernetes + Rook-Ceph deployment. Everything seemed fine — until Ceph started acting weird:

• OSD pods would crash or fail to report Ready
• RGW endpoint wasn’t reachable from other pods
• ceph -s showed missing or degraded services

Our first suspicion? Networking. Kubernetes overlay networks often add complexity that can break service discovery and inter-pod communication.

---

2. Diagnosis: Overlay Networking ≠ Cluster Stability

We tried multiple CNIs: Flannel, Calico, and custom setups. The result was inconsistent:

• Sometimes pods could talk to each other, sometimes not
• Ceph Monitors had difficulty forming quorum
• Internal service discovery (via DNS) randomly failed

It turned out that the Kubernetes overlay network was creating unpredictable routing paths. Layer 3 visibility across nodes was unreliable.

---

3. The Fix: hostNetwork: true to the Rescue

The root cause was clear: overlay networks were too brittle for certain Ceph components.

We modified the Rook-Ceph cluster.yaml to enable host networking for critical components:

network:
  hostNetwork: true

This allowed Ceph pods (Mon, MGR, OSD, RGW) to bind directly to the node’s host interface IPs. No overlay, just raw sockets. After that change:

• Monitors formed quorum instantly
• RGW became reachable via its public endpoint
• No more dropped packets or timeouts

---

4. Additional Tweaks That Helped

We didn’t stop at hostNetwork. To ensure stability, we also:

• Set explicit public_network and cluster_network CIDRs in Ceph config
• Disabled unnecessary K8s probes to avoid false-negatives
• Adjusted MTU to match underlying interfaces
• Ensured cloud-init brought up static IPs before kubelet

---

5. Why This Matters: Reliable Production-Grade Infrastructure

We use this setup to:

• Deploy multi-tenant Ceph S3 clusters
• Run OpenStack + Ceph in hybrid environments
• Build demo environments that mirror production

But these environments need to be predictable. And network failures make them anything but.

By forcing host networking and tuning other components, we’ve been able to create repeatable, stable, and fast Ceph clusters that reflect real-world behavior.

---

6. Lessons Learned (And Why You Might Want Help)

If you’re deploying Rook-Ceph in a Kubernetes setup:

• Don’t rely on default networking
• Validate pod-to-pod connectivity manually
• Use hostNetwork: true when in doubt

This post should give you a head start. But if you’re building a production pipeline or a serious infrastructure stack, getting expert help can accelerate things.

At Kubedo, we operate Rook-Ceph clusters at scale — on bare metal, OpenStack, and hybrid environments. We know the gotchas, and we build automation to avoid them.

---

Want help building your own Ceph-powered infrastructure? Let’s talk.