How We Fixed Networking Issues with Rook + Ceph in Kubernetes Environments
We use nested virtualization only for Testing, Development, and POC. But the real value of this setup comes when you move to production-ready Kubernetes clusters — especially when deploying Rook-Ceph in complex environments.
In real-world production setups, networking issues can be just as tricky as in labs. Ceph daemons timing out, RGW endpoints not reachable, OSD pods flapping — we’ve seen all of this at scale.
Here’s how we diagnosed and solved these issues — and how you can too.
1. The Problem: Ceph Services Couldn’t Reach Each Other
We were running a full Kubernetes + Rook-Ceph deployment. Everything seemed fine — until Ceph started acting weird:
- OSD pods would crash or fail to report Ready
- RGW endpoint wasn’t reachable from other pods
ceph -s
showed missing or degraded services
Our first suspicion? Networking. Kubernetes overlay networks often add complexity that can break service discovery and inter-pod communication.
2. Diagnosis: Overlay Networking ≠ Cluster Stability
We tried multiple CNIs: Flannel, Calico, and custom setups. The result was inconsistent:
- Sometimes pods could talk to each other, sometimes not
- Ceph Monitors had difficulty forming quorum
- Internal service discovery (via DNS) randomly failed
It turned out that the Kubernetes overlay network was creating unpredictable routing paths. Layer 3 visibility across nodes was unreliable.
3. The Fix: hostNetwork: true
to the Rescue
The root cause was clear: overlay networks were too brittle for certain Ceph components.
We modified the Rook-Ceph cluster.yaml
to enable host networking for critical components:
network:
hostNetwork: true
This allowed Ceph pods (Mon, MGR, OSD, RGW) to bind directly to the node’s host interface IPs. No overlay, just raw sockets. After that change:
- Monitors formed quorum instantly
- RGW became reachable via its public endpoint
- No more dropped packets or timeouts
4. Additional Tweaks That Helped
We didn’t stop at hostNetwork
. To ensure stability, we also:
- Set explicit
public_network
andcluster_network
CIDRs in Ceph config - Disabled unnecessary K8s probes to avoid false-negatives
- Adjusted MTU to match underlying interfaces
- Ensured
cloud-init
brought up static IPs before kubelet
5. Why This Matters: Reliable Production-Grade Infrastructure
We use this setup to:
- Deploy multi-tenant Ceph S3 clusters
- Run OpenStack + Ceph in hybrid environments
- Build demo environments that mirror production
But these environments need to be predictable. And network failures make them anything but.
By forcing host networking and tuning other components, we’ve been able to create repeatable, stable, and fast Ceph clusters that reflect real-world behavior.
6. Lessons Learned (And Why You Might Want Help)
If you’re deploying Rook-Ceph in a Kubernetes setup:
- Don’t rely on default networking
- Validate pod-to-pod connectivity manually
- Use
hostNetwork: true
when in doubt
This post should give you a head start. But if you’re building a production pipeline or a serious infrastructure stack, getting expert help can accelerate things.
At Kubedo, we operate Rook-Ceph clusters at scale — on bare metal, OpenStack, and hybrid environments. We know the gotchas, and we build automation to avoid them.
Want help building your own Ceph-powered infrastructure? Let’s talk.