If you’re a Proxmox user and you’re interested in distributed storage, Ceph storage on Proxmox is most likely your best option. It functions extremely well with high availability and is scalable. Ceph can handle large amounts of data, is self-healing, and can help protect your VMs and Containers in the event of a hardware failure. This will be a quick look at how you can configure Ceph storage on Proxmox.
Benefits of Using Ceph with Proxmox
There are a few benefits that you’ll have if you decide to use Ceph storage on Proxmox.
- Scalability: The storage is distributed which allows you to scale out your storage as your needs grow, without downtime.
- High Availability: Ceph ensures that your data is always available, even if some of the storage nodes fail.
- Cost-Effectiveness: You can build a Ceph cluster without the need for expensive proprietary storage solutions.
- Flexibility: Ceph supports block, object, and file storage.
With that said, it’s recommended by Proxmox to use at least a 10 Gb network for Ceph since Ceph is a distributed storage solution. Keep in mind this is suggested and not required and I was able to get it working well on a 1 Gb test environment, but that doesn’t mean it’ll function as well as it probably should.
Hardware Best Practices for Ceph on Proxmox
Server Configuration
- Minimum Nodes: Use at least three nodes for a basic setup, but since Ceph storage scales (for better fault tolerance and performance) five or more nodes are recommended.
- Identical Hardware: It is a best practice to ensure that all nodes have identical hardware specifications to maintain consistency and balance.
CPU
- Type: Ensure the CPU supports virtualization (though this is a requirement for Proxmox VE as well).
- Performance: To keep it simple, use the fastest CPU you can find.
Memory
- Proxmox VE: Minimum 2 GB for OS and Proxmox VE services.
- Ceph: In my opinion, at minimum, 16GB of Memory. For larger clusters, 64GB to 128GB of RAM is recommended.
Storage
- Drive Types: Use SSDs for best performance. NVMe drives are highly recommended for OSD nodes due to their high IOPS.
Networking
- Bandwidth: Use at least 10GbE networking for Ceph storage to function optimally.
Setting Up Ceph Storage on Proxmox
Setting up a hyper-converged Proxmox VE Cluster is fairly straightforward, however, you must have storage available that hasn’t been configured yet. It’s also important to note that since Ceph is distributed storage, you must have multiple different Proxmox nodes in order for everything to function its best.
If you’re hoping to make your Cluster High Availability, you’d want to set this up on each node in the Proxmox cluster.
1. Open your Proxmox host and select Ceph inside of your Proxmox VE Node. On mine, I’ll configure it on all three nodes since I’m planning on using this with high availability. Select Install Ceph.
2. Make sure the No-subscription repository is selected, select the Ceph version you’d like to use, then select Start installation.
3. Enter yes to continue with the Ceph installation.
4. In the Configuration section, select the Public Network and Cluster Network you’d like to use.
5. Select OSD and then select the Disk you’d like to use for Ceph.
6. Create OSD storage on each of your node(s) if you’d like to use it with High Availability. Then, in the Monitor tab, create a Ceph monitor for each node.
7. In the Pools section, create a new Ceph Pool.
8. Moving forward, when creating your Virtual Machines or Containers, you can select the Ceph storage pool that was created and it’ll automatically use the storage we configured.
Conclusion & Final Thoughts on Ceph Storage for Proxmox
Combining a Proxmox VE Cluster with Ceph storage offers powerful, scalable, and resilient storage for your Proxmox server. It’s recommended by the Proxmox team to use Ceph storage with at least a 10Gb network infrastructure, but that’s not to say it’s mandatory. Assess your Proxmox server as well as the requirements you have to determine if it’s an option for you.
