storage limits questions freenas vs ceph

[iliak]

we are growing and we need a larger approx 50-200 TB (redundant) ( we will scale withing a year to the full capacity, sdds will be added on demend)
currently i cant decide if we should go for freenas or ceph

ceph

• cons
  • much higher storage price, due to 2 or 3 replications
• pros

freenas

• cons
  • our main concern the throughput will be limited, as a single node storage, will be limited by cpu\network of single node
  • single node failure
• pros
  • cheaper storage, at lower redundancy rate

we are planning to go for 4 Tb ssds (new sata or refubrished sas\u2)
what do you think?

 

[blanchet]

FreeNAS is easier to setup than CEPH, and can be operated by quite anybody.
For CEPH, you probably need a seasoned storage engineer to setup and maintain the system.

 

[HoneyBadger]

Boilerplate questions:

What are you storing?
What is the access pattern?
What are the redundancy requirements?
What is the speed requirement?

That will determine if you should have a single system or a scale-out.

 

[iliak]

FreeNAS is easier to setup than CEPH, and can be operated by quite anybody.
For CEPH, you probably need a seasoned storage engineer to setup and maintain the system.

we all ready have proxmox that is integrated to a DC, and a lot of other systems, I'll manage to set it up

What are you storing?
What is the access pattern?

we need two main pools\stores one for lxc\vm and it will be based on high end u2 ssds, but because the size is small this is not an issue,
currently the containers hostess on qnap\freenas storage the we do a daily image backup, but if we will have a hardware failure it will take a lot of time and effort to recover

Ohe main issue is a large data store (mainly for read access) based on files (1-4 GB each)

What are the redundancy requirements?

we are trying to remove single point of failure that us used by our qnap and current freenas server for our critical services

What is the speed requirement?

our large data pool will be access by hundreds of clients, each client read one file every 60 seconds
current requires size is 100TB -200TB

 

[blanchet]

If you want to stay with FreeNAS without a single point of failure, you can purchase a TrueNAS HA server. iXsystems has special offers for this quarter.

For only 200TB, Ceph is probably overkill, except if you want to scale later to peta-bytes or thousands of clients.

 

[iliak]

If you want to stay with FreeNAS without a single point of failure, you can purchase a TrueNAS HA server. iXsystems has special offers for this quarter.

For only 200TB, Ceph is probably overkill, except if you want to scale later to peta-bytes or thousands of clients.

TrueNAS HA seems good but it not cheap, we need two pools one small and fast and and one large for read only tasks so ti dong think it give use good value for $$

we would like to receive the option of scalability, i am a big fan of opensource projects, so that way i love freenas and currently have one that work quite good.

can i configure freenas HA on my own hardware ?

 

[blanchet]

You cannot configure HA on FreeNAS. You can only minimize down times with a cold spare.

On the other hand, some people have tried to use HAST on vanilla FreeBSD. But it is not very common, I am not sure that it is production-ready.

• https://www.ixsystems.com/blog/configuring-a-highly-available-service-on-freebsd-part-1-hast/
• https://mwl.io/archives/221
• https://www.ixsystems.com/community/threads/high-availability-freenas-setup.15867/

If you want to run an open source and HA system, you should use only solutions with demonstrated running instances.
Basically it means either Ceph or Lustre.

Public Health England builds Ceph and Lustre cloud for analytics | Computer Weekly

Healthcare analytics and modelling organisation built an open source cloud from Red Hat Ceph object storage and Lustre scale-out file access storage to petabyte scale

www.computerweekly.com

 

[HoneyBadger]

TrueNAS HA seems good but it not cheap, we need two pools one small and fast and and one large for read only tasks so ti dong think it give use good value for $$

we would like to receive the option of scalability, i am a big fan of opensource projects, so that way i love freenas and currently have one that work quite good.

can i configure freenas HA on my own hardware ?

My opinion is that you have two very different storage needs here, so you should probably build two very different systems/infrastructures to deliver it.

For your container and VM storage, you need high performance read/write of block (or block-like) storage from the ProxMox hosts. As far as configuring your own HA solution on top of FreeNAS - I consider FreeNAS to be an "appliance" of sorts, and rather than try to shoehorn and modify the appliance to do something it wasn't intended to, I would suggest creating it from the ground up on your own platform of choice, be that FreeBSD, Linux, or a Solaris derivative. Use a number of high-performance SSDs, build and tune the unit(s) directly for high performance at the appropriate block sizes.

You can then build a larger, scale-out ceph/glusterFS solution that is used and purpose-built to handle the massive capacity of your long-term file store, and focus it towards the heavy read use while decreasing the emphasis on the ingest rate.

As with all self-built solutions though, please make sure that you understand the consequences of choosing to become the "vendor" to the array. That means they will call you when things go wrong or don't work as they think they should. Make sure you're being paid enough to deal with that level of stress. ;)

 

[iliak]

My opinion is that you have two very different storage needs here, so you should probably build two very different systems/infrastructures to deliver it.

For your container and VM storage, you need high performance read/write of block (or block-like) storage from the ProxMox hosts. As far as configuring your own HA solution on top of FreeNAS - I consider FreeNAS to be an "appliance" of sorts, and rather than try to shoehorn and modify the appliance to do something it wasn't intended to, I would suggest creating it from the ground up on your own platform of choice, be that FreeBSD, Linux, or a Solaris derivative. Use a number of high-performance SSDs, build and tune the unit(s) directly for high performance at the appropriate block sizes.

You can then build a larger, scale-out ceph/glusterFS solution that is used and purpose-built to handle the massive capacity of your long-term file store, and focus it towards the heavy read use while decreasing the emphasis on the ingest rate.

I know, but i'm trying to make one solution to match all the requirements. and till now i cannot decide what to pick, case freeass is fast and simple but without HA, going with truenas is over our price range, and cept i cannot estimate the performance and make sure it will be fast enough

As with all self-built solutions though, please make sure that you understand the consequences of choosing to become the "vendor" to the array. That means they will call you when things go wrong or don't work as they think they should. Make sure you're being paid enough to deal with that level of stress. ;)

I already mange all the devops and IT of the company, so we will add one more system

 

[blanchet]

20 years ago, most systems had no HA at all. But people runs business without it: they just had ready-to-use spare and maintenance windows for hardware/software upgrade.

A viable option would be:

• buy two identical SuperMicro servers (with dual power supply and redundant boot devices)
• The first one will be use to run the production.
• The second one will be use for spare and to run pre-production test (for example to test FreeNAS software upgrade).
• You can achieve easily 99.99% uptime with such a setup
• expand with JBODs when you need more space

You should also consider buying a backup system, because you can destroy a dataset by mistake. In general, people buy another freenas server but with large spinning disks (14 TB) instead of tiny SSD.

You can buy FreeNAS certified servers from iXsystems or from many other vendors in the world. For example Thomas-Krenn.
Nevertheless iXsystems servers have the best hardware compatibility with FreeNAS..
For example iXsystems sells their servers with Chelsio NICs instead of Intel NICs like other vendors do.
For most users, that uses only NFS/CIFS/iSCSI it makes no difference, but virtual networks (Jails+VNET or BHyve vm) are instable with ixl(4) on FreeNAS 11.2u6. On the other hand, virtual networks works normally with cxl(4).

 

[HoneyBadger]

I know, but i'm trying to make one solution to match all the requirements. and till now i cannot decide what to pick, case freeass is fast and simple but without HA, going with truenas is over our price range, and cept i cannot estimate the performance and make sure it will be fast enough

No HA solution will ever be "simple" - buying a pre-made one just shifts that cost from "your own time" to "dollars and cents."

I do have to agree that Ceph likely won't be as fast as a non-scale-out solution, even with fast networking and storage devices.

If you're willing to build multiple Ceph nodes, have you considered building two separate FreeNAS systems for the two separate purposes? One that's designed to accept a bunch of U.2 NVMe disks and serve it very fast, and another that's much less intense and uses chained JBOD disk shelves for expansion? Assuming you went with 12-drive 2U shelves and 14TB drives, you could make each shelf into 2x 6-drive Z2's for 8x14=112T of usable space in each.

There's also opportunities to use something like Linux/xfs and SMR drives to make a scale-out blob (Binary Large OBject) solution on the cheap, but that's outside the purview of this forum.

 

[Jessep]

Dell R740xd2 is (26) 3.5" drives per 2U.