Cascading Storage

[Griwa]

Hi, just need some advice on my decisions.

We currently run 2 FreeNAS systems at my work, and need a large storage pool for our ongoing projects. The idea is to get a third server, with 2 pools. One which will be store frequent data for our workers (mostly CAD drawings). And second pool, less frequently accessed, which will store large PointCloud data. We calculated at our current expanding rate, we would require close to 1 PB of PointCloud data. After looking at different systems, I am leaning on getting something like: THIS and having it fill up with the main frequent pool disks. Obviously the RAM would have to be swapped for decent ECC memory, but with 6 x8 PCI-E 3.0 lanes, there is lots of room for HBA cards(intent for the 1 x 16 PCI-E lane is fibre network card). The idea is to add cascading JBOD chassis as we need them, for the less frequent accessed PointCloud pool(s).

The point is to not spend too much money initially, and have room for growth as we expand. Will the hardware listed above be up to the task? Should we consider another HBA card, other than the provided LSI 9210-8i? All shares are SAMBA windows shares.

Any suggestions welcome, although I have some experience with FreeNAS, it would be my first time building a cascading system. Thank you in advance.

 

[joeinaz]

If it were me using FreeNAS I would look into 3 things:

1. An external HBA that can be flashed into IT mode. External HBAs support the attachment of a large number of external disks.
2. A high capacity disk enclosure that can be attached to an external SAS controller. SuperMicro makes an external enclosure that holds 90 disks in 4U.
3. Deploying a tape library (or autoloader) using LTFS to deploy a tiered storage solution. LTFS allows for a "disk type" file system to be deployed using tape. The big difference is the access time. This may be especially useful as tape is still less expensive compared to disk. LTFS requires at least an LTO-5 tape drive. Current LTO-8 technology supports up to 30GB per tape cartridge. The LTFS would be deployed outside of FreeNAS.

 

[Griwa]

If it were me using FreeNAS I would look into 3 things:

1. An external HBA that can be flashed into IT mode. External HBAs support the attachment of a large number of external disks.
2. A high capacity disk enclosure that can be attached to an external SAS controller. SuperMicro makes an external enclosure that holds 90 disks in 4U.
3. Deploying a tape library (or autoloader) using LTFS to deploy a tiered storage solution. LTFS allows for a "disk type" file system to be deployed using tape. The big difference is the access time. This may be especially useful as tape is still less expensive compared to disk. LTFS requires at least an LTO-5 tape drive. Current LTO-8 technology supports up to 30GB per tape cartridge. The LTFS would be deployed outside of FreeNAS.

Thank you for the prompt response.

1. An external HBA that can be flashed into IT mode. External HBAs support the attachment of a large number of external disks.
- was part of the plan, hence the linked MoBo, with plenty of potential HBA slots. I'm just curious if the hardware is good enough, or should I consider buying new?

2. A high capacity disk enclosure that can be attached to an external SAS controller. SuperMicro makes an external enclosure that holds 90 disks in 4U.
- that would be the intention for the expanding PointCloud pool(s).

3. Deploying a tape library (or autoloader) using LTFS to deploy a tiered storage solution. LTFS allows for a "disk type" file system to be deployed using tape. The big difference is the access time. This may be especially useful as tape is still less expensive compared to disk. LTFS requires at least an LTO-5 tape drive. Current LTO-8 technology supports up to 30GB per tape cartridge. The LTFS would be deployed outside of FreeNAS.
- I don't think this would be an option for us. Typically we keep project data for 1-2 years, as it becomes outdated after that period. During the 1-2 years, the data would be accessed, just not frequently. Hence I think pushing it onto cascading SAS would be practical for us.

To be honest, I think we kind of have a decent plan on how we are going to be doing it, I just need confirmation that the choices are give or take in the right direction for our circumstances. And if there are anything I should be aware of, prior to purchasing such a system.

 

[joeinaz]

"Current LTO-8 technology supports up to 30GB per tape cartridge. "
Correction 30TB per tape!

 

[Jessep]

LTO-8, 12TB RAW, 10.xTB formatted, 20TB compressed depending on your data (video etc. doesn't compress well if at all).
LTO-M8 are much cheaper (LTO-7 tapes with LTO-8 formatting) 9TB RAW, 8TB formatted.

On our workloads where we archive VM backups which are already compressed we don't see any additional compression on the tapes, we get formatted space.

Drives are expensive (~$3k-5k), tapes are fairly cheap($150-200).

 

[joeinaz]

LTO-8, 12TB RAW, 10.xTB formatted, 20TB compressed depending on your data (video etc. doesn't compress well if at all).
LTO-M8 are much cheaper (LTO-7 tapes with LTO-8 formatting) 9TB RAW, 8TB formatted.

On our workloads where we archive VM backups which are already compressed we don't see any additional compression on the tapes, we get formatted space.

Drives are expensive (~$3k-5k), tapes are fairly cheap($150-200).

Jessep,

The data in question are CAD drawings which may be more compressible than video. In his case consider the following:

1PB data need: with 2 possible solutions:

1. 1PB of disk with 20% overhead for ZFS = about 130 disks (10GB each)
Assuming a cost of $250 for a new 10TB disk. Your cost for disk (not including enclosures) is $32,500

2. 1PB of storage using a 48 slot LTO-8 auto loader and a smaller quantity of disk (for example fill the 36 disk enclosure they were considering with 10TB disks. I show LTO-8 compressed capacity of up to 30TB per cartridge but in this case let's use a capacity of 20TB per tape. The breakout on costs are as follows:

Assuming a cost of $250 for a new 10TB disk. Your cost for disk (using customer server enclosure) is $9,000; Disk capacity after overhead is about 288TB; LTO-8 (48 slot) SAS autoloader; about $8,000; 36 LTO-8 cartridges (@$150 ea) to total about 720TB about $5400. total cost about $22,500.

Both solutions provide about 1PB of storage. The LTO solution is about $10k less expensive (even excluding the cost of the disk enclosures needed for about 100 disks.) The LTO solution also has the added benefit of lower power and cooling costs and it can also be used to provide a system backup for offsite storage. Finally, the use of LTFS can give the users "pseudo" random access to the files on tape.

 

[Chris Moore]

TrueNAS R-Series with OpenZFS & Scale-Up & Scale-Out Storage

The TrueNAS R-Series combines the powerful features of OpenZFS with budget-friendly, single controller hardware for scale-out storage.

www.freenas.org

I highly recommend contacting iXsystems sales staff to get them to give you a quote. They can quote you for FreeNAS certified hardware and the prices are very competitive with what it would cost for you to buy the hardware elsewhere. Unless you are sure you want used gear, because the big cost of something like this will be the drives and I would think that you would want new ones for the capacity and reliability.

 

[Chris Moore]

To be honest, I think we kind of have a decent plan on how we are going to be doing it, I just need confirmation that the choices are give or take in the right direction for our circumstances. And if there are anything I should be aware of, prior to purchasing such a system.

The system you pointed out on eBay should be fine to start and you can add SAS controllers when you are ready to add external enclosures. I had one where I used to work that had four external enclosures connected. Over 140 drives. No trouble.

 

[Griwa]

Thank you all for the replies. I have contacted iXsystems for a quote, although I am thinking of used hardware, with new drives, as a primary solution due to pricing. I have had good luck with used enterprise hardware before.

The data we store, is divided into two segments: CAD drawings (minimal space and frequently accessed; not worth putting on tapes), PointCloud data (largest data pool; has minimal compression due to the large amount of images used to generate it)

After doing a bit more research, since we use SMB for access, it appears clock speeds may be more important than core counts. So I am looking at alternative used hardware to see if I can spot anything, although I think the 2 Xeon E5-2670 v2 may actually be decent.

The LTO-8 tapes may be a good idea for long term storage, but in practice I think it would add complexity if we would need to access them more frequently. Remember our PointCloud data, is really only good for 1-2 years, before it gets completely overridden, during those years we may be accessing the image data for reference.

Talking on here has allowed me to formulate my ideas better, and given me some peace of mind. It is a bit of a learning curve getting into hardware for servers, especially when you are trying to match things and make sure it works on a budget.