New FreeNASer in Cary NC

[FlyingBear]

Greetings! After reading the super-helpful advice on here, I'm awaiting delivery (from The Server Store) of:

SuperMicro 6027R-E1R12N 2U server
Dual Xeon E5-2687 v2 8-core 3.3GHz
256GB DDR-3 ECC Registered (16x16GB)
1 empty tray for 120GB Sandisk SSD boot drive (the cheapest brand-name SSD I could find)
1 Intel DC S3500 800GB SSD
10 Seagate 8TB 7.2K 12Gbps SAS hard drives
2 LSI 9210-8I in IT mode
BPN-SAS2-826EL2 backplane
Intel X540-T2 PCIe 2x10GBase-T
Dual PWS-920p-SQ PSUs
Rails

I plan to create 2 VDevs, each 5x8TB RAIDZ2, and put them in the same pool
The 800GB SSD is for tbd use. Maybe for VMs, i.e. a single drive pool backed up to the main pool daily. In the unlikely event that I need to cache (because 256GB memory), this could also be a future cache.

Any tips or advice please? Thanks!

 

[MatthewSteinhoff]

What is your use case? That really determines your VDEV/pool configuration.

You're buying 6Gbps HBAs but 12Gbps drives with a SAS2 backplane?

Cheers,
Matt

 

[FlyingBear]

Thanks for your offer of help!

On the 12Gbps drives, I needed the 8TB size and I couldn't find any SAS drives at 6Gbps. I didn't look that hard because I preferred to get everything from The Server Store, as they warranty the whole package.

On the use case: there are three:
1. The primary one is a growing repository of population health data: about 100GB added per month as 50 or so text files that I need to parse and structure with a multithreaded app that'll run on the FreeNAS. Hence the beefy Xeons and mucho cores.
2. A secondary use case is for 10-20 TB of raw photo images.
3. Tertiary use case is as a repository mirrored with a Roon server of music.

 

[MatthewSteinhoff]

RAIDZ2 with two VDEVs isn't going to be especially peppy. Going from 8TB SAS as a RAIDZ2 to 10TB SATA as a mirrored stripe looks to be the same price, 2TB more space and with the primary difference being that the striped mirror will have three times the IOPS and bandwidth, give or take.

You might be better off with a processing pool and an archive pool. You could do a mirrored stripe for the processing pool for top performance and then a RAIDZ2 group for efficient storage of relatively inactive data (photos, music, old data sets).

Finally, on the topic of storage, it is easy to add capacity to mirrored stripes two drives at a time. Depending on your initial storage requirements and growth curve, you may be better off not buying all the drives up front but only what you need for 12-18 months. When you're ready to buy more storage, you may be able to pick up 12T drives then for the same price you're paying now for 8/10T drives. In addition to saving money or increasing total capacity, you're not putting hours on drives you don't need, sucking extra power and are reducing your cooling load.

If you're only adding 100GB a month and doing all the processing locally, do you really need 10G? If price, power consumption or heat generation are at all a concern, 10G may not be warranted.

If you're doing large data manipulations locally, do you have enough RAM? Add RAM before L2ARC.

Otherwise, everything looks good and should play well together.

Cheers,
Matt

 

[FlyingBear]

RAIDZ2 with two VDEVs isn't going to be especially peppy. Going from 8TB SAS as a RAIDZ2 to 10TB SATA as a mirrored stripe looks to be the same price, 2TB more space and with the primary difference being that the striped mirror will have three times the IOPS and bandwidth, give or take.

You might be better off with a processing pool and an archive pool. You could do a mirrored stripe for the processing pool for top performance and then a RAIDZ2 group for efficient storage of relatively inactive data (photos, music, old data sets).

Finally, on the topic of storage, it is easy to add capacity to mirrored stripes two drives at a time. Depending on your initial storage requirements and growth curve, you may be better off not buying all the drives up front but only what you need for 12-18 months. When you're ready to buy more storage, you may be able to pick up 12T drives then for the same price you're paying now for 8/10T drives. In addition to saving money or increasing total capacity, you're not putting hours on drives you don't need, sucking extra power and are reducing your cooling load.

If you're only adding 100GB a month and doing all the processing locally, do you really need 10G? If price, power consumption or heat generation are at all a concern, 10G may not be warranted.

If you're doing large data manipulations locally, do you have enough RAM? Add RAM before L2ARC.

Otherwise, everything looks good and should play well together.

Cheers,
Matt

Very helpful, thanks. I run 10G for my NASes and workstations. I've kindof gotten used to real-world 200-300MB/s file transfers . But, you're right: it's gratuitous versus necessary.

I appreciate your advice on the disks....very good point. Thanks!

 

[FlyingBear]

RAIDZ2 with two VDEVs isn't going to be especially peppy. Going from 8TB SAS as a RAIDZ2 to 10TB SATA as a mirrored stripe looks to be the same price, 2TB more space and with the primary difference being that the striped mirror will have three times the IOPS and bandwidth, give or take.

You might be better off with a processing pool and an archive pool. You could do a mirrored stripe for the processing pool for top performance and then a RAIDZ2 group for efficient storage of relatively inactive data (photos, music, old data sets).

Finally, on the topic of storage, it is easy to add capacity to mirrored stripes two drives at a time. Depending on your initial storage requirements and growth curve, you may be better off not buying all the drives up front but only what you need for 12-18 months. When you're ready to buy more storage, you may be able to pick up 12T drives then for the same price you're paying now for 8/10T drives. In addition to saving money or increasing total capacity, you're not putting hours on drives you don't need, sucking extra power and are reducing your cooling load.

If you're only adding 100GB a month and doing all the processing locally, do you really need 10G? If price, power consumption or heat generation are at all a concern, 10G may not be warranted.

If you're doing large data manipulations locally, do you have enough RAM? Add RAM before L2ARC.

Otherwise, everything looks good and should play well together.

Cheers,
Matt

I just realized that I may not have understood you. Please forgive my ignorance. If I wanted to make all 10 drives give maximum performance, which option would I use please?

1. Create 5 vdevs, each one a mirror of 2 drives. Then create a pool of those 5 vdevs.
2. Create 2 vdevs, each one of a 2 mirrors of 2 drives, with those two mirrors striped. Create a pool of those 2 vdevs. This leaves two drives over as a separate 2-drive mirror, which I could use as a separate pool.
3. Something else.

I'm trying to reconcile your advice for "three times" IOPS/bandwidth. I know that you're recommending a separate archive pool, which is different from what I'm hypothesizing here. Maybe you meant that the processing pool would be 3 vdevs, each of 2 mirrored drives, combined into one pool?

Thanks in advance.

 

[MatthewSteinhoff]

Each VDEV has the same performance of the slowest drive in the VDEV, give or take. When you add VDEVs to a pool, you get the VDEV cumulative speed, give or take.

RAIDZ2 + RAIDZ2 (no matter how many drives are in each RAIDZ2) = two VDEVS = 200 IOPS, 400 MB/s

mirror + mirror + mirror + mirror + mirror = five VDEVS = 500 IOPS, 1,000 MB/s

That's using the exact same number of drives. We can argue the details or do the math with the exact specs but these numbers are close enough.

To get the maximum performance, you'd put all the drives into a mirrored strip. Mirrors make inefficient use of space which is what you're giving up by going that route.

If you knew your working data set was never going to exceed a reasonable amount (1TB-2TB), you could put your active data mirrored SSDs (or a strip of mirrored SSDs). That would give you at least 100,000 IOPS and 500 MB/s. Your calculations would FLY.

The reason I'm recommending multiple pools is that, in most cases, you have distinct tiers of data. VMs and databases are often stored on SSDs because IOPS are critical to their proper functioning. Meanwhile, relatively static data (backups, finished projects, etc.), large media (music, photos, movies) and the like don't care about IOPS and don't especially need massive bandwidth.

Multiple pools are often the most cost efficient method for storage. Fast data gets fast storage. Slow data gets high capacity but slow storage.

Let's say you need 2TB of lighting fast storage for your active data set and 50TB of more sedentary storage.

SSD 1TB mirror + SSD 1TB mirror = 2TB storage, 200,000 IOPS, 1,000 MB/s, four bays/ports used, $800 for drives

7 disk RAIDZ2 with 10T drives = 50T storage, 100 IOPS, 200 MB/s, seven bays/ports used, $2,300 for drives

Cheers,
Matt

 

[FlyingBear]

I kinda "knew" that, but your explanation is the clearest that I've seen despite days of research. It really demonstrates that the real striping is between vdevs in a pool. MANY thanks.

 

[anmnz]

1 empty tray for 120GB Sandisk SSD boot drive (the cheapest brand-name SSD I could find)

Don't do it!
https://redmine.ixsystems.com/issues/35065

 

[FlyingBear]

Don't do it!
https://redmine.ixsystems.com/issues/35065

Thanks! I cancelled with Amazon JUST in time. Nice timing on your part, thank you.

What do you recommend for a boot drive please?

 

[MatthewSteinhoff]

What do you recommend for a boot drive please?

We use 16G and 32G USB sticks. The USB advantage is they don't take up bays or HBA ports. Why tie up what could be used by a 10T disk when all you need is a 4G (or less) partition? Get two sticks and mirror them. Your Supermicro motherboard even has an exposed USB port so you can put a stick inside.

If you're not into USB, check out SATA DOM. A bit more expensive than a conventional SSD drive but it won't eat a drive bay.

Cheers,
Matt

 

[anmnz]

Thanks! I cancelled with Amazon JUST in time. Nice timing on your part, thank you.

What do you recommend for a boot drive please?

Anything not mentioned in the posts linked to from that bug report!

I think someone recently mentioned finding Crucial 60GB SSDs available in the US, that's a good brand. Anything Intel will be solid. I bought a low-end Kingston to replace the SanDisk I reported errors with and it's been perfectly fine. People have been using all sorts of cheap SSDs without problems before now.

It looks like that bug report has been rewritten and it no longer makes clear that WD Green is one of the SSDs with reports of corruption; probably best to check the forum posts it links to.

 

[FlyingBear]

Thanks for the replies. I ordered SuperMicro SATA DOM. What a neat solution.