Best practices for large amount of discs

[cyberjock]

Ok, but that actually means absolutely nothing. He's throwing around numbers that make assumptions too. The 5GB/TB of disk space and "his" 20GB/TB of disk space cannot account for every scenario. If your data is 512-byte blocks, you would need something like 800GB of RAM to store 1TB of 512-byte blocks. On the flip side, 1TB of the largest block size (128KB) is something like 1.7GB.

This stuff is nothing more than voodoo magic. 5GB/1TB seems to be a good starting point for most people, and it accomplishes the two things it needs to accomplish:

1. If you have more than 10TB or so of disk space, the cost of RAM is going to be excessively expensive.
2. Just like the manual says "there is no upper limit" and "the pool will not mount" there is clear and present danger to doing this.

Frankly, I can't believe we're still discussing this at all. Dedup only works for very explicit situations and only at significant risks. Why it is even *worth* discussing besides "don't do it or you'll be sorry" is not going to change the outcome- you still shouldn't do it or you'll be sorry.

 

[KempelofDoom]

Ok, but that actually means absolutely nothing. He's throwing around numbers that make assumptions too. The 5GB/TB of disk space and "his" 20GB/TB of disk space cannot account for every scenario. If your data is 512-byte blocks, you would need something like 800GB of RAM to store 1TB of 512-byte blocks. On the flip side, 1TB of the largest block size (128KB) is something like 1.7GB.

This stuff is nothing more than voodoo magic. 5GB/1TB seems to be a good starting point for most people, and it accomplishes the two things it needs to accomplish:

1. If you have more than 10TB or so of disk space, the cost of RAM is going to be excessively expensive.
2. Just like the manual says "there is no upper limit" and "the pool will not mount" there is clear and present danger to doing this.

Frankly, I can't believe we're still discussing this at all. Dedup only works for very explicit situations and only at significant risks. Why it is even *worth* discussing besides "don't do it or you'll be sorry" is not going to change the outcome- you still shouldn't do it or you'll be sorry.

The article did say there were two camps on the subject and I was really trying to see why those who use it do. Then I could make an educated decision on if I should bother or stay the course and thanks to everyone's responses and articles, I definitely will avoid.

The primary reason I was even curious is that a lot of the content I put on the server has a high rate of being duplicate on the order of maybe 10TB out of the 50TB volume. With my plans of using the expansion case shot down till I can afford SAS drives, this was something I pondered would be of use not only to save space but to get familiar with another feature of FreeNAS. A lot of the people I work with in the tech industry have never heard of BTRFS or ZFS and I see those as being the future file systems we will all use and by getting experience with them, I can hopefully use it to stay employed. :)

Thanks again everyone for your input!

 

[cyberjock]

There's WAAAAY more than 2 camps. There's about 50 camps depending on how detail-oriented you want to be. Covering every feasible combination of settings and uses quickly results in so many possible outcomes the only way to truly handle the situation for yourself. Of course there is lots of experience that makes suggestions like "dedup is ungodly slow because of CPU resource limitations" but when you hit that so-uncalled 'limit' varies based on many factors and isn't easy to give a solid number.

@KempelofDoom

Just because of the size of your pool and the amount of data you use I feel very confident in saying that dedup is something that is going to be dangerous, perform inadequately slow, and will be excessively expensive in comparison to just having that much real disk space.

 

[mjws00]

KOD, I'd add a couple things. You have a pretty powerful platform to experiment with, and there isn't a person here that isn't all for you setting it up and hammering it to learn something. Dedup is a 'hot' topic, partially due the fact that we can clone an ungodly amount of servers and vm's very quickly... and a HUGE proportion of their filesystem may be identical. But reality strikes REALLY hard when we do it in real time at a block level using the ZFS implementation. There are use cases.... but they are rare, and likely demand a whole hardware layer, and likely vendor, dedicated to solving the problem.

If I had a spare 512GB RAM, Dual or Quad E5 machine kicking around with 50TB; or more likely only 24 SSDs < 24TB. I'd certainly be testing and poking and prodding and breaking crap to discover and documents the limits. Be certain though to play in that space effectively, you can't think about a fusion-io card, and a whack of the fastest ssd's. You need to have a bunch you can just pull off the bench ;) Even then you'll only confirm, that the workload that benefits is extremely narrow... and costly.

Btw, no need to wait for SAS drives for your enclosure if you don't want to. Your expander is designed to live in the expansion chassis. But to run a single 8088 you'd normally install a res2cv240, it's only a couple bucks more. Unfortunately the last 4 drives on your set up are driven by the onboard M1015, but you could leave em in the head unit? Or buy great cables and hope it works. 20 drives will be perfect for sure.

I don't think you really hit any decent conclusion on best practices either. The bottom line is you optimize for a specific workload, and allocate drives to vdevs accordingly. The only real considerations are don't go wider than 12 drives, if you need iops stick with mirrors. Everything is a trade-off, speed, vs space, vs redundancy.

As long as you're learning and having fun all is well. :)

 

[KevinM]

The best performance is achieved by using 2^x + y drives, where x is 1, 2 or 3 and y is the number of extra/parity drives. So, for example, a RAIDZ3 setup of 7 drives (2^2 + 3) or a RAIDZ2 setup of 10 drives (2^3 + 2). So if I had 24 drives available, I would probably configure them as two sets of 11 drives each in RAIDZ3 (2^3 + 3) and keep two drives as spares.

Which would be faster, 2 x 11-drive RAIDz3 vdevs or 4 x 6-drive RAIZ2 vdevs? Both layouts follow the 2^x+y rule and with 24 drives they would have the same usable capacity.

 

[SweetAndLow]

More vdevs will be faster

 

[KempelofDoom]

Which would be faster, 2 x 11-drive RAIDz3 vdevs or 4 x 6-drive RAIZ2 vdevs? Both layouts follow the 2^x+y rule and with 24 drives they would have the same usable capacity.

Well since I have 2 almost identical servers I can do some benchmarks. I'll have to wait till I finish transferring my data from the "old" one and can re-set it up so it's 4 groups of 6 disks in RAIDZ2. The server I'm transferring to is 2 vdev's of 11 disks in RAIDZ3. I have 2 drives setup as hot spares. So far, transferring to the new one has been smooth. I have been running scrubs on the new one in between each dataset I transfer. Those have been consistently 900+MBps whereas the previous one of 3 vdev's of 8 disks in RAIDZ1 would only average 250MBps for it's scrubs. Once I finish reconfiguring the "old" one, I'll transfer one of the largest datasets back to it and will report the rates I see. I'm using SCP to transfer so I can avoid adding a middle man to the process and slow down the transfer. Stay tuned!


EDIT
I should also note that I chose the RAIDZ3 setup on the new one because I will be running this box for some time and will likely exceed the time when RAIDZ2 is no longer enough to protect me against data loss. Although, it stings just a little to have 2 drives running as hot spares but this is enterprise stuff, and this is how enterprise stuff is done. ZFS provides me with benefits that I can't get with any current FS and BTRFS is still a ways from being as polished.

 

[KempelofDoom]

So far I have to say that going with the 4 vdev's of 6 disks each in RAIDZ2 seems to be giving the best performance hands down. The scrubs are more than twice as fast at completing. The transfer speeds hit 1Gps network saturation easily. However, I am not comparing apples to apples completely here. The server running the 2 x 11 disk vdev's in RAIDZ3 are all 4TB WD Reds. The server running the 4 x 6 disk vdev's in RAIDZ2 are half 4TB Seagate 7200 RPM drives and the other half are 3TB Seagate 7200 RPM drives. The processor, RAM, mobo, controller and expander are all identical as is the case and cables. So the drives are the differentiating factor. I would think with this many drives, spindle speed wouldn't factor in and I'm still not convinced it does. As was stated before, more vdev's usually translates to more throughput.

Though with the size of the drives, how "safe" or "smart" is it to go the RAIDZ2 route in my situation over the RAIDZ3 as I do plan to run this server for many, many years to come. Eventually, when 6TB and 8TB drives drop in price, I'll swap out the drives to expand the pool so I'm thinking RAIDZ3 is the "future proof" way to go but I really was shocked to see how awesome the RAIDZ2 setup went.

Basically,
RAIDZ3 - 80~100 MBps transfers on average
RAIDZ2 - 100~120 MBps transfers on average

This is just for serving media but I do transfer gigs of data on the norm and the faster it is moved, the less time I spend tending to the farm and more time consuming my media.