freenas for rsync

[retroman]

Hello everybody,

I have 15 synology boxes around the country. they save via rsync every night on my old freenas. total is around 7Tb. 1 million files. office/jpegs and so. No vm and no very large file except a dozen of pst around 20 gb.
snapshot taken everyday and kept for 3 month.

The old freenas is 16gb non ecc, i5 2500 and 4x4tb raidz1. lz4 and no dedup.
backup time takes long because there is 4 or more rsync at the same time. It's not a problem when sending data (syno boxes are around 1mbps each) but when comparing file, I think my disks have hard time. busy around 100%

I'm building a new freenas with better and more reliable hardware. HP ml350 gen9 with 6 core xeon, 32gb ecc and 4x8tb. I want to keep my old freenas for a backup of the backup every night via replication
Should I raidz1, raidz2 or mirror ? any 1tb ssd will speed up ? more ram ?

 

[kdragon75]

If you looking for dedup on large datasets, you need a LOT of RAM for it to be stable. I would START with 128GB and double that for any kind of performance. Also be sure to set a large block size for any datasets or zvols. I only mention this because you said you were looking for dedupe in your into thread. I would not consider 32GB a safe about to dedupe several TB of data.

 

[retroman]

no, i'm not interested in dedup.
just more i/o for
- rsync comparing file or checksum or I don't know how it works...
- replication task at the same time of rsync if needed

 

[Chris Moore]

4x8tb.

not interested in dedup.
just more i/o for

More IO requires more disks. Instead of going to 4 x 8TB drives, you should be looking to go to 8 or 12 drives or more.
The number of vdevs dictate the amount of IO you can handle. If you have two vdevs, you can handle twice the IO.
You are getting bigger drives, but you need to be looking at getting more drives. We have systems where I work that are running around 300 drives to get the needed IO and each individual drive is only around 1TB.
As an example and these are estimates based on number from thin air because I don't know your exact hardware or your exact requirements, if you had two vdevs of 6 drives each, you should expect to get around 580 IO/s (input output operations per second), but if you only have one vdev, the number of IO/s drops to around 290. This is a mechanical limitation of the spinning disk media and the only way to get faster is to get more disks.
I can give you more detail if you need it.

 

[Chris Moore]

HP ml350 gen9 with 6 core xeon, 32gb ecc and 4x8tb.

Be sure that HP has a plain SAS HBA and not some hardware RAID controller. HP is one of the vendors that likes to integrate hardware RAID and we don't want that with FreeNAS.

 

[retroman]

yes I added a cheap 6gb/s mini sas/sata card to avoid the raid.
I understand what you mean. more i/o more disks.
Is it possible/wise to build a pool with 4x8tb + 8x1tb in mirror ? I have a bunch of 1tb for free...
what about l2arc ssd ?

 

[kdragon75]

Is it possible/wise to build a pool with 4x8tb + 8x1tb in mirror ? I have a bunch of 1tb for free...

No. In ZFS pools are striped vdevs, vdevs are disks either mirrored or in distributed parity (Like RAID5 or RAID6). vdevs should contain all matching drives. If drives in a vdev do not match, the smallest/slowest will limit the entire vdev.
You can still have two vdevs, one 4x8 in RAIDz2 (large disks should always have double redundancy) and another RAIDz2 (old disks should not be trusted) of the 1TB drives with one or two keep as cold (offline) spares.

Unless you can explain how L2ARC works, you will gain nothing as it requires careful consideration and tuning to work in a beneficial way.

 

[Chris Moore]

what about l2arc ssd ?

L2ARC is mostly about being able to read data from the pool and I don't think it would help you at all in the backup configuration you have described.

Is it possible/wise to build a pool with 4x8tb + 8x1tb in mirror ? I have a bunch of 1tb for free...

It is totally doable to make a pool such as this, however it would give you around 10TB of usable capacity. You would be able to replace the 1TB drives later to increase capacity.
Since this system is for backup, and speed is the big issue, I think this could be reasonable, but keep a close eye on drive health.

 

[Chris Moore]

PS. Like @kdragon75 said, you would want to have some spares on hand to make the replacement of failing disks quick, before another disk has a chance to fail. You should plan to replace old drives with newer ones at the earliest time because drive failure rates increase sharply around the 5 year mark.

 

[retroman]

10tb ? not 12 tb ?
my 1tb are new (thanks to notebook shipped without ssd...)

 

[Chris Moore]

10tb ? not 12 tb ?

Just off the top of my head, but a drive that is sold (I mostly hate marketing people) as being 1TB is actually only around 877 MB of real usable space, so the 8 x 1TB drives should give you about 3.5 TB of space. The 4TB drives suffer similarly from marketing hype and you should get around 7 TB of usable space there, so probably about 10.5 total and you need to keep the pool under 80% capacity or you can run into performance issues related to the copy-on-write nature of the ZFS file system.

my 1tb are new (thanks to notebook shipped without ssd...)

Notebook (2.5 inch) drives are not high performance by any means. Probably around 100 IO/s per mirror set on those. What kind of 4TB drive are you looking at?

 

[retroman]

ok for the 10tb. but
You got wrong. I'm going to build 4x8tb not 4x4tb. seagate ironwolf.
if notebook drive are slow, should I just 4x8tb

And I made a mistake, today I have a "slow" freenas with 3x6tb red

 

[Chris Moore]

You got wrong. I'm going to build 4x8tb not 4x4tb. seagate ironwolf.

Sorry, I don't know where I got the idea it was 4 x 4 TB drives.. That changes the capacity a bit, but the number of vdevs will still influence the overall speed of the pool. The 8TB drives have a max data rate of 210 MB/s, so you should see (with two mirror vdevs) around 400 IO/s if you just use the four drives as two mirror sets. If you add the additional 8 drives as additional mirror vdevs, it may improve overall throughput to the pool because read and write operations are paralleled across more vdevs. I have not tried using notebook drives in a storage pool, so I am not sure how well they will perform.

https://www.seagate.com/www-content/datasheets/pdfs/ironwolf-12tbDS1904-9-1707US-en_US.pdf

 

[retroman]

ok thanks for all of this
Is there an easy way to benchmark iops on freenas ?