Seani
Dabbler
- Joined
- Oct 18, 2016
- Messages
- 41
Hi.
My server has been running FreeNAS for ~3 years now without any significant problems and I'd be a happy camper if it weren't for the lack of free space (my two pools are at over 95% capacity - which is, as I know, not recommended at all).
The server as it is now:
motherboard: Supermicro X10SRH-CLN4F
cpu: Intel Xeon E5-2620v4 8x 2.10GHz
ram: 128GB ECC DDR4 4 of 8 slots occupied
boot drive: 64GB kingston SSD connected via Sata
pool 1: 8x 8TB IronWolf as Raid Z2 connected via Sata
pool 2: 8x 8TB IronWolf as Raid Z2 connected via 2 SFF-8643 to 4x Sata cables
case: Lian Li D8000 (8 3.5" slots occupied on each side, leaving 2 free on each)
power supply: 750W 80+ platinum plugged into a UPS
fans: several 120 & 140mm noctua PWM fans that are controlled via this script:
5.25" to 5x 3.5" enclosure:
The case itself leaves me 4 more 3.5" slots for additional hard drives. That won't suffice for a decently sized 3rd pool tho (or alternatives I'm addressing further below) which is why I want to use a 3x 5.25" to 5x 3.5" hdd cage (the Lian Li D8000 has 3 unoccupied 5.25" slots at the top left side).
I'm considering to get this one, replacing the fan with a 92mm PWM noctua fan, plugging that directly into the motherboard and including it in the fan control script mentioned above.
SAS Expander vs. additional HBA:
To connect more than 1 additional drive (I have 1 free Sata port left) I either need a SAS Expander to connect to the LSI 3008 chip that is on my motherboard or an additional HBA.
Expander:
I'd need 2 SFF-8643 to SFF-8087 cables to connect an affordable LSI 2008 expander (sufficient for hdds) + 4 SFF-8087 to 4x Sata cables to connect the drives and would effectively lose the 48€ I already spent on the 2 SFF-8643 to 4x Sata cables currently in use.
HBA:
What I'm aiming for is flashing a Dell Perc H310 (LSI 2008) to IT mode because they're inexpensive and yet fast enough for hdds (they go for around 35-40€ on ebay where I'm from) and then connecting the hard drives via 2 SFF-8087 to 4x Sata cables. That would leave the existing wiring in place and only cost around 50-60€ total.
So in the end I'd connect the boot drive and 9 hdds via sata, 8 hdds into the mainboard SAS HBA and another 8 into the Dell Perc H310, ending up with 26 slots and drives.
Or are there reasons for using an expander instead of a PCI/e HBA that I'm unaware of?
Raid Z level + hard drive counts:
With the 5.25" to 3.5" enclosure the possible hard drive count would rise to 9 additional drives. While reading up further about Raid Z1/2/3 hard drive counts and the most space efficient drive counts per level and pool I found that my current setup of 8 drives in Z2 seems not to be optimal in terms of space-efficency.
I had the idea that instead of adding a third pool of 9 drives in yet another Raid Z2 I could "merge" (I know that's not literally possible) the two existing pools into a single Z3 pool of 15 hard drives (leaving me with 1 drive as a spare) and adding a second pool of 10 new drives (size depends on which are cheap on sale in the near future) as Raid Z2.
If the calculator is to be trusted I'd end up with 152.2TB of usable space with 2x 8x 8TB Z3 + 1x 9x 12TB Z2.
Versus 157,2TB of usable space with 15x 8TB Z3 + 10x 12TB Z2. So I'd basically only win 1TB of usable space.
Any thoughts on existing 2x 8x 8TB Z2 pools + an additional 9x xTB Z2 pool vs 15x 8TB Z3 pool + 10x xTB Z2 pool?
Copying the files to the new pool:
If I go with the new pool arrangement of 15x 8TB Z3 and 10x xTB Z2 I would have to copy the contents of the existing pools to the new pool before destroying them to create the new 15 drive Z3 pool.
If I go for 10 drives of 12TB each that should theoretically give me enough space to copy over the entire contents of the 2 existing pools.
A downside would be that I would not be able to fully connect both old pools and the new pool at the same time (since I'm one Sata/SAS port and 3.5" slot short).
I could unplug one of the existing pools, plug in the 10 new drives, copy over pool 1, then unplug pool 1, plug in pool 2 and copy that over as well. And after that I could delete the old pools and connect all the drives in their final (I hope) place.
Is it worth the hassle of copying over so many files to win over a few TB in the end since the configuration is a bit more space efficient? The autist in me says yes. My practical mind says I should simply add another 9 drive Z2, cut my losses and redistribute the data between the 3 pools in a way that leaves enough free space on each of them for ZFS to properly work.
The only upside that comes to mind is that it would kind of reset the ZFS problem of having an increasingly harder time to find fitting free space to write to in a pool. I don't know if that's a problem after 3 years with relatively low write load tho (usually there are maybe 20GB written/deleted per day).
Insight/experiences would be greatly appreciated.
RAM:
I recall reading that you should have 1GB of RAM for each TB of RAW storage space you have in your server. Does that still apply on a huge (for me) scale? Do I need to buy another 128GB of ECC RAM for FreeNAS to work properly (4 slots are unoccupied currently) or do you think it will run without problems even if I add 9 disks with 12TB each?
I've been mulling it all over in my head for weeks and would greatly appreciate it if anyone has some ideas or an opinion to what I intend to do.
My server has been running FreeNAS for ~3 years now without any significant problems and I'd be a happy camper if it weren't for the lack of free space (my two pools are at over 95% capacity - which is, as I know, not recommended at all).
The server as it is now:
motherboard: Supermicro X10SRH-CLN4F
cpu: Intel Xeon E5-2620v4 8x 2.10GHz
ram: 128GB ECC DDR4 4 of 8 slots occupied
boot drive: 64GB kingston SSD connected via Sata
pool 1: 8x 8TB IronWolf as Raid Z2 connected via Sata
pool 2: 8x 8TB IronWolf as Raid Z2 connected via 2 SFF-8643 to 4x Sata cables
case: Lian Li D8000 (8 3.5" slots occupied on each side, leaving 2 free on each)
power supply: 750W 80+ platinum plugged into a UPS
fans: several 120 & 140mm noctua PWM fans that are controlled via this script:
5.25" to 5x 3.5" enclosure:
The case itself leaves me 4 more 3.5" slots for additional hard drives. That won't suffice for a decently sized 3rd pool tho (or alternatives I'm addressing further below) which is why I want to use a 3x 5.25" to 5x 3.5" hdd cage (the Lian Li D8000 has 3 unoccupied 5.25" slots at the top left side).
I'm considering to get this one, replacing the fan with a 92mm PWM noctua fan, plugging that directly into the motherboard and including it in the fan control script mentioned above.
SAS Expander vs. additional HBA:
To connect more than 1 additional drive (I have 1 free Sata port left) I either need a SAS Expander to connect to the LSI 3008 chip that is on my motherboard or an additional HBA.
Expander:
I'd need 2 SFF-8643 to SFF-8087 cables to connect an affordable LSI 2008 expander (sufficient for hdds) + 4 SFF-8087 to 4x Sata cables to connect the drives and would effectively lose the 48€ I already spent on the 2 SFF-8643 to 4x Sata cables currently in use.
HBA:
What I'm aiming for is flashing a Dell Perc H310 (LSI 2008) to IT mode because they're inexpensive and yet fast enough for hdds (they go for around 35-40€ on ebay where I'm from) and then connecting the hard drives via 2 SFF-8087 to 4x Sata cables. That would leave the existing wiring in place and only cost around 50-60€ total.
So in the end I'd connect the boot drive and 9 hdds via sata, 8 hdds into the mainboard SAS HBA and another 8 into the Dell Perc H310, ending up with 26 slots and drives.
Or are there reasons for using an expander instead of a PCI/e HBA that I'm unaware of?
Raid Z level + hard drive counts:
With the 5.25" to 3.5" enclosure the possible hard drive count would rise to 9 additional drives. While reading up further about Raid Z1/2/3 hard drive counts and the most space efficient drive counts per level and pool I found that my current setup of 8 drives in Z2 seems not to be optimal in terms of space-efficency.
I had the idea that instead of adding a third pool of 9 drives in yet another Raid Z2 I could "merge" (I know that's not literally possible) the two existing pools into a single Z3 pool of 15 hard drives (leaving me with 1 drive as a spare) and adding a second pool of 10 new drives (size depends on which are cheap on sale in the near future) as Raid Z2.
If the calculator is to be trusted I'd end up with 152.2TB of usable space with 2x 8x 8TB Z3 + 1x 9x 12TB Z2.
Versus 157,2TB of usable space with 15x 8TB Z3 + 10x 12TB Z2. So I'd basically only win 1TB of usable space.
Any thoughts on existing 2x 8x 8TB Z2 pools + an additional 9x xTB Z2 pool vs 15x 8TB Z3 pool + 10x xTB Z2 pool?
Copying the files to the new pool:
If I go with the new pool arrangement of 15x 8TB Z3 and 10x xTB Z2 I would have to copy the contents of the existing pools to the new pool before destroying them to create the new 15 drive Z3 pool.
If I go for 10 drives of 12TB each that should theoretically give me enough space to copy over the entire contents of the 2 existing pools.
A downside would be that I would not be able to fully connect both old pools and the new pool at the same time (since I'm one Sata/SAS port and 3.5" slot short).
I could unplug one of the existing pools, plug in the 10 new drives, copy over pool 1, then unplug pool 1, plug in pool 2 and copy that over as well. And after that I could delete the old pools and connect all the drives in their final (I hope) place.
Is it worth the hassle of copying over so many files to win over a few TB in the end since the configuration is a bit more space efficient? The autist in me says yes. My practical mind says I should simply add another 9 drive Z2, cut my losses and redistribute the data between the 3 pools in a way that leaves enough free space on each of them for ZFS to properly work.
The only upside that comes to mind is that it would kind of reset the ZFS problem of having an increasingly harder time to find fitting free space to write to in a pool. I don't know if that's a problem after 3 years with relatively low write load tho (usually there are maybe 20GB written/deleted per day).
Insight/experiences would be greatly appreciated.
RAM:
I recall reading that you should have 1GB of RAM for each TB of RAW storage space you have in your server. Does that still apply on a huge (for me) scale? Do I need to buy another 128GB of ECC RAM for FreeNAS to work properly (4 slots are unoccupied currently) or do you think it will run without problems even if I add 9 disks with 12TB each?
I've been mulling it all over in my head for weeks and would greatly appreciate it if anyone has some ideas or an opinion to what I intend to do.
