I have an All-in-one I built in 2016. In 2019 I had 2 drives go bad. Recently I have had another drive go bad in my zfs array. Its time to at minimum replace the drives. It may also be time to do some upgrades. My ZFS dataset has been going strong for 10+ years, starting back on solaris and early Napp-it. I love ZFS.
I have upgraded hardware several times over the years, this last iteration I built and has been running since 2016. It was built with mostly second hand parts. I also run a few VMs.
VMs
ESXi VMs:
• Win Server for AD, local DNS, WINS: 2core 8gb
• FreeNAS -Passthrough of HBAs: 4cores 64gb of ram
• Hosts 2 pools. One main 2vdev-Z2 pool for data storage (just one user accessing and copying, working on large files stored in pool), and a mirrored plex storage pool on 2 disks for plex to read and keep the main pool spun down.
• PlexServer - Windows VM - Transcoding up to 6 streams at a time: 16cores and 16gb allocated. Quadro added for transcoding
• Crashplan Pro Windows VM - Set to backup entire TrueNAS pool using mapped drives: 2core 16gb. Crashplan seems to use a lot of RAM for large backup dataset.
• Windows VM for downloading, 7zip, Unrar: 6core 16gb
• Windows VM for management: 2core 6gb
Hardware
2x Supermicro SC846TQ 24 bay chassis. One is setup as an JBOD Chassis
Main Chassis - ESXi 6
Supermicro SC846TQ 846TQ-R900B - Upgraded power supplies to SQ, PWS-920P-SQ
BPN-SAS-846TQ Backplane
Supermicro X9DRi-LN4F
2x E5-2960 v1
128gb DDR3 1333mhz (16 x 8gb)
INTEL Dual Port 10GB SFP+ 82598EB 10 GIGABIT - Going to 10gbe switch (my desktop is 10gbe)
LSI 9261-8i PCIe 3.0 Raid card w/bbu for VM storage.
2x 480gb Samsung MZ7GE480HMHP-00005 PM853T OEM Enterprise SSD, mirrored (ssd mounted in case)
4x SAS Ultrastar 2tb drives in raid 10.
1x 4tb SATA drive for VM temporary storage of downloads. (internally mounted in case)
LSI SAS9207-8i PCIe 3.0 HBAs to intel expander (running passthrough to Freenas VM)
LSI SAS9207-8E to expander chassis (running passthrough to Freenas VM)
Intel RES2SV240 PCIe x4 SATA SAS RAID 6Gb/s 24-port Expander Card
SAS Breakout fan cables to backplane
Quadro P2000 for Plex transcoding since E5 does not have iGPU (Passthrough to Plex VM)
Expander JBOD Chassis - 20 free drive bays. Has a number of spare drives in it now that I can remove.
SC846TQ 846TQ-R900B 24 bay chassis - PWS-920P-SQ
BPN-SAS-846TQ Backplane
Supermicro CSE-PTJBOD-CB3 JBOD IPMI controller
Intel RES2CV240 SG27402 RAID Expander
SAS breakout cables to chassis backplane
Everything is on UPS and communicating for shutdowns.
Freenas VM Config
Main pool - 32TB. Almost full, 3TB left
2x Z2 vdevs of 10 3tb HGST Ultrastar 7K4000 3TB
2x Hotspares
Plex Pool of 2x WD Red 8TB drives (no redundancy), this is almost full with 2TB left. Pool gets data mirrored to it through a cron job every 2 hr from the main zfs pool to protect from bitrot. Plex reads this pool to keep the main array spun down to save power.
I get 480MB/s Upload and 600MB/s Download through SMB clients.
24 total drives in main case. 4x for VM storage, 20 for ZFS array. 4 drives in JBOD Expander Case
Power Usage:
Main Chassis avg 280w
JBOD Chassis avg 150w
Goals
• Update old 3tb drives that seem to be increasing in failure (could be drives, could also be backplane or breakout cables causing issues). Thinking 8tb or 10tb drives
• Add additional space to ZFS pools, both main pool and plex pool. Target of at least 2-3x the storage.
• Want to maintain zfs system for bitrot protection
• Potentially upgrade mobo/cpu/ram for lower power usage.
• I feel like the majority of the power usage is from the number of disks and the ram.
• Take old 3tb drives and use them for an on-site backup pool
• Simplify if possible.
Questions are:
General recommendations on what to do?
Should I upgrade my hardware or just my disks?
Should I stick with the passive backplanes in my cases or go to expander backplanes?
Should I create a new pool and move the data or replace drives one by one and resliver (seems like a lot of work on older drives)?
Should I add a zil or slog? If so what?
What kind of pool should I create for a backup pool? Single vdev z1?
Whats the best price/performance disk for my main pool these days? Shucking external drives? Buying used enterprise SAS or SATA drives? I am thinking 8-10tb drives. I don’t want shingled drives.
Thanks!
I have upgraded hardware several times over the years, this last iteration I built and has been running since 2016. It was built with mostly second hand parts. I also run a few VMs.
VMs
ESXi VMs:
• Win Server for AD, local DNS, WINS: 2core 8gb
• FreeNAS -Passthrough of HBAs: 4cores 64gb of ram
• Hosts 2 pools. One main 2vdev-Z2 pool for data storage (just one user accessing and copying, working on large files stored in pool), and a mirrored plex storage pool on 2 disks for plex to read and keep the main pool spun down.
• PlexServer - Windows VM - Transcoding up to 6 streams at a time: 16cores and 16gb allocated. Quadro added for transcoding
• Crashplan Pro Windows VM - Set to backup entire TrueNAS pool using mapped drives: 2core 16gb. Crashplan seems to use a lot of RAM for large backup dataset.
• Windows VM for downloading, 7zip, Unrar: 6core 16gb
• Windows VM for management: 2core 6gb
Hardware
2x Supermicro SC846TQ 24 bay chassis. One is setup as an JBOD Chassis
Main Chassis - ESXi 6
Supermicro SC846TQ 846TQ-R900B - Upgraded power supplies to SQ, PWS-920P-SQ
BPN-SAS-846TQ Backplane
Supermicro X9DRi-LN4F
2x E5-2960 v1
128gb DDR3 1333mhz (16 x 8gb)
INTEL Dual Port 10GB SFP+ 82598EB 10 GIGABIT - Going to 10gbe switch (my desktop is 10gbe)
LSI 9261-8i PCIe 3.0 Raid card w/bbu for VM storage.
2x 480gb Samsung MZ7GE480HMHP-00005 PM853T OEM Enterprise SSD, mirrored (ssd mounted in case)
4x SAS Ultrastar 2tb drives in raid 10.
1x 4tb SATA drive for VM temporary storage of downloads. (internally mounted in case)
LSI SAS9207-8i PCIe 3.0 HBAs to intel expander (running passthrough to Freenas VM)
LSI SAS9207-8E to expander chassis (running passthrough to Freenas VM)
Intel RES2SV240 PCIe x4 SATA SAS RAID 6Gb/s 24-port Expander Card
SAS Breakout fan cables to backplane
Quadro P2000 for Plex transcoding since E5 does not have iGPU (Passthrough to Plex VM)
Expander JBOD Chassis - 20 free drive bays. Has a number of spare drives in it now that I can remove.
SC846TQ 846TQ-R900B 24 bay chassis - PWS-920P-SQ
BPN-SAS-846TQ Backplane
Supermicro CSE-PTJBOD-CB3 JBOD IPMI controller
Intel RES2CV240 SG27402 RAID Expander
SAS breakout cables to chassis backplane
Everything is on UPS and communicating for shutdowns.
Freenas VM Config
Main pool - 32TB. Almost full, 3TB left
2x Z2 vdevs of 10 3tb HGST Ultrastar 7K4000 3TB
2x Hotspares
Plex Pool of 2x WD Red 8TB drives (no redundancy), this is almost full with 2TB left. Pool gets data mirrored to it through a cron job every 2 hr from the main zfs pool to protect from bitrot. Plex reads this pool to keep the main array spun down to save power.
I get 480MB/s Upload and 600MB/s Download through SMB clients.
24 total drives in main case. 4x for VM storage, 20 for ZFS array. 4 drives in JBOD Expander Case
Power Usage:
Main Chassis avg 280w
JBOD Chassis avg 150w
Goals
• Update old 3tb drives that seem to be increasing in failure (could be drives, could also be backplane or breakout cables causing issues). Thinking 8tb or 10tb drives
• Add additional space to ZFS pools, both main pool and plex pool. Target of at least 2-3x the storage.
• Want to maintain zfs system for bitrot protection
• Potentially upgrade mobo/cpu/ram for lower power usage.
• I feel like the majority of the power usage is from the number of disks and the ram.
• Take old 3tb drives and use them for an on-site backup pool
• Simplify if possible.
Questions are:
General recommendations on what to do?
Should I upgrade my hardware or just my disks?
Should I stick with the passive backplanes in my cases or go to expander backplanes?
Should I create a new pool and move the data or replace drives one by one and resliver (seems like a lot of work on older drives)?
Should I add a zil or slog? If so what?
What kind of pool should I create for a backup pool? Single vdev z1?
Whats the best price/performance disk for my main pool these days? Shucking external drives? Buying used enterprise SAS or SATA drives? I am thinking 8-10tb drives. I don’t want shingled drives.
Thanks!
