Advise on performance of new build

[bent98]

I'm currently in the final stages of acquiring the necessary components for my upcoming build. Everything has been purchased, except for the hard drives. My current setup involves a TrueNas 13 Core server with 7x8TB WD Red 5400rpm HDs in a single pool, serving as storage for file backups, UHD 4K movies/Mp3s, and media streaming to my 4K TV. Recently, I experienced a scare when the pool went offline, but with the assistance of Honey Badger and Joe, I successfully recovered the pool without any data loss.
Although all the hard drives and hardware are currently error-free, the drives have accumulated 55,000 hours of use. The goal is to use the newly built server to back up my existing server. In the event of hard drive or component failures, I aim to resilver the server with less stress, knowing I have a backup server ready.
Now, I'm navigating the decision-making process, weighing factors such as price, performance, and practicality, recognizing the trade-offs involved in configuring the pools. Additionally, I plan to upgrade the NIC card in both my existing server and main PC to 10GE, as I transfer files between them on a daily basis.

See the new build below:

The distinction between my upcoming server and my current one lies in my decision to opt for 6x20TB WD Red Pro drives in a RaidZ2 configuration, as opposed to the 7x8TB drives in my existing server. The memory capacity will also see an increase, going from 32GB in the existing server to 128GB in the new one.

Upon initial discussion in forums, some members expressed concerns about re-silvering time in the event of drive failure, with recommendations for mirroring over RAIDZ2 for better performance. However, I didn't delve into this matter until after stabilizing my existing server and securing components for the new one, along with upgrading my network infrastructure to accommodate 10GE.

Both write and read speeds are crucial, as this server is slated to become my primary one eventually. Re-silvering times also bear significance in the event of drive failure. Some forum users shared anecdotes about their 18-20TB drives taking anywhere from 12 to 18 hours to re-silver, while others reported duration of up to 5 days. As a novice, I seek guidance on optimizing performance while minimizing re-silvering times. Pool size and drive size are acknowledged as contributing factors, but I'm unsure of their exact impact. Balancing the need for more storage space, especially as my movie collection expands, is critical. Creating multiple pools with smaller drives is less feasible due to the higher cost per MB for 8-12TB drives compared to larger 18-22TB ones, and I am hesitant to use used drives.

As I approach the final and most costly purchase for my build, I seek valuable insights from the knowledgeable individuals in this forum. I deeply appreciate the input of the fantastic members, with special thanks to HoneyBadger, Constantin, joeschmuck, and Davvo.

 

[Constantin]

I have harassed the OP pretty relentlessly as part of this build process, so if anyone sees something objectionable, it's likely my fault.

As for resilvering, my pool has experienced multiple drive failures over the years and goharddrive has replaced or refunded every one as promised by their 5-year warranty. My pool consists of a single Z3 VDEV, so while I have multiple, pre-qualified cold spares to plug in at a moments notice, I don't lose a lot of sleep as resilvers are done in less than 24 hours, IIRC. My VDEV width (5 data drives, 3 parity), lack of content (less than 1/2 full), low pool fragmentation, and a sVDEV for very small files and metadata all help in this - see posts @jgreco and @sretalla here.

Between SMART, scrubs, & backups, I kind of consider the resilver issue somewhat overblown considering how infrequently the need usually arises. Yes, a pool rebuild can further stress the remaining drives into failure also, but there is also a balance to be struck between running lots of smaller drives, power consumption, space and heat, etc. If the case ventilation was done right, then drive temperatures should not significantly rise within the pool as a function of a scrub or resilver.

Ever-larger drives are being value-priced for and used in data centers for a reason, regardless of the resilver issue (which affects a data center also). Meanwhile, I am preparing for the inevitable by stocking up on replacement drives & by pre-qualifying them. At $78 for 10TB of He-filled happiness and a 5-year warranty, it makes sense to get the replacement drives ready for use as the pool drives approach 40k hours.

 

[joeschmuck]

While the WD Red Pro line should be CMR recording format, you should verify by using the Part Number of the drives you plan to purchase. Here is a link to the data sheet for the WD Red Pro line, ensure the model number matches.

https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-pro-hdd/product-brief-western-digital-wd-red-pro-hdd.pdf

As for your pool setup, I will not be able to provide you any really good advice here. I don't use mirrors. The following link I think explains everything very well. If this were me, and I did not have an external backup of my files, and the data I was storing was important to me, I would use 3-way mirrors (~36TB usable) if I were using six 20TB drives, but that severely limits the storage capacity. To change the scenario a little bit, if I also had a backup of my data on say an old server with new drives, even if it were a stripe, then I'd be fine with a RAIDZ2 (~72TB usable). But I would also humor a RAIDZ3 (~54TB usable) setup as well, depending on my storage requirements. And I would know that in any RAIDZx configuration, adding one more drive adds all of that capacity to the pool, but you need to determine that before you initialize that pool. So would I need a RAIDZ3 with eight 20TB drives (~90TB) or would seven 20TB drives (~72TB) all in a RAIDZ3 setup be good enough.

My friend, you have a decision to make.

@bent98 The speed of your NAS is not the mail issue, it is the capacity.

Okay, I'm jealous now. An Intel Scalable processor? I originally wanted a Scalable processor but I wanted a top of the line current model which was way out of my price range.

Heatsink compound, didn't see it listed and it is very important. I also didn't see if the heatsink comes with any or an already installed pad/layer. I did some research last month and my results were to use Prolimatech PRO-PK3-1,5G Nano Aluminum High-Grade Thermal Compound. Item # 9SIAGPF7KA1046 from Newegg. While it does not have the best thermal characteristics, only "Wet and Conductive" material holds that title, this was the best thermally conductive Non-conductive compound that didn't break the bank. I'm very impressed with it, so much better than Artic Silver 5, which use to be a very good product to use.

 

[joeschmuck]

@Constantin I have never looked at goharddrive but I have heard others say good things about it. $80 USD for a 10TB He drive with a 5 year warranty, that is a great value. What is "[Power Disabled]"? This will be the next place I purchase a hard drive from. My old primary server is becoming my backup server and will need three more hard drives in the future.

 

[bent98]

While the WD Red Pro line should be CMR recording format, you should verify by using the Part Number of the drives you plan to purchase. Here is a link to the data sheet for the WD Red Pro line, ensure the model number matches.

https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-red-pro-hdd/product-brief-western-digital-wd-red-pro-hdd.pdf

As for your pool setup, I will not be able to provide you any really good advice here. I don't use mirrors. The following link I think explains everything very well. If this were me, and I did not have an external backup of my files, and the data I was storing was important to me, I would use 3-way mirrors (~36TB usable) if I were using six 20TB drives, but that severely limits the storage capacity. To change the scenario a little bit, if I also had a backup of my data on say an old server with new drives, even if it were a stripe, then I'd be fine with a RAIDZ2 (~72TB usable). But I would also humor a RAIDZ3 (~54TB usable) setup as well, depending on my storage requirements. And I would know that in any RAIDZx configuration, adding one more drive adds all of that capacity to the pool, but you need to determine that before you initialize that pool. So would I need a RAIDZ3 with eight 20TB drives (~90TB) or would seven 20TB drives (~72TB) all in a RAIDZ3 setup be good enough.

My friend, you have a decision to make.

@bent98 The speed of your NAS is not the mail issue, it is the capacity.

Okay, I'm jealous now. An Intel Scalable processor? I originally wanted a Scalable processor but I wanted a top of the line current model which was way out of my price range.

Heatsink compound, didn't see it listed and it is very important. I also didn't see if the heatsink comes with any or an already installed pad/layer. I did some research last month and my results were to use Prolimatech PRO-PK3-1,5G Nano Aluminum High-Grade Thermal Compound. Item # 9SIAGPF7KA1046 from Newegg. While it does not have the best thermal characteristics, only "Wet and Conductive" material holds that title, this was the best thermally conductive Non-conductive compound that didn't break the bank. I'm very impressed with it, so much better than Artic Silver 5, which use to be a very good product to use.

Yes Joe, I have plenty of thermal paste when building my other PCs and it’s a bitch to spread.
:) View attachment 73997 View attachment 74002

 

[joeschmuck]

That is very cool looking. I like the very clean look, everything where is should be. That AIO with the digital temperature, very cool! But I'm not sure you have enough fans What is that thing that looks like an LCD display? At first I thought yo created a very professional looking label to state what is in the box, but after zooming in, it looks to be live data. I am curious about this build and while I personally would not want all the glowing lights, I know people who would love that stuff. Most of the kids working for me are into this stuff. Kids = 32 to 65 year olds. I'm 61 for the next month, then 62. Old as dirt! I was around when the salem which trials were going on, they missed my wife

I found the best way is to actually use the pea sized method. I use to spread the compound across the CPU surface and that was a pain, but after reading some information about the possibilities of leaving an air pocket in the gap, I have steered away from hand spreading it. But to each his own. I can't say my way is the only right way, I wouldn't do that without concrete proof.

Damn, 12 fans that I can see, and a possible 3 more on the top.

 

[bent98]

 

[bent98]

That is very cool looking. I like the very clean look, everything where is should be. That AIO with the digital temperature, very cool! But I'm not sure you have enough fans What is that thing that looks like an LCD display? At first I thought yo created a very professional looking label to state what is in the box, but after zooming in, it looks to be live data. I am curious about this build and while I personally would not want all the glowing lights, I know people who would love that stuff. Most of the kids working for me are into this stuff. Kids = 32 to 65 year olds. I'm 61 for the next month, then 62. Old as dirt! I was around when the salem which trials were going on, they missed my wife

I found the best way is to actually use the pea sized method. I use to spread the compound across the CPU surface and that was a pain, but after reading some information about the possibilities of leaving an air pocket in the gap, I have steered away from hand spreading it. But to each his own. I can't say my way is the only right way, I wouldn't do that without concrete proof.

Damn, 12 fans that I can see, and a possible 3 more on the top.

That is called a sensor panel. You run a program called AIDA64 and then you can make custom skins specific to your motherboard. It took me about 25-30 hours to make each one of the sensor panels. (Not sure why its rotating some of the screenshots)??

 

[Constantin]

What is "[Power Disabled]"? This will be the next place I purchase a hard drive from. My old primary server is becoming my backup server and will need three more hard drives in the future.

IIRC, that has to do with some drives having issues with 3.3V on a specific pin triggering a disable mode on some drives.

In response, wiring harnesses were developed to purposely prevent 3.3v reaching said sata power pins or whatever. Never was an issue with my rig so I cannot comment on all that. Those He10 drives just work and come shipped in a beautiful box w/o the need to shuck.

 

[bent98]

I just got the server up in running. I have a few questions.

1) When installing TrueNAS, it asked if I wanted to use a 16GB swap file on my OS SSD drive which is 1TB. I have 128 Gigs. Is that safe? What if the SSD drive gets corrupted?

2)Whats the best way to keep sync the data from my existing server, which is of to the one I just built for backup purposes?

3) How to I stress test all 6x of my 20TB drives before I copy all files over?
I already ran memtest86+ for 2 days with no errors so I am confident that once I test the drives, the system should be stable.

4) I also set up scrubs and smart tests on my old sever, I remember I followed ar guide to set it up. Not sure how to do that now as its been 6 years and cant find the guide


Thanks

 

[Constantin]

I just got the server up in running. I have a few questions.

1) When installing TrueNAS, it asked if I wanted to use a 16GB swap file on my OS SSD drive which is 1TB. I have 128 Gigs. Is that safe? What if the SSD drive gets corrupted?

2)Whats the best way to keep sync the data from my existing server, which is of to the one I just built for backup purposes?

3) How to I stress test all 6x of my 20TB drives before I copy all files over?
I already ran memtest86+ for 2 days with no errors so I am confident that once I test the drives, the system should be stable.

4) I also set up scrubs and smart tests on my old sever, I remember I followed ar guide to set it up. Not sure how to do that now as its been 6 years and cant find the guide


Thanks

1) with the amount of RAM you have, I doubt it will be an issue. I used the default setting.

2) I suggest pushing data from the primary to the backup server using zfs send. That can be set up using a replication task. See Tasks -> Replication Tasks, then set up a SSH connection between the two machines. To speed things along, use SSH+NetCat as your transport option. If you use push, make the primary machine the netcat active side = "Local"

3) Use badblocks and a tmux connection to test out all the drives. See this resource on how to do it.

4) This is also done under Tasks -> Scrub Tasks to set up scrubs and Tasks -> SMART Tests to set up SMART tests.

 

[bent98]

1) with the amount of RAM you have, I doubt it will be an issue. I used the default setting.

2) I suggest pushing data from the primary to the backup server using zfs send. That can be set up using a replication task. See Tasks -> Replication Tasks, then set up a SSH connection between the two machines. To speed things along, use SSH+NetCat as your transport option. If you use push, make the primary machine the netcat active side = "Local"

3) Use badblocks and a tmux connection to test out all the drives. See this resource on how to do it.

4) This is also done under Tasks -> Scrub Tasks to set up scrubs and Tasks -> SMART Tests to set up SMART tests.

Thanks! Question, Joe recommended solnet-array-test-v3 for the drive testing. Do one have an advantage over the other?

 

[Constantin]

Apologies, I do not know.

 

[Constantin]

I looked into solnet a little and the sense I get is that it does not destructively write to every block & read from it on a HDD like badblocks does, rather it does a lot of non-destructive testing, including on live pools, to see what is performing as expected, and what is not.

Based on the very little I have gleaned from @jgreco's description for solnet, his scripts will uncover issues with disks not running optimally, which is very helpful if a pool is not performing as well as one would like.

Unless @jgreco states otherwise, I'd start by qualifying all disks with badblocks + tmux. Once the drives are qualified and the pool built I'd transition to solnet to verify happiness. But it's entirely possible / likely that I misinterpreted @jgreco's description of solnet!

 

[bent98]

I looked into solnet a little and the sense I get is that it does not destructively write to every block & read from it on a HDD like badblocks does, rather it does a lot of non-destructive testing, including on live pools, to see what is performing as expected, and what is not.

Based on the very little I have gleaned from @jgreco's description for solnet, his scripts will uncover issues with disks not running optimally, which is very helpful if a pool is not performing as well as one would like.

Unless @jgreco states otherwise, I'd start by qualifying all disks with badblocks + tmux. Once the drives are qualified and the pool built I'd transition to solnet to verify happiness. But it's entirely possible / likely that I misinterpreted @jgreco's description of solnet!

Question, I just finished setting up pool and server configs.

The long smartctl takes 28 hours for each drive. how long will badblocks + tmux take to run? It seems i have to do it one at a time. Also, will it kill my server configs and pool? I copied over a script for smartctl from my other server.

 

[Constantin]

That’s the beauty of tmux, you can run all those sessions in parallel. It will take about 2 days for a 10TB drive in my limited experience.

So start a badblock on one drive, tmux, start the next one, tmux, etc.

 

[bent98]

1) with the amount of RAM you have, I doubt it will be an issue. I used the default setting.

2) I suggest pushing data from the primary to the backup server using zfs send. That can be set up using a replication task. See Tasks -> Replication Tasks, then set up a SSH connection between the two machines. To speed things along, use SSH+NetCat as your transport option. If you use push, make the primary machine the netcat active side = "Local"

3) Use badblocks and a tmux connection to test out all the drives. See this resource on how to do it.

4) This is also done under Tasks -> Scrub Tasks to set up scrubs and Tasks -> SMART Tests to set up SMART tests.

I am going on day 7 running badblocks, its 66% done. The only thing left to do after this is to copy my data over from one sever to another.


You suggest pushing data from the primary to the backup server using zfs send. That can be set up using a replication task. See Tasks -> Replication Tasks, then set up a SSH connection between the two machines. To speed things along, use SSH+NetCat as your transport option. If you use push, make the primary machine the netcat active side = "Local"

Its there a tutorial or recommend script I can steal in preparation of the copy?

 

[Constantin]

They have a fairly good resource on this, and a built in wizard, see here. BTW, what is your power consumption ATM? I’m intrigued re: power difference between a Xeon scalable vs. my D1537 + LSI HBA.

 

[bent98]

They have a fairly good resource on this, and a built in wizard, see here. BTW, what is your power consumption ATM? I’m intrigued re: power difference between a Xeon scalable vs. my D1537 + LSI HBA.

How would I know what my power consumption is ?

 

[chuck32]

That’s the beauty of tmux, you can run all those sessions in parallel. It will take about 2 days for a 10TB drive in my limited experience.

In my experience it should take around a week. It was ~ 5 days for 8 TB and that is consistent with the ~ 2.5 days for my 4 TB drives.

You're not quite done then, it's another round of smart tests then (although I just went for the long tests instead of short, conveyance and long) lately.

For the power consumption you would need some sort of smart plug / power meter to measure it. Word of advice, although it only happened to me once: when using a smart plug like a Shelly: do not put it in between your UPS and Server, put it in front of the UPS. I have one old and one new version and the new version (I have no other explanation) decided to turn off suddenly. I haven't had Autostart every 60 s (it will turn on when it was off for 60s) configured and restore power was set to last state instead of on. So my servers shutdown, but because the plug just killed power to the UPS it shut down gracefully.