First FreeNAS Server

[thatflashcat]

I am currently working on setting up a freenas server for the files of everyone in my household, which mostly consists of my video files, family photos and videos, and documents. In the future I intend to build a freenas system with newer hardware but for now I intend to use an existing system running an intel xeon w3530 and 12gb ddr3 ecc memory (I am not sure of the motherboard, just that it is from a dell t3500 workstation). This approach seeks to save some of the upfront cost while still using a capable system (12gb of ram should be plenty of space for a small system). My specific questions are as follows:

1. RAID configuration. Firstly, I have chosen WD red drives over seagate ironwolf ones. I was originally planning to do a raidz1 setup with three 3tb drives for a total usable space of 6tb, but my research indicates that this would be unwise and that raid z2 is safer. However, since raid z2 allows for 2 drives to fail, with a 4 drive setup it only allows for 50% usable storage, so I am thinking of either using 4 drives in raid 10 or 2 drives in raid 1 instead. The former option would give the benefit of higher read and write speeds (in the future I wish to upgrade my network past gigabit) and would consist of 4 * 4tb drives for a total cost of $480 and $30 per raw TB of storage. For the latter option I would purchase 2 * 6 tb drives for a total cost of $360 and $30 per raw TB of storage. (I'm still deciding how much space I need, taking into account that a ZFS pool reduces drastically in performance past 80-85% capacity) That aside, should I go for raid 01, 10, raid z2, or just raid 1?

2. Scrubs and SMART tests. The default scrub interval is 35 days and I have seen many different people with many different schedules. Would it be a wise idea to switch this to every 14 days and run smart tests every 14 days as well (offset from the scrubs of course)?

3. Data backups. I am well aware that RAID is not a backup though it is a useful tool in preventing data loss. At this point in time I do not have another freenas system that I can send snapshots to so another solution will be necessary. Would it be possible to use rsync to send a copy of a dataset to a windows machine using an rysnc program which then writes it to a removable drive that is then brought offsite? Backups are probably my weakest point here as I feel reasonably confident about the rest.

I want to setup freenas the right way so any input here is greatly appreciated. :)

 

[Chris Moore]

After reading through your post, I see that your terminology is still all about hardware RAID instead of ZFS. So I think it would help you to review some of the guides. Please take a look at these:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

I am not sure of the motherboard, just that it is from a dell t3500 workstation

Did you move that board over to another chassis or is it still in the Dell chassis? Nothing wrong with this, except the limited number of drive bays, but there are way around that.

However, since raid z2 allows for 2 drives to fail, with a 4 drive setup it only allows for 50% usable storage,

RAIDz2 is your safe option if you are going for bulk storage. That is better than some of the ideas you were throwing out and it can be made to work with the T3500 chassis if that is what you are working with.

Scrubs and SMART tests. The default scrub interval is 35 days and I have seen many different people with many different schedules. Would it be a wise idea to switch this to every 14 days and run smart tests every 14 days as well (offset from the scrubs of course)?

Short SMART tests usually take less than 5 minutes, some drives it is as little as 2 minutes, so I do that every day at 1 AM. I have my system send me a report by email at 7 AM and then I have it run a long SMART test at 8 AM on Monday, Wednesday and Friday. I may decrease the frequency of the long test to one a week, but it is my opinion that it is better to do that with some frequency as that is the test that is most likely to detect a drive fault before the drive fault can impact your data. The scrub is all about checking the data to ensure it is still healthy and if your pool (the collection of dives) is healthy and running on quality hardware, the data should be healthy. I have never had a scrub detect and correct an error because I am fanatical about replacing a disk as soon as it starts to misbehave. I run my scrub on a schedule that kicks it off on a weekday (Monday to Thursday) after 'threshold days' is greater than 21. That means it runs once every 3 weeks or, on rare occasions, it will be 4 weeks.
My pool only takes about 3 hours to complete a scrub, and I have the schedule set in such a way that the scrub starts after 8 AM (because I am away from home during the day) so it doesn't interfere with my use of the system. Systems I manage for work, I have the scrub scheduled to run over the weekend when nobody is there. The schedule is something you need to consider based on how you use the system. You can't run a short SMART test at the same time a long SMART is already running for example and if you run a scrub at the same time as a long SMART test, they interfere with one another and both will take longer to complete. It is sometimes difficult to figure the schedule out. The larger the drives, the longer the long SMART test takes. The more data in the pool, the longer the scrub takes, except that the scrub can be faster based on how quickly your drive configuration is able to read the data. My pool is able to read for a scrub at (if fluctuates) around 1000MB/s because I have two vdevs of 6 drives each. I picked that layout for the explicit purpose of having more speed because the previous layout I had was still 12 drives, but it was as a single vdev in RAIDz3. Wide vdevs are slower than narrow vdevs and each vdev behaves (in very rough terms) like a single drive.

I am well aware that RAID is not a backup though it is a useful tool in preventing data loss.

Mostly, it is about preventing down time due to a single drive failure and to allow for larger storage capacity than any single drive would allow. It isn't about preventing data loss. If you setup local snapshots, that might protect you from accidental deletions or corruption of the data by a virus, but that is a feature of ZFS, and when you just say RAID, you are talking about hardware RAID. There is a good reason why ZFS uses different terminology, it is to differentiate from the other thing.

Would it be possible to use rsync to send a copy of a dataset to a windows machine using an rysnc program which then writes it to a removable drive that is then brought offsite? Backups are probably my weakest point here as I feel reasonably confident about the rest.

If you are going to use a Windows system to make the backup, you might want to use a Windows tool that can run on the Windows system and pull the data from the network drive to a local drive to keep the local drive in sync with the network drive.
SyncToy is made by Microsoft, here it the link: https://www.microsoft.com/en-us/download/details.aspx?id=15155
It can be scheduled to run through the Windows task scheduler. If you are using Windows 10, there is an extra step required which is detailed here: https://stackoverflow.com/questions/43844750/schedule-sync-in-synctoy-2-1-windows-10

I asked some questions, if you will get back to me on those, we can have a little discussion. I used a pair of Dell Precision T3500 workstations for my home NAS solution for about 2 years, so I have some experience with that, including connecting external enclosures to add additional disks for more storage than would fit internally.

 

[thatflashcat]

First, let me thank you for writing this for me!

After reading through your post, I see that your terminology is still all about hardware RAID instead of ZFS. So I think it would help you to review some of the guides. Please take a look at these:

I guess old habits die hard. :p I've always been used to the terms raid 1 and 0 vs mirrored and striped vdevs, respectively.

Did you move that board over to another chassis or is it still in the Dell chassis? Nothing wrong with this, except the limited number of drive bays, but there are way around that.

I'm still using the dell chassis but would like to switch to a better case (perhaps one like this) for better cooling though I would need a new power supply since the stock one has very short cables. At the very least I will get a new case when I replace the rest of the hardware.

RAIDz2 is your safe option if you are going for bulk storage. That is better than some of the ideas you were throwing out and it can be made to work with the T3500 chassis if that is what you are working with.

Does raidz2 suffer a resilver speed penalty compared to mirrored and stripped vdevs? Also, using correct terminology this time ;), I'm still trying to decide between 4x4tb drives in raid z2 vs 2*6tb mirrored vdevs.

Short SMART tests usually take less than 5 minutes, some drives it is as little as 2 minutes, so I do that every day at 1 AM.

You can't run a short SMART test at the same time a long SMART is already running for example and if you run a scrub at the same time as a long SMART test, they interfere with one another and both will take longer to complete.

I'll probably run short SMART tests every night then and run less frequent SMART long tests and scrubs. Will anything "bad" happen if a scrub and SMART long test overlap, aside from increasing the time they both take?

Mostly, it is about preventing down time due to a single drive failure and to allow for larger storage capacity than any single drive would allow. It isn't about preventing data loss.

I meant that if a drive fails, it is possible to rebuild the data from the remaining drives, so it provides *some* data loss protection in a sense.

If you are going to use a Windows system to make the backup, you might want to use a Windows tool that can run on the Windows system and pull the data from the network drive to a local drive to keep the local drive in sync with the network drive.

Is there any downside from a reliability standpoint in using windows for the data backups? I know that usb drives can be problematic at times, at least when used directly with freenas.

 

[Chris Moore]

Does raidz2 suffer a resilver speed penalty compared to mirrored and stripped vdevs?

With quality drives, you won't be resilvering often enough to be concerned with that. Last year when I had both of my servers filled with drives that were over five years old and I was replacing a drive a month, it was still only a couple hours out of an entire month and with proper monitoring and just a little luck, you get to choose the time of day that you do a resilver. When resilvering, you have a very intense operation going on that takes IO away from anything else that is going on. That would be true of ANY array configuration. If you only have two drives (a single mirror) and you have to replace one, it is going to impact your overall IO capacity. If you want a high performance system, you need more vdevs. That is easier to do with mirrors as each mirrored pair is a vdev. I have one of my pools that is made up of eight mirrors and it has almost as much perfomance as the SSD my desktop boots from.
My iSCSI pool:
The SSD in my desktop:
You just are not going to get as much performance out of only a couple of drives.

I'm still trying to decide between 4x4tb drives in raid z2 vs 2*6tb mirrored vdevs.

I would never consider a 4 drive pool in RAIDz2, you can, but why. Get more drives and have five or even six drives in RAIDz2. It can't be that the drives are so expensive.
This is the kind of drives I am using in my pool right now. They work great and are only $73 with free shipping:
https://www.ebay.com/itm/Seagate-4TB-SATA-5900RPM-3-5-ST4000DM000-Replacement-HDD/173420135370
or
https://www.ebay.com/itm/Seagate-De...rnal-5900RPM-3-5-ST4000DM000-HDD/132731310144
or:
https://www.ebay.com/itm/Seagate-De...-SATA-Hard-Drive-HDD-ST4000DM000/232882349381

Will anything "bad" happen if a scrub and SMART long test overlap, aside from increasing the time they both take?

It does slow them both down because they are competing for IO time but that is about the end of it.

I meant that if a drive fails, it is possible to rebuild the data from the remaining drives, so it provides *some* data loss protection in a sense.

I understand what you are saying. That is the reason why many people think of RAID as being a backup because it provides a layer of protection. That is a protection against the down time that it would take to restore from a backup. In RAIDz2, you could loose two drives before data loss, where in RAIDz (RAIDz1) you can only loose ONE drive. My problems with mirrors (and important data) is that you still only have ONE drive failed before you loose your layer of protection. For my important data, I always use RAIDz2 because it takes two drive failures to cost me my data and I have had two drives fail at the same time in a RAIDz2 pool at home, but at work I have had three fail in the same pool at the same time and I was only saved by the fact of having a hot-spare.

It takes a little work to make this fit, but you can use this adapter to fit 3 hard drives in the 5.25 inch bay:
https://www.ebay.com/itm/EverCool-Dual-5-25-in-Drive-Bay-HDD-Cooling-Box/202373555072

If I recall correctly, there is a 2.5 inch bay below the 5.25 inch bays and there are two side mount 3.5 bays. All together, there is room for six drives plus a boot drive in this chassis, you just need a couple more ports for the data drives to run on because there are only five internal SATA ports. You can use a SAS controller like this:
https://www.ebay.com/itm/LSI-SAS-92...ternal-HBA-Both-Brackets-IT-MODE/152937435505
I used a similar model in my T3500 to connect all the data drives and just use the SATA ports for the boot drive. A single small capacity SSD works great for that.
To connect the SAS HBA to the data drives, you will need forward breakout cables like this:
https://www.ebay.com/itm/Lot-of-2-M...-Forward-Breakout-Internal-Cable/371681252206
Investing in the SAS controller is a good idea, even if you move to a larger chassis later because you can connect (if I recall correctly) 256 drives to that controller using SAS expanders like this to branch the signal:
https://www.ebay.com/itm/Intel-RES2...-SATA-PCI-Express-x4-with-cables/332630203208

 

[thatflashcat]

First, let me apologize for the delay in replying. I've been doing a bunch of other things the past few days and neglected to reply. :P

With quality drives, you won't be resilvering often enough to be concerned with that.

Ok I won't worry about that then. Also, much of the time the only thing I'll be accessing are school documents so good IO performance is hardly necessary for that.

I would never consider a 4 drive pool in RAIDz2, you can, but why. Get more drives and have five or even six drives in RAIDz2. It can't be that the drives are so expensive.

I am certainly trying to minimize the cost but aside from that I have no need for a storage array greater than 8tb (usable). 4tb are also a much better deal than 3tb drives (I'm planning on WD Red; I'm not at all inclined to buy used drives off ebay), so I wouldn't do a 6 * 3tb instead, for example. I certainly want to be future proof but my current storage use really isn't that extreme.

If I recall correctly, there is a 2.5 inch bay below the 5.25 inch bays and there are two side mount 3.5 bays

Yes, only the bay beneath the 5.25 inch ones is a 3.5 inch. I think I may just upgrade the case immediately to make the drive installations easier.

Investing in the SAS controller is a good idea, even if you move to a larger chassis later because you can connect (if I recall correctly) 256 drives to that controller using SAS expanders

I'll definitely keep that in mind. Is the SAS controller you mentioned likely to have compatibility issues with any future hardware or will it generally work with anything?

 

[Chris Moore]

I'll definitely keep that in mind. Is the SAS controller you mentioned likely to have compatibility issues with any future hardware or will it generally work with anything?

You might run into some old or low-end hardware that it wouldn't work with, but I have not had a compatibility issue with the SAS controllers I have used.

Yes, only the bay beneath the 5.25 inch ones is a 3.5 inch.

That was a typo. If I recall correctly, there is room for two 3.5" drives in just below the two 5.25" bays. I had six drives in my T3500 when I was using it and I added four more in an external enclosure. I switched to a different case to get better cooling for the drives, but you will have a very difficult time getting the T3500 system board mounted in a different chassis. Difficulty factor, it would be an easier thing to just buy a different system board.

I just suggested something like this to someone else a couple weeks ago, but I couldn't find the post to point you at. Here is what I would suggest for a tower build that you want quiet and inexpensive while still capable.

CASE: Fractal-Design-Define-R5-FD-CA-DEF-R5-BK-Black-Silent-ATX-Midtower-Computer-Case - US $99.99
https://www.ebay.com/itm/263674323506

POWER: Corsair-Certified-CS-M-Series-CS650M-650W-80-Plus-Gold-Active-PFC-Modular-Power - US $64.99
https://www.ebay.com/itm/382130407495

System Board: Super Micro X9SCM-F Motherboard w/ Heatsink/Fan & I/O Shield - US $75.00
https://www.ebay.com/itm/273400083962

OR different heatsync

System Board: SuperMicro X9SCM-F Intel Xeon E3 Intel C204 Chipset Micro ATX Server Motherboard - US $79.99
https://www.ebay.com/itm/192561781616

CPU: Intel SR00H Xeon E3-1230 3.20GHZ Socket 1155 - - US $55.09
https://www.ebay.com/itm/273313040616

OR faster / newer

CPU: Intel Xeon E3-1230V2 3.30GHz Quad-Core CPU Processor SR0P4 LGA1155 - C737 - US $94.95
https://www.ebay.com/itm/263869242230

Memory: 8GB Memory RAM for SuperMicro X9 Series - - US $82.00 * 2 = $164
https://www.ebay.com/itm/163130855012

Drive Controller: LSI-SAS-9211-8i-8-port-6Gb-s-PCI-E-Internal-HBA-Both-Brackets-IT-MODE - US $59.99
https://www.ebay.com/itm/152937435505

Drive Cables: Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable - - US $12.99
https://www.ebay.com/itm/371681252206

Thermal Compound: Noctua NT-H1 Thermal Paste Grease Conductive Compound for CPU/GPU - US $6.95
https://www.ebay.com/itm/302624513215

I may have missed some accessorie, and I didn't include drives, but this should get you all the key components

I just picked these on eBay and they are all available right now.

 

[Chris Moore]

PS.

4tb are also a much better deal than 3tb drives

I would never suggest 3TB drives; they make me nauseous. I don't like them, and it doesn't matter who the manufacturer is.
However, you should not hate on the eBay drives. You can get some really good bargains that are quality equipment.
Take a look at this:
https://www.ebay.com/itm/HP-Seagate...-3-5-SATA-6-0GB-s-Enterprise-HDD/153070795097

These are enterprise class Seagate drives that have never even been powered on. I would buy some myself if I were in the market to buy drives.

 

[thatflashcat]

Here is what I would suggest for a tower build that you want quiet and inexpensive while still capable.

That is tempting, though I'd probably get a horizontal server case like the one I linked above over a vertical desktop one. Not that I have anything against fractal design cases (I use a meshify c for my gaming system/workstation) but a horizontal case would work better. I'm always a little nervous about buying used parts but it's just the cpu and hard drive controller. As long as they continue to work reliably it's not a big deal. Now that I've thought about it those drives *do* look nice with 128mb of cache and they are specifically NAS drives, and at that price I'd get 5 and have a 12tb usable pool (9.6 tb leaving 80% empty for ZFS of course).

Edit: The new drives you linked in your last post just went up to $120 each. :(

 

[Chris Moore]

Edit: The new drives you linked in your last post just went up to $120 each. :(

These are just as good:
https://www.ebay.com/itm/THGNN-DELL...SATA-HDD-W-TRAY-ST4000NM0033-REF/252579021323

 

[Chris Moore]

If you are looking for a inexpensive solution for the chassis, I bought two of these and I think it is a good deal:

https://www.ebay.com/itm/221693812159

The fans run full blast, so it is loud, but you can slow them down by using this product between the fan and the fan header:

https://www.ebay.com/itm/263253480854

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[thatflashcat]

So sorry for not replying! I've been on vacation and haven't done anything with my NAS plans over that period. I think I'm good with the chassis I found on amazon since hotswap isn't a huge priority.

About the used parts, in your experience do used processors and drive controllers have good longevity?

Also, how do these drives compare? I'm always slightly nervous about purchasing refurbished hardware but the refurbished drives you linked above do look better than the ones I just linked. 64mb cache is a bit small for a 4tb drive and there is no manufactor listed on the drive. I've never looked at older hardware on ebay much so I'm not sure what to look for.

 

[Chris Moore]

About the used parts, in your experience do used processors and drive controllers have good longevity?

Out of all the used components I have had over the years, which is a good quantity because I have been doing almost exclusively used gear at home since sometime around 2003, I have had very few failures and only one in recent history and that one may well have been my fault for letting it get too hot. I don't worry about buying used. It is kind of like buying a used car instead of buying a new car. You can have problems with either of them but some people still insist on buying new. It is a choice.

Also, how do these drives compare? I'm always slightly nervous about purchasing refurbished hardware but the refurbished drives you linked above do look better than the ones I just linked.

The drives you linked "White Label" are very suspicious to me. They look like Western Digital, but some of the users here have tried them and they are not as reliable. @danb35 bought six of them and, if I recall correctly, he said that he got two of them replaced under warranty, but ultimately they all failed and quicker than he would have expected. I would buy a low-hours pull before buying the White Label drives.
The thing about the Dell and HP drives I linked to before, they are the Seagate Constellation "Enterprise" grade server drives that will certainly last longer than the White Label drives and the Dell ones are a little bit less expensive and come with a 1 year warranty from the seller.

https://www.ebay.com/itm/THGNN-DELL...SATA-HDD-W-TRAY-ST4000NM0033-REF/252579021323

 

[danb35]

@danb35 bought six of them and, if I recall correctly, he said that he got two of them replaced under warranty

Out of six disks, eight have failed. Six of the eight were replaced under warranty, though.

 

[Chris Moore]

Out of six disks, eight have failed. Six of the eight were replaced under warranty, though.

Worse than I thought. Are any of them still running at this point?

 

[danb35]

Worse than I thought. Are any of them still running at this point?

That vdev now has two of the "white label" disks, one WD, and three Seagates, IIRC. Two of the white label disks were in a test machine, where one of them has failed.

 

[thatflashcat]

You can have problems with either of them but some people still insist on buying new. It is a choice.

Excellent thoughts! I definitely feel better about used hardware as long as it's high quality.

The drives you linked "White Label" are very suspicious to me.

That was my thought as well. I'll definitely go for low-hours pull drives instead.

Out of six disks, eight have failed. Six of the eight were replaced under warranty, though.

I'll definitely skip those drives!

Also, I neglected to mention that I have an old HP server that isn't working correctly but may still be usable. I'm almost positive the only problematic part is the power supply (I originally had given up on it but after seeing the availability of used parts I may be able to repair it at a reasonable cost.) It's an HP Proliant DL380 G7 with dual 6 core xeon processors (I forget the model) and a whopping 144gb of RAM (9 * 8 gb per processor). The issue I have is it won't turn on immediately when it's been unplugged and plugged in again unless I let it sit for maybe 15 minutes, and it also tends to turn itself off and on again every few seconds repeatedly. This seems like an issue with the power supplies as it ran flawlessly when I managed to get it running. I also tried swapping the power supplies or using just one and nothing helped.

I found this as a possible replacement but the backplane may also need replacement. I would also need a larger case as the stock one only supports 2.5 inch drives. Another alternative would be some sort of external drive enclosure but I believe they tend to be expensive. Any thoughts on this?

 

[Chris Moore]

It's an HP Proliant DL380 G7 with dual 6 core xeon processors (I forget the model) and a whopping 144gb of RAM (9 * 8 gb per processor).

It might be useful to know the model of the processor, but it is an older system that will generate more heat and probably make more noise than a newer system would.

The issue I have is it won't turn on immediately when it's been unplugged and plugged in again unless I let it sit for maybe 15 minutes, and it also tends to turn itself off and on again every few seconds repeatedly. This seems like an issue with the power supplies as it ran flawlessly when I managed to get it running. I also tried swapping the power supplies or using just one and nothing helped.

If you are having trouble with it, you might want to scrap it. With the problem description you gave and the age of the system, I would start by looking closely at the system board for swollen capacitors. If you have bad capacitors on the system board, and the 2.5" drives, you probably don't want to spend the money trying to fix this. While it could be the power supplies, I hate to suggest you replace them and the problem continues.

 

[thatflashcat]

It might be useful to know the model of the processor, but it is an older system that will generate more heat and probably make more noise than a newer system would.

Intel Xeon x5650. Rather old but still powerful enough for FreeNAS though it would definitely run hotter than a newer processor. Noise isn't a huge deal since this will be away from the rest of the house.

With the problem description you gave and the age of the system, I would start by looking closely at the system board for swollen capacitors.

All the capacitors or are there particular areas of the board that are most likely to be problematic?

 

[jgreco]

Also, I neglected to mention that I have an old HP server that isn't working correctly but may still be usable. I'm almost positive the only problematic part is the power supply (I originally had given up on it but after seeing the availability of used parts I may be able to repair it at a reasonable cost.) It's an HP Proliant DL380 G7 with dual 6 core xeon processors (I forget the model) and a whopping 144gb of RAM (9 * 8 gb per processor). The issue I have is it won't turn on immediately when it's been unplugged and plugged in again unless I let it sit for maybe 15 minutes, and it also tends to turn itself off and on again every few seconds repeatedly.

Check what the power on delay is set to. Some of these can be set pretty high, like 10 minutes.

 

[Chris Moore]

All the capacitors or are there particular areas of the board that are most likely to be problematic?

I pulled up a photo of the system board and there are not many capacitors on it. I would look at them all closely but it isn't a guarantee.

I didn't think about the delay that @jgreco mentioned, do look in the BIOS for that.

The power supplies are only about $22, and the system board looks to be available for $48, so it might be worth fixing if you don't mind the extra power consumption and heat.