Will my first build FreeNAS?

[Graf_Euler]

Hi everyone,

I'm having problems with my current "storage solution", which is a cheap raid5 chassis with 4x 3TB WD Red HDDs. During PrimeDay I bought some new blurays but copying them onto the storage kind of damaged the file system. After what I've read online, I may have hit an unresolved bug in the ext4 file system, but anyway. Since this is not the first time I have problems with this solution (I already lost all my data once, but the important data was backed up), I decided it's time to move on and get a proper system up and running. Actually a system I have control over the disks and can monitor them.

After reading a lot about possible solutions I found FreeNAS with zfs a very nice idea and I want to go for it. It seams pretty robust and offers a lot of possible configurations. (And finally, after several years of Linux servers, BSD will be back at home ;) ). The main use cases I'm aiming at are:

• Storage for my BluRay Collection (I do have a separate machine for playback. I would simply mount the NAS share on that machine)
• Nextcloud (~3 active users) for Contacts, Files, Pictures, Calendar

My proposal for hardware:

• Mainboard: Supermicro X11SSM-F Intel C236 So.1151
• CPU: Intel Core i3 7100 2x 3.90GHz So.1151 BOX
• RAM: 1-2x 16GB Samsung M391A2K43BB1-CTD DDR4-2666 ECC DIMM CL19 Single
• HDDs:
  • Data: 8-10x 4000GB WD Red WD40EFRX 64MB 3.5" (8.9cm) SATA 6Gb/s
  • OS: 60GB Patriot Flare 2.5" (6.4cm) SATA 6Gb/s MLC NAND (PFL60GS25SSDR)
• Power supply: 650 Watt Seasonic FOCUS Plus Modular 80+ Platinum
• Chassis: Fractal Design Node 804 or Fractal Design Define R5
• Sata extention: InLine 76617E 4 Port PCIe 2.0 x1

Now, there are some open questions:

1. Chassis: I would love to get rack mount. I searched on amazon, ebay (maybe with wrong keywords?) and only found very expensive ones ~500+ Euro. Reading older forum posts, I read about 2U 12bay chassis for $185, but with dead ebay links. If I can't find a rack mount I would probably go for the Node 804.
2. The mainboard + CPU: I checked in the Community Hardware Guide, but couldn't find a shop selling the Intel Core i3-6300, thus I checked for the above mentioned CPU. As an alternative I could also use the Intel Pentium Dual-Core G4600 SR35F 3.6GHz LGA1151 which is a bit cheaper and probably also fast enough for my usage.
3. Still unsure if I should add an SSD as L2ARC. I guess only Nextcloud could benefit from it. On the other hand, the "often used data" from Nextcloud is relatively small, maybe 32GB ram are already okay to start with?
4. Sata extension: I found a guide on building a cheap 10GBe FreeNAS using the Supermicro X10SDV-4C+-TLN4F board. The guy used this Addon-Card for additional sata ports and I assumed it may also work on my hardware. I'll be needing an extension at least for the OS SSD and also for the L2ARC.
5. Configuration: RaidZ-2 seems to be the way to go for me looking at the amount of drives. I'm currently unsure about buying the required HDDs. I only have 3 of them right now at home and still need to buy the remaining ones. It would be wise to buy, for example, two on eBay, two on amazon, etc...

What do you think about this setup, does it make sense? Did I maybe miss something?

 

[MarioW]

Just a comment:

I am going to watch this thread. I want to build something similar. My current virtual shopping cart:
- Same Mainboard Supermicro X11SSM (so far I haven't noticed the need for a bigger supermicro mainboard).
- Intel Xeon E3-1220 v6, 4x 3.00GHz
- slightly different RAM: 1x16GB Samsung M391A2K43BB1 2400 Mhz (maybe doubling it in future) (The Xeon only supports 2400 Mhz, but for some reasons, your 2666 MHz RAM seems to be even more cheaper?)

Propably 1 small SATA SSD for FreeNAS, and 2 (later 4) Seagate Ironwolf HDDs (not yet choosen).

My main target is a reliable and fast NAS storage, especially a media server / e.g. PLEX , but I also want to be able to run 1-2 VMs in parallel.

 

[pschatz100]

For your use case, I would not bother with the L2ARC. If you have concerns about performance, go with 32Gb RAM. Or, you could build the system with 16Gb RAM and monitor memory usage for a while. If you find yourself maxing out the RAM, you can always add more.

That motherboard has 8 sata ports, so you are good to go if you build the system with 8 drives. If you decide you need more, then go with a proper HBA. They are not much more expensive than a PCIe extender and are a lot more reliable. A search on the forums will turn up suggestions for good HBA's as well as instructions for configuring them.

For the OS SSD, you can try using a USB to SATA adapter. I have not done this myself, but a couple of posters here on the forum have had success with this setup.

Pay attention to cooling. You are going to have a lot of disks in the case.

 

[GBillR]

Chassis: I would love to get rack mount. I searched on amazon, ebay (maybe with wrong keywords?) and only found very expensive ones ~500+ Euro. Reading older forum posts, I read about 2U 12bay chassis for $185, but with dead ebay links. If I can't find a rack mount I would probably go for the Node 804.

I have used this chassis in a build... not too bad really. The power supply location does make it a little challenging to use all 8 HD slots, but not impossible. I also have several R5 builds, and love that chassis... if you have the space.

Still unsure if I should add an SSD as L2ARC. I guess only Nextcloud could benefit from it. On the other hand, the "often used data" from Nextcloud is relatively small, maybe 32GB ram are already okay to start with?

I personally do not think you would see a significant benefit from this in your use case. If you feel you need it after you get it up and running, you could add it later.

Sata extension: I found a guide on building a cheap 10GBe FreeNAS using the Supermicro X10SDV-4C+-TLN4F board. The guy used this Addon-Card for additional sata ports and I assumed it may also work on my hardware. I'll be needing an extension at least for the OS SSD and also for the L2ARC.

I think this is a bad idea. Last I looked, SATA multipliers or extensions are not recommended. Personally, I would recommend picking up a cheap SAS HBA from ebay or a reputable supplier. There are plenty of recommended HBAs that work very well with FreeNAS. This would give you the extra ports you need to support 8+ HD (you can use it with the onboard SATA as well, and will be a much better and supported solution.

RaidZ-2 seems to be the way to go for me looking at the amount of drives. I'm currently unsure about buying the required HDDs. I only have 3 of them right now at home and still need to buy the remaining ones. It would be wise to buy, for example, two on eBay, two on amazon, etc...
What do you think about this setup, does it make sense?

For ease of expansion, in your case mirrors would probably make sense. You can start with a 2-disk mirror and just add more 2-disk mirrors as you acquire additional disks. You'll see the best read speeds from mirrors. Sourcing from multiple suppliers is okay, but probably overkill today. Just be sure to test the drives fully (see the resource section) prior to use and you should be okay.

Did I maybe miss something?

Be sure your power supply has enough SATA connections to support your intended setup. SATA power splitters are risky at best (search google), and not recommended unless completely unavoidable. Oh.. and those FOCUS supplies have a fairly large cap/filter under heat shrink wrapped up in the mainboard power cable that makes it somewhat rigind and difficult in tight builds. I would recommend one of the other seasonic supplies.

Good luck!

 

[Newfoundland.Republic]

I would check on eBay for a server. Many of us have used "retired" servers successfully.

 

[Graf_Euler]

Thank you very much for the answers!

Regarding the cache and memory, I will go with the 32GB RAM and skip the SSD as L2ARC for now.

Cooling is also a big topic, but since I'm not 100% clear on the chassis right now, I will postpone this a little further.

The hint with the USB to SATA adapter is quite nice. If I stick with the 8 hdds, without the cache, the adapter would be the best solution to avoid an HBA right now.

I think this is a bad idea. Last I looked, SATA multipliers or extensions are not recommended. Personally, I would recommend picking up a cheap SAS HBA from ebay or a reputable supplier. There are plenty of recommended HBAs that work very well with FreeNAS. This would give you the extra ports you need to support 8+ HD (you can use it with the onboard SATA as well, and will be a much better and supported solution.

I'll have a look at the recommended HBAs. I simply choose this controller card because of the success story I read, but I agree. It may be a better solution to get a proper SAS HBA.

For ease of expansion, in your case mirrors would probably make sense. You can start with a 2-disk mirror and just add more 2-disk mirrors as you acquire additional disks. You'll see the best read speeds from mirrors. Sourcing from multiple suppliers is okay, but probably overkill today. Just be sure to test the drives fully (see the resource section) prior to use and you should be okay.

This is one of my main topics right now. I started thinking about the "old" 3TB disk from my current setup. I need to check the SMART stats, but they're not that old and I could include them in the new system. I could create two Z1 pools, one using the 4x 3TB WD Red (actually, its only 3x 3TB and 1x 4TB. I already started upgrading the old raid5 system) from my old setup and a second pool with 4x 4TB WD Red. This would make it easier to further extend the system. I could either replace the 3TB discs with bigger discs or buy 4 additional discs and add a third Z1 pool.
Does this make sense? I just installed FreeNAS in a virtual machine to start playing around and get familiar with the configurations. But I just started...

Be sure your power supply has enough SATA connections to support your intended setup. SATA power splitters are risky at best (search google), and not recommended unless completely unavoidable. Oh.. and those FOCUS supplies have a fairly large cap/filter under heat shrink wrapped up in the mainboard power cable that makes it somewhat rigind and difficult in tight builds. I would recommend one of the other seasonic supplies.

Good hint, thanks! I found a new one already:
650 Watt Seasonic Prime Ultra Modular 80+ Gold
If I read it correctly, I could connect 6 SATA and use 3 adapters from 5.25" to SATA, which gives me 9 connectors.

 

[Graf_Euler]

A small update:

Today I found a chassis that fits to my needs: Logic Case 4U 15-disks

For cooling, it has 4 fans in the front and 2 rear fans. I'll go with the preinstalled fans and check the noise level and temperatures. If they don't perform well enough I'll upgrade them to Noctua NF-A8 PWM.

Now I just need to decide about the pool layout/configuration. The more I read in the forum, the less I'm sure about the way to go. I tested some configurations in my VM and all of them worked fine for me.
1) Mixed-Mode-Max-Space: Start with 3x 3TB + 5x 4TB. I create two vdevs with 4 disks each using raid-z1 (3-3-3-4 + 4-4-4-4), which will result (theoretically) in 9 + 12 = 21TB pool space.
2) Mixed-Mode: Again, 3x 3TB + 5x 4TB. Create a single vdev including all eight disks using raid-z2. This would result in 18TB pool space.
3) Same-Sized-Disks: 8x 4TB. In this setup I would again create a single vdev using raid-z2, but now I would have a pool size of 24TB.

I like the idea of 1), because it gives me the flexibility to later extend the system. I do have options to replace the 3TB disks with either 4TB disks or even bigger ones to get a larger pool. Also, I could add a third vdev using four new disks. I saw other people talking about such setups here in the forum and it is always mentioned that a second disk failure can only be tolerated if the second disk belongs to the other vdev. It's kind of a 3 in 7 chance to complety loose all the data, in contrast to 2). Option 3) is not really an option for me, since I don't need that extra space right away and I would prefer 2).
Anyone any additional thoughts on that?

 

[pschatz100]

When the disks are larger than 2TB, RaidZ1 is not recommended. You should use RaidZ2.

You have never said how much storage capacity you need. Do you need more than 18TB space?

Another reasonable option could be one pool of 6 disks in RaidZ2 (3-4-4-4-4-4) for your storage, which would give you 12 TB capacity, plus another pool of 3-3 mirrored that you could use for Nextcloud storage. You would have a total raw capacity of 15TB, and if you replace the one 3TB disk in the 6 disk array the capacity of that pool would expand to 16TB.

 

[Graf_Euler]

When the disks are larger than 2TB, RaidZ1 is not recommended. You should use RaidZ2.

Thanks for that hint, didn't know that! Just for understanding: Do you know the rational behind that recommendation?

You have never said how much storage capacity you need. Do you need more than 18TB space?

Another reasonable option could be one pool of 6 disks in RaidZ2 (3-4-4-4-4-4) for your storage, which would give you 12 TB capacity, plus another pool of 3-3 mirrored that you could use for Nextcloud storage. You would have a total raw capacity of 15TB, and if you replace the one 3TB disk in the 6 disk array the capacity of that pool would expand to 16TB.

oh, you're right.. Currently I have around 9-10TB of data, so I'm looking for a single pool size of at least 18TB right now. Since I should always have 20% of free space on the file system, this gives me ~15TB of usable space. The Nextcloud data is currently < 500GB.

I like the idea of the splittet pools for Nextcloud and the other data. The Nextcloud data is used quite frequently, because pictures taken from the smartphones are automatically uploaded, whereas the other pool isn't used that often.

 

[Newfoundland.Republic]

@Graf_Euler I am looking forward to the response on the RAIDZ1/RAIDZ2 myself. My thought is that despite FreeNAS being, well, free, it can be used for an enterprise solution. In the enterprise you want to reasonably reduce your risk and maximise your availability. You are willing to pay the cost for the risk reduction and maximising availability. (e.g., I had many "discussions" with clients who could not understand why IT had to pay $1,000 for a hard drive when they could drop down to Best Buy and get the same size for $150).

I used RAIDZ2 for my home implementation because I wanted the additional availability - I could lose a couple of drives and still function. I have the drives and drive bays available. Previous to this with my old 2-bay Synology and Qnap NASes I was using mirrors and was fine with it.

But always remember: RAID is NOT backup!

 

[Graf_Euler]

@Graf_Euler I am looking forward to the response on the RAIDZ1/RAIDZ2 myself. My thought is that despite FreeNAS being, well, free, it can be used for an enterprise solution. In the enterprise you want to reasonably reduce your risk and maximise your availability. You are willing to pay the cost for the risk reduction and maximising availability. (e.g., I had many "discussions" with clients who could not understand why IT had to pay $1,000 for a hard drive when they could drop down to Best Buy and get the same size for $150).

Indeed, I could imagine this. I was just wondering, why the single drive capacity is the factor for the decision. I can easily understand that it makes more sense using RAIDZ2 than RAIDZ1 on a system with, lets say 8 disks independent of their size. For more disks its more likely that a second disk would also fail, after the first one failed.

I will most likely also use RAIDZ2 on my setup, since I'm striving towards a 10 disks solution. Use a mirror for the Jail(s) and maybe 8 disks for the main storage pool. In this setup it will be very hard to later upgrade to more disc space, so I guess its worth investing a bit more in the beginning.

 

[CraigD]

If you do get the 15 bay case, it will hold 16 drive if you add the drive hanger like the one in my signature

You could start with an 8 wide RAIDz2 vdev, then add another at a later date without a rebuild

Have Fun

 

[Constantin]

For your use case, I would not bother with the L2ARC. If you have concerns about performance, go with 32Gb RAM. Or, you could build the system with 16Gb RAM and monitor memory usage for a while. If you find yourself maxing out the RAM, you can always add more.

Don't discount L2ARC out of hand. I found it did wonders when set to metadata only mode to speed up disk directory browsing (and rsync performance. See here.).

 

[Constantin]

A small update:

Today I found a chassis that fits to my needs: Logic Case 4U 15-disks

For cooling, it has 4 fans in the front and 2 rear fans. I'll go with the preinstalled fans and check the noise level and temperatures. If they don't perform well enough I'll upgrade them to Noctua NF-A8 PWM.
...
Anyone any additional thoughts on that?

If that is the case is the rebranded version of a Norco 431 like it looks to be, have a look at my build report before buying it. The Norco 431 is a great disk drive cooker, SATA connector destroyer, and generally breaks stuff by trying to fit 10lb into a 5lb bag. There is nothing but pain associated with the Norco-431, especially since cable management is non-existent (as are cooling channels for the hard drives).

I have found the Lian Li series to be decent (A75) or at a lower price point the Sharkoon T9 Value ATX or eqv. The Lian Li will benefit from SATA backplanes, the Sharkoon will need at least two of those 5 x 3.5" in a 3-bay 5.25" enclosure docks - ideally with a fan and a SATA backplane.

I'd also consider using fewer, but larger disks, i.e. starting off with six 8TB disks in a Z2 and then adding another like VDEV in the future once that space runs out.

 

[pschatz100]

When the disks are larger than 2TB, RaidZ1 is not recommended. You should use RaidZ2.

Thanks for that hint, didn't know that! Just for understanding: Do you know the rational behind that recommendation?

The reason that RaidZ2 is preferred for large disks relates to resilvering. When a disk goes bad in a pool, the probability is higher that another disk may also be at risk of going bad. Resilvering requires lots of heavy disk access. Large disks with a large amount of data can take a lot of time to resilver - increasing the odds that another disk in the pool might fail during resilvering. If this happens to a RaidZ1 pool, then you lose your data. If this happens to a RaidZ2 pool, then you are still OK. Make sense?

In my first NAS, I had 3x2Tb RaidZ1 pool. After a few years, one of the disks died. Although I was able to replace the bad disk without incident, it was a very uncomfortable feeling knowing that, if another disk had problems during resilvering then I would be at risk of losing data. Since then, I have always used RaidZ2.

A search through the forum or on the web will turn up lots of discussion about this.

 

[pschatz100]

Don't discount L2ARC out of hand. I found it did wonders when set to metadata only mode to speed up disk directory browsing (and rsync performance. See here.).

It my understanding that L2ARC supplements ARC, and is intended to improve the performance of random read workloads by caching recently accessed files. For a media server where files don't change often, I don't think this would be much of an issue.

 

[Constantin]

If you set the L2ARC to metadata only mode as I have, it primarily helps with browsing. That can be very helpful with traversing / browsing media collections.

See my rsync tests - as the cache got “hot” (ie metadata stored in the L2ARC) the rsync speeds went up considerably. It’s no different with every day directory browsing for media, as the directory contents are served up from a SSD not a HDD.

Granted, I’d first build out the RAM to 32GB or even 64GB first (to prevent the L2ARC index from starving the ARC for resources) but I would not dismiss the L2ARC out of hand. Test first, you might be surprised how much it can help.

 

[Graf_Euler]

The reason that RaidZ2 is preferred for large disks relates to resilvering. When a disk goes bad in a pool, the probability is higher that another disk may also be at risk of going bad. Resilvering requires lots of heavy disk access. Large disks with a large amount of data can take a lot of time to resilver - increasing the odds that another disk in the pool might fail during resilvering. If this happens to a RaidZ1 pool, then you lose your data. If this happens to a RaidZ2 pool, then you are still OK. Make sense?

Yes, that makes sense to me. Thanks for the explanation! As I'm currently thinking about dropping the 4TB discs and use 8TB instead, I definitely go for RaidZ2 then.

If you set the L2ARC to metadata only mode as I have, it primarily helps with browsing. That can be very helpful with traversing / browsing media collections.

See my rsync tests - as the cache got “hot” (ie metadata stored in the L2ARC) the rsync speeds went up considerably. It’s no different with every day directory browsing for media, as the directory contents are served up from a SSD not a HDD.

Granted, I’d first build out the RAM to 32GB or even 64GB first (to prevent the L2ARC index from starving the ARC for resources) but I would not dismiss the L2ARC out of hand. Test first, you might be surprised how much it can help.

I read your tests and the stats are quite nice. Anyway, my first build will not have an L2ARC, as I will only use 32GB of RAM right now with upgrade possibilities to 64GB. Before installing an L2ARC I would upgrade the memory and than install the L2ARC. But as a first step I just like to get the system up and running and test how it performs without the cache.

I also had a look at your build report using the Norco 431. The chassis looks very much like mine, but yours seem to be ~10cm shorter than the one I was looking at. Looking at the front of the chassis they are pretty much the same, but the rest of the chassis look different. Having read your report and having already ordered the chassis (it should be delivered tomorrow) I'm a bit skeptical now, but still hoping the best. As it offers space for 15 disks and I'm starting with only 7 discs right now, I guess cooling should be okay for now...

 

[Constantin]

Watch your cabling. Support it well to avoid breaking SATA connectors as the hard drive array is rotated up for loading / unloading.

Also good to know that you can leave a lot of cooling channels via the unpopulated HDD slots. Just keep in mind that eventually you may have to engineer a working solution for a “full” array.

I’m glad that your chassis is longer - that should make it a lot easier to install a standard atx power supply as well as hard drives. But I still wonder to what extent this system / case will cook hard drives once filled, no matter what you do.