Finally A Decent FreeNAS Build

[ThroughTheUnknown]

This is my first time posting so bare with me.

I'm finally ready to get my head out of the sand and stop swimming in my pool of Freenas do-nots. My current build is abysmal, and I've known for quite some time. It's just a dinky little €500 Eur laptop rocking 16GB of DDR4 Ram and a Core i3 with absolutely no ECC capabilities (I like living on the edge ;)). I've been saving up for some time and I have a budget of about €1500. Currently the whole build will take up about €1350 of this budget but I already have 4 WD Reds and I'm probably gonna buy a few more somewhere down the line.

As for the usage that this will be seeing, it'll have 4x WD Red 2TB and 2x 128GB SSDs (as boot drives), a few jails (Sonarr, Couchpotato, Transmission, Plex and Syncthing) and maybe 1 VM (running ubuntu server) doing some very light stuff which I can probably do right from a jail but that is something that I'll learn eventually.

Parts List:
Supermicro X11SSM
Intel Core i3 7320
Noctua NH-L9x65 (This might change in the future)
Kingston ValueRAM KVR24E17D8/16 Server CL17 16GB 2400MHz DDR4 ECC DIMM) (I'll be getting 2 of these for 32GB and upgrading in the future to 64GB)
Corsair HX750i COR 750W (Might switch to the one without the digital stuff)
LSI LSI00301 8 Port 6Gbps SAS 9207-8i SGL PCI-E Host Bus Adaptor
Cabled eConn Internal Mini SAS 36pin (SFF-8087) Male to 4 SATA
Silverstone CS380
TP-LINK TL-SG1024

I wanted to see about anyone's thoughts regarding the case. I know it's built down to a price but how bad is the quality? I chose this because having hot-swappable drive bays would be pretty awesome although not a necessity. If anyone has any other suggestions I'd need it to hold up to 8 drives.

Any thoughts on the HBA card and Network Switch? I'm buying a managed switch to hopefully take advantage of the Link aggregation that Freenas would offer with the 2 gigabit ports on the mobo. It isn't at all required because I would be the only person using this machine but I could have multiple devices syncing through Syncthing

Thanks in advance!

 

[Bozon]

This MOBO has 6 8 SATA ports so you don't actually need the SAS card right now, if you want to reallocate that money to beer, memory, or something else.

 

[CraigD]

Firstly welcome

I recommend starting with 6-8 data drives in one pool, you will thank me later, you need to add a HBA to do this (skip the boot mirror if cost is an issue, be sure to keep the config file on another computer)

I'm thinking of building a backup system in this case, I found this mod guide or review for it, hot swap bays are awesome

I started with an old case loaded with drive cages, this works fine until I needed to replace drives, soon after trashed the cages for hot swap bays, wasting time and money

Have Fun

 

[ThroughTheUnknown]

This MOBO has 6 8 SATA ports so you don't actually need the SAS card right now, if you want to reallocate that money to beer, memory, or something else.

Yeah that makes sense. Definitely beer though (for the build, yah know)

Firstly welcome

I recommend starting with 6-8 data drives in one pool, you will thank me later, you need to add a HBA to do this (skip the boot mirror if cost is an issue, be sure to keep the config file on another computer)

I'm thinking of building a backup system in this case, I found this mod guide or review for it, hot swap bays are awesome

I started with an old case loaded with drive cages, this works fine until I needed to replace drives, soon after trashed the cages for hot swap bays, wasting time and money

Have Fun

Why would I need 6-8 drives? Is it a reliability issue or a speed issue?

Do I really need an HBA to do this if I have 6 HDDs and 2 SSDs? Sure, if I have 8HDDs then I would need it for the SSD boot drives. As for the SSD boot drives, I already have them I just need to test them

 

[Chris Moore]

but I already have 4 WD Reds and I'm probably gonna buy a few more somewhere down the line.

The reason @CraigD said to start with 6 to 8 is so you can create a RAIDz2 pool, which gives you the best mix of capacity and reliability. While you can create a RAIDz2 pool with 4 drives, you can't presently add more drives to the vdev to expand capacity. It is a matter of eliminating difficulty down the line, I imagine.
How had you planned to use the drives you have?

 

[Bozon]

Yeah that makes sense. Definitely beer though (for the build, yah know)



Why would I need 6-8 drives? Is it a reliability issue or a speed issue?

Do I really need an HBA to do this if I have 6 HDDs and 2 SSDs? Sure, if I have 8HDDs then I would need it for the SSD boot drives. As for the SSD boot drives, I already have them I just need to test them

Because of drive size RAIDZ2 is the current standard and 6 to 8 drives work better with it because you can think of 2 drives as being used as parity, and the rest as data ( this isn't what is actually going on but it is a good enough analogy for this. If you want your head to hurt, or you are into this sort of thing, here are the details of what is going on http://dtrace.org/blogs/ahl/2006/06/18/double-parity-raid-z/). This allows up to 2 drives to fail before you lose any data. The rebuild time is what is driving the need for RAIDZ2.

 

[ThroughTheUnknown]

The reason @CraigD said to start with 6 to 8 is so you can create a RAIDz2 pool, which gives you the best mix of capacity and reliability. While you can create a RAIDz2 pool with 4 drives, you can't presently add more drives to the vdev to expand capacity. It is a matter of eliminating difficulty down the line, I imagine.
How had you planned to use the drives you have?

I had planned to put them in a RaidZ2... I guess it ain't the best then

Because of drive size RAIDZ2 is the current standard and 6 to 8 drives work better with it because you can think of 2 drives as being used as parity, and the rest as data ( this isn't what is actually going on but it is a good enough analogy for this. If you want your head to hurt, or you are into this sort of thing, here are the details of what is going on http://dtrace.org/blogs/ahl/2006/06/18/double-parity-raid-z/). This allows up to 2 drives to fail before you lose any data. The rebuild time is what is driving the need for RAIDZ2.

Thanks for the link. I'll definitely get my hands on 2 more then. Why 6 to 8 drives though, if those 2 parity drives fail doesn't all your data just vanish? So 4 or 8 drives, if those 2 drives fail you still lose your data?

 

[Chris Moore]

Why 6 to 8 drives though, if those 2 parity drives fail doesn't all your data just vanish? So 4 or 8 drives, if those 2 drives fail you still lose your data?

No. If you have six drives in RAIDz2 and two of them fail, all the data is still available on the four remaining drives. It is highly unlikely that two drives should fail at once.

 

[ThroughTheUnknown]

No. If you have six drives in RAIDz2 and two of them fail, all the data is still available on the four remaining drives. It is highly unlikely that two drives should fail at once.

Okay, makes sense. I'm going on a tangent now but what difference would it make if 3 drives fail in a RaidZ2 made up for 4 or 8 drives? I assume your data is lost in either case once 3 drives fail, so is the only advantage to having more disks in a RaidZ2 just a simple case of having more storage?

 

[Chris Moore]

Okay, makes sense. I'm going on a tangent now but what difference would it make if 3 drives fail in a RaidZ2 made up for 4 or 8 drives?

The 2 in RAIDz2 allows two drives of parity, that means two drives can fail.. If three drives fail, you loose the pool. That is a constant of a RAIDz2 vdev regardless of if the vdev has 4 drives or 14 drives, only two can fail without data loss. If you want three drives of redundancy (parity) then you need RAIDz3, which gives you the most protection available in ZFS. You can have a hot-spare drive in any ZFS storage pool.
The idea of six or eight drives is because that is the 'optimal' number based on some complicated information and it sounds like you might want to do more reading. Here are some links for more understanding:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

You might also like to look at some of this data:

Six Metrics for Measuring ZFS Pool Performance Part 1

Six Metrics for Measuring ZFS Pool Performance Part 2

There is so much to learn about ZFS. There have been multiple books written on the subject.

 

[Chris Moore]

PS. In a single ZFS storage pool, you can have many vdevs and each vdev would be made of an equal number of drives and at the same level of redundancy.
For example, I have a pool at work that is made of ten vdevs of six drives each, with each vdev being at RAIDz2.
Here is what that looks like:

Code:

zpool status
  pool: Pogo-60x10TB
 state: ONLINE
  scan: scrub repaired 0 in 0 days 17:00:17 with 0 errors on Sat Jan 19 00:26:25 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Pogo-60x10TB                                    ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/4b380e51-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/4c2a3fa0-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/4d316c90-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/4e279c43-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/4f3b17f7-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5048ec53-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/51a5cb98-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/52b0c4f5-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/53c02fc8-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/54b2adad-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/55c0f47a-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/56c14c23-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-2                                      ONLINE       0     0     0
            gptid/584f5745-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5956d7f8-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5a5a6070-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5b564fc1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5c630294-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5d6a2431-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-3                                      ONLINE       0     0     0
            gptid/5ef1faf6-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/5ffb8961-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6105cc1d-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/62120300-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/631e28a1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/642dd1ea-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-4                                      ONLINE       0     0     0
            gptid/65d9d859-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/66de6408-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/67ee6f2c-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6905e892-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6a1e7078-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6b2a5922-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-5                                      ONLINE       0     0     0
            gptid/6cf142e9-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6e039cc3-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/6f1bdf1d-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7026b5ed-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/712a87a1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/723bc10e-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-6                                      ONLINE       0     0     0
            gptid/7416f42f-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7531a5f8-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/76428144-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7760f346-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7866427c-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/798d143e-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-7                                      ONLINE       0     0     0
            gptid/7b875ee7-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7c9c17f7-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7db105ca-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7ed28e84-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/7ff52ad1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/80faf9ba-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-8                                      ONLINE       0     0     0
            gptid/831d73e2-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/843431a1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/85534614-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/866f8059-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/8791862e-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/88abe418-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
          raidz2-9                                      ONLINE       0     0     0
            gptid/8ad8a1e1-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/8bf8c94e-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/8d14e578-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/8e2c0e1e-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/8f4586ee-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
            gptid/905d5231-f24d-11e8-bd4a-ac1f6b418926  ONLINE       0     0     0
        logs
          gptid/df8c688c-1b50-11e9-bd4a-ac1f6b418926    ONLINE       0     0     0
        cache
          gptid/e1f09019-1b50-11e9-bd4a-ac1f6b418926    ONLINE       0     0     0

errors: No known data errors

 

[ThroughTheUnknown]

The idea of six or eight drives is because that is the 'optimal' number based on some complicated information and it sounds like you might want to do more reading. Here are some links for more understanding:

I will do that. Just finished University exams so I've got some time on my hands. Thanks for your help


Before I hit the big yellow button to check-out, does this build look solid? I found another i3 CPU (7350K), this won't be overclocked of course even though it is unlocked. Would this CPU be better than the Core i3-7320?

Also, how would I make sure that I get my HDDs from separate batches? Is it just a simple case of buying from different vendors or is there another better way?

I also wanted to sorta take advantage of the other LED headers on the Mobo (like the fan fail, overheat, etc.) is there some kit or something that I can just slap onto the case even if it's a third party sorta thing (doesn't have to look pretty). I know I can make my own but I'm wondering if there's something ready-made that would probably look better than anything I can make

 

[pschatz100]

Okay, makes sense. I'm going on a tangent now but what difference would it make if 3 drives fail in a RaidZ2 made up for 4 or 8 drives? I assume your data is lost in either case once 3 drives fail, so is the only advantage to having more disks in a RaidZ2 just a simple case of having more storage?

Yes, the issue is really about storage space.

I have 4 drives in a RaidZ2 and it works just fine. With 4 drives in RaidZ2, one half of my space is going to redundancy. If I had six drives in RaidZ2, one third of my capacity would be going to redundancy. For instance, I am currently using 4 TB drives, so I have about 8 TB of storage (actually a bit less after formatting, etc) in my four disk volume. If I were to have 6 drives in RaidZ2, then I would have about 16 TB of storage with the same redundancy. Make sense?

Now, if I want to increase the capacity of my four disk volume by adding two more disks, it is not possible to simply add two disks to the four disk volume and expand it to six disks. The volume would have to be rebuilt, which means backing up all the data, destroying the old volume then building a new volume. Therefore, when it comes time to increase capacity I will have two choices: Buy four disks of larger capacity and swap them out on my old volume one-by-one, or go through the work to save all my data somewhere, then destroy the old volume and build a new one.

Also remember... redundancy is not a substitute for maintaining proper backups.

 

[Yorick]

Therefore, when it comes time to increase capacity I will have two choices

You might have a third choice depending on when your “when” is. By 2020/2021 I expect that raidz expansion will be production ready.

 

[ThroughTheUnknown]

Yes, the issue is really about storage space.

I have 4 drives in a RaidZ2 and it works just fine. With 4 drives in RaidZ2, one half of my space is going to redundancy. If I had six drives in RaidZ2, one third of my capacity would be going to redundancy. For instance, I am currently using 4 TB drives, so I have about 8 TB of storage (actually a bit less after formatting, etc) in my four disk volume. If I were to have 6 drives in RaidZ2, then I would have about 16 TB of storage with the same redundancy. Make sense?

Now, if I want to increase the capacity of my four disk volume by adding two more disks, it is not possible to simply add two disks to the four disk volume and expand it to six disks. The volume would have to be rebuilt, which means backing up all the data, destroying the old volume then building a new volume. Therefore, when it comes time to increase capacity I will have two choices: Buy four disks of larger capacity and swap them out on my old volume one-by-one, or go through the work to save all my data somewhere, then destroy the old volume and build a new one.

Also remember... redundancy is not a substitute for maintaining proper backups.

Ah okay, now it's sorta all making sense. Thanks for that, and yes, backups will be done to another machine on-site, an external HDD that will be kept with me and a cloud service (probably Backblaze)

You might have a third choice depending on when your “when” is. By 2020/2021 I expect that raidz expansion will be production ready.

I might just buy the disks now, it's the perfect time to store my data elsewhere and re-create the volume. Would anyone have any advice on how to get disks from different batches properly?

 

[Chris Moore]

Would anyone have any advice on how to get disks from different batches properly?

I wouldn't worry about that. Statistically, you are not very likely to see a failure. When I commissioned a new server at work, with 60 drives in it, I had only three drives fail in the first year. That is not a very high percentage and it is in line with industry expectations. In a small batch of six or even eight drives, your not likely to have a failure at all, but there is always a possibility. That is why we suggest a disk burn-in before you commit your data to the pool. Even after that, you should monitor the system for signs that the drives are having problems. Buying from different batches or even different manufacturers was once more of an issue but drive failure rates are a little better now than they were historically. As long as you get quality drives.
Here is a link to a guide that discusses setup of a new FreeNAS system, including disk burn-in:

Uncle Fester's Basic FreeNAS Configuration Guide
https://www.familybrown.org/dokuwiki/doku.php?id=fester:intro

Here is a nice trove of scripts to monitor system health:

Github repository for FreeNAS scripts, including disk burnin
https://forums.freenas.org/index.ph...for-freenas-scripts-including-disk-burnin.28/

 

[ThroughTheUnknown]

I wouldn't worry about that. Statistically, you are not very likely to see a failure.

Thanks for that

Just the last few questions though. Regarding the HBA I listed which was a SAS 9207-8i, is that compatible with Freenas and would I have to flash it in IT mode (HBAs are completely new to me). I was looking around and saw a recommendation for the SAS 9240-8i instead and flash that into IT mode. Any guidance?

Also, I listed that I was considering a different CPU which was the Intel i3-7350K, I know there shouldn't be any problems since it's unlocked but would there be any problems? It has all the same features as the i3-7320 that I mentioned before except that the clock speed goes up to 4.20GHz from 4.10GHz and that the 7350K is unlocked

 

[pschatz100]

Standard file reading/writing tasks are not CPU intensive, and I doubt you would see much difference between these processors for most tasks you listed. If the prices are similar, I would go with the faster processor. You could probably use the stock cooling solution and save €50 by skipping the Noctua cooler. You will not see much benefit from overclocking the unlocked processor.

The exception is Plex. If you decide to transcode videos using Plex on your FreeNAS, then you need enough CPU power to support that activity. An i3 will have enough capability to transcode one or two DVD quality video streams, but it won't have enough power to transcode Blu-ray. Personally, I don't do much transcoding with Plex... I prepare my videos on another system and convert them to h.264 or h.265 before loading them to FreeNAS. My players, 4K compatible ROKUs, read these formats natively. Most modern smart TV's also support h.264 formats (4K TV's also support h.265), therefore transcoding is not an issue most of the time. On my system, the only time I actually transcode is when accessing Plex over the web or on a mobile device - this is not a big percentage of my Plex activity.

When I frst built my system, I started with an i3 and two mirrored disks. Eventually, I upgraded the i3 to a Xeon, when I came across one for a very good price. There is more potential upside to a 4 core processor than a two core processor and I have been very happy with the performance of the Xeon. I looked at Amazon.de and there are mid-range Xeon's that are only marginally more expensive than the i3's you mentioned. Especially, if you buy a boxed processor that comes with a cooler and skip the Noctua cooler.

 

[ThroughTheUnknown]

Standard file reading/writing tasks are not CPU intensive, and I doubt you would see much difference between these processors for most tasks you listed. If the prices are similar, I would go with the faster processor. You could probably use the stock cooling solution and save €50 by skipping the Noctua cooler. You will not see much benefit from overclocking the unlocked processor.

The exception is Plex. If you decide to transcode videos using Plex on your FreeNAS, then you need enough CPU power to support that activity. An i3 will have enough capability to transcode one or two DVD quality video streams, but it won't have enough power to transcode Blu-ray. Personally, I don't do much transcoding with Plex... I prepare my videos on another system and convert them to h.264 or h.265 before loading them to FreeNAS. My players, 4K compatible ROKUs, read these formats natively. Most modern smart TV's also support h.264 formats (4K TV's also support h.265), therefore transcoding is not an issue most of the time. On my system, the only time I actually transcode is when accessing Plex over the web or on a mobile device - this is not a big percentage of my Plex activity.

When I frst built my system, I started with an i3 and two mirrored disks. Eventually, I upgraded the i3 to a Xeon, when I came across one for a very good price. There is more potential upside to a 4 core processor than a two core processor and I have been very happy with the performance of the Xeon. I looked at Amazon.de and there are mid-range Xeon's that are only marginally more expensive than the i3's you mentioned. Especially, if you buy a boxed processor that comes with a cooler and skip the Noctua cooler.

The 7350K is actually cheaper than the 7320. As for transcoding on plex, my current setup (which is an i3 laptop) handles my needs perfectly fine so I see no need to go for a xeon at this point although I have considered it and I was tempted but if all my needs are handled perfectly fine by a laptop CPU then a desktop i3 which is 3 generations newer should be more than enough. Apart from this, I would be the only person streaming from plex

 

[Yorick]

The i3-8100 is more affordable again, more powerful - enough for one CPU-only transcode of a 4k stream, is my guess - and fits into an X11SCH-F, which is the crux: Those aren't available everywhere yet. However, they are available near you, so more power to you! https://www.servershop-bayern.de/de/supermicro-x11sch-f-max-128gb-m-2-2x1gbe-8xsata