Hello, I'm learning and getting ready for my first FreeNAS build.

[ZeroBit]

So far I've read through CyberJock's FreeNAS Guide 9.10 PPT, the links in the PPT, many forum threads, lots of Google hits to my searches, Intel datasheets, and wanted to get started reading through the FreeNAS manual (which required me to sign-up, so here I am). I'm a programmer that has worked on everything from the Sony PSP, PS2, PS4, and Microsoft XBox One, to Enterprise Legal Database systems to Cellular Network Infrastructure (Base Station Controllers). In my college life I was a computer consultant that built and maintained network systems for small and medium sized businesses, as well as home users, and I've continued supporting my own home network as needed.

I'm hoping to build a FreeNAS system to house my source-code, photos and videos, and want to add jails for Plex, Minecraft and maybe MyCloud. I'll also have to run Samba to serve files to Win7 and Win10 desktop/laptops (probably SMB 2 after reading CyberJock's PPT, but still need to read the FreeNAS manual and FreeBSD docs). My thought at the moment is to use the following hardware, subject to change as I go through the manual, or get advice:

Intel Xeon E3-1225 V5 3.3GHz Quad-Core Processor
SuperMicro X11SSH-F-O Server Board with Intel C236 chipset, 2x Intel i210-AT GbE LAN, 8x SATA3 and RAID 0, 1, 5, 10
SuperMicro 32GB SATA DOM SSD-DM-32-PHI
SeaSonic 650W 80+ Gold Certified Fully-Modular ATX Power Supply
Samsung M391A2K43BB1-CPB 16GB ECC Unbuffered DDR4-2133 Memory (x2 for a total of 32GB)
Fractal Design Node 804 MicroATX Cube Case
CyberPower CP1500PFCLCD UPS

The hard drives are still kind of a question mark, but after reading through the forums, I'm leaning towards the Western Digital Red 2TB drives x6. Reason for 6 drives is that I think I want to setup one zpool with one VDev in a RaidZ2 (data's important, and I don't want to be one of those 'statistics' everyone keeps talking about ;) ); though I'm still trying to figure out if ZFS requires RaidZ1,2,3 or if it can work with Raid 10, and which would be more beneficial, RaidZ2 or Raid 10. If I can stretch the budget I would like to go to the 3TB, but I have to keep it all under $1600, so it may be 2TB drives for now, and upgrade the VDev with 3TB drives later. My understanding so far is that since VDevs cannot be modified after they are setup, I have to put all 6 drives into one VDev for RaidZ2, or I won't be able to support RaidZ2 in that VDev later. I figure better to spend the cash getting the server components now, rather than trying to build one of those Frankenstein systems everyone keeps warning people about, which means the storage size has to suffer.

While the processor has AES-IN, I'm not interested in encrypting my data. I also don't think I'll ever push the system enough to benefit from ZIL or L2ARC, so no need for an SSD for read or write caching. The dB level of the system is of great concern, I'm somewhat sensitive to fan whine, and this will be sitting either in my home office or my living room. I've considered putting in a CPU Cooler like a be quiet! Dark Rock 3, but I'm not sure about clearance and can start with the CPU cooler that comes with the Xeon retail box processor.

My current consumer grade NAS is only 1TB (D-Link DNS-323, with 2 x 1TB hard drives in Raid 1 configuration), and most of that is taken up with videos I can re-download from the various services where they were purchased. So I don't have a large amount of data that needs to be backed up, and my current plan for backups is to transfer the files via network to a system with an external 1TB drive. I welcome any suggestions or thoughts people have for backing up a FreeNAS system.

I usually don't post much, but I look forward to being part of the community.

 

[Ericloewe]

(probably SMB 2 after reading CyberJock's PPT, but still need to read the FreeNAS manual and FreeBSD docs)

You can just stick with FreeNAS' default. SMB3 features are mostly stable in current Samba versions.

Intel Xeon E3-1225 V5 3.3GHz Quad-Core Processor
SuperMicro X11SSH-F-O Server Board with Intel C236 chipset, 2x Intel i210-AT GbE LAN, 8x SATA3 and RAID 0, 1, 5, 10
SuperMicro 32GB SATA DOM SSD-DM-32-PHI
SeaSonic 650W 80+ Gold Certified Fully-Modular ATX Power Supply
Samsung M391A2K43BB1-CPB 16GB ECC Unbuffered DDR4-2133 Memory (x2 for a total of 32GB)
Fractal Design Node 804 MicroATX Cube Case
CyberPower CP1500PFCLCD UPS

I'd save a few bucks and go with an X11SSM-F and get extra PCI-e connectivity. Otherwise, looks good.

; though I'm still trying to figure out if ZFS requires RaidZ1,2,3 or if it can work with Raid 10, and which would be more beneficial, RaidZ2 or Raid 10.

We don't call it RAID10 around here, but ZFS can easily do that. We call them striped mirrors, since you have several mirror vdevs striped together (like vdevs always are).
RAIDZ2 is safer, mirrors provide more IOPS (which you probably don't need).

My understanding so far is that since VDevs cannot be modified after they are setup, I have to put all 6 drives into one VDev for RaidZ2, or I won't be able to support RaidZ2 in that VDev later.

Right, but you can freely add or remove mirrors from mirror vdevs, including single drives.

The dB level of the system is of great concern, I'm somewhat sensitive to fan whine, and this will be sitting either in my home office or my living room. I've considered putting in a CPU Cooler like a be quiet! Dark Rock 3, but I'm not sure about clearance and can start with the CPU cooler that comes with the Xeon retail box processor.

The Noctua fans in my server are essentially inaudible, along with the stock cooler.

I welcome any suggestions or thoughts people have for backing up a FreeNAS system.

The textbook solution is a second FreeNAS server. Your old NAS thing may also work as an ok solution, if you can run rsync on it.

 

[ZeroBit]

You can just stick with FreeNAS' default. SMB3 features are mostly stable in current Samba versions.

I'd save a few bucks and go with an X11SSM-F and get extra PCI-e connectivity. Otherwise, looks good.

We don't call it RAID10 around here, but ZFS can easily do that. We call them striped mirrors, since you have several mirror vdevs striped together (like vdevs always are).
RAIDZ2 is safer, mirrors provide more IOPS (which you probably don't need).

Right, but you can freely add or remove mirrors from mirror vdevs, including single drives.

The Noctua fans in my server are essentially inaudible, along with the stock cooler.

The textbook solution is a second FreeNAS server. Your old NAS thing may also work as an ok solution, if you can run rsync on it.

Wow! Thank you for the critique and help. I've got a lot here to digest and incorporate. I'll see if I can locate a source for the board you mention.

Once I order the parts, I'm hoping to spend about a month trying different configurations and playing around with the system, to understand how it works and what different options will do (hopefully remembering to write down crucial lessons). After that I'll wipe the whole thing and configure it for real.

 

[Stux]

6 way Raidz2 or mirrors. Either are good.

If you can't afford to go big now, mirrors might be a good solution.

You still need a backup.

One approach build your 6x z2 on the cheap and backup to your old nas (critical stuff)

Then go bigger in the future and repurpose the now obselted system as your backup.

And you can always keep the old NAS for more paranoia.., maybe offsite.

 

[ZeroBit]

6 way Raidz2 or mirrors. Either are good.

If you can't afford to go big now, mirrors might be a good solution.

You still need a backup.

One approach build your 6x z2 on the cheap and backup to your old nas (critical stuff)

Then go bigger in the future and repurpose the now obselted system as your backup.

And you can always keep the old NAS for more paranoia.., maybe offsite.

Thank you. I will likely do something like what you mention, repurposing my old NAS as the backup for the critical stuff, using rsync as Ericloewe suggested. After getting a rsync script setup to do copy all the critical folders, I'll create a cron job to run the script. :)

 

[ZeroBit]

So, at the end of November, I pulled the trigger and ordered parts for my FreeNAS server, with a few changes. For example, I followed Ericloewe's motherboard recommendation, to get the extra PCI-e connectivity, and bumped the processor up to the Intel Xeon E3-1245 V5, from the Intel Xeon E3-1225 V5, to get hyper-threading for running jails like Plex. Here's what I finally ordered:

Intel Xeon E3-1245 V5 3.5GHz Quad-Core Processor
SuperMicro X11SSM-F-O Server Board with Intel C236 Chipset, 2x Intel i210-AT GbE LAN, 8x SATA3, 1x RJ45 Dedicated IPMI Port
SuperMicro 32GB SATA DOM SSD-DM-32-PHI
Western Digital 3TB, 5400RPM NAS Red Drives (x6 for total of 10.5GB in RaidZ2 mode)
SeaSonic 650W 80+ Gold Certified Fully-Modular ATX Power Supply
Samsung M391A2K43BB1-CPB 16GB ECC Unbuffered DDR4-2133 Memory (x2 for a total of 32GB)
Fractal Design Node 804 MicroATX Cube Case
CyberPower CP1500PFCLCD UPS

One thing I'm wondering is if it is possible to mirror the 32GB SATA DOM, containing the OS, with a 32GB SanDisk USB Drive? Or would that slow down the SATA DOM in its attempt to mirror the writes?

NewEgg had a great Pre-Black Friday sale on the drives, so the price difference between the WD 2TB 5400RPM Red NAS drives and the WD 3TB 5400RPM Red NAS drives was a no-brainer (yeah, between upgrading the processor and the drives, I blew the initial budget by a bit, but I believe it will be better for it in the long run). After putting all the parts together, I ran memtest86 for 192 hours (and got 6 full passes with no errors, yay!). Next I ran CPUStress, from the Ultimate Boot CD, for 120 hours, also with no errors. Next up was getting FreeNAS installed, so I could start running tests on the hard-drives.

I tried following a number of the ISO to USB Boot drive setups, without much success:

Preparing the Media - First I started with reading the FreeNAS documentation recommendations for Windows, and then moved on to attempting other methods once that failed.
Windows USB/DVD Tool - Microsoft appears to have modified the tool to recognize the difference between one of their ISOs, and any other ISO, or there's an issue with the way FreeNAS ISO is created.
Win32DiskImager - the drive wouldn't boot, and unfortunately this made two USB drives completely unusable. Fortunately, I was able to use diskpart to wipe the USB drives and re-initialize them, making them usable once again.
Burning an IMG File - FreeNAS no longer distributes an IMG file with the latest stable version, but this article kept coming up in Google searches, and it led me back to the docs, which led me to investigate using dd for Windows with the ISO, similar to how it's done for Mac or Linux.
dd for Windows - close, but there seems to be an issue with GRUB, it freezes while loading the GRUB menu. Maybe in the future dd for Windows might be an alternative.
UNetbootbin - UNetbootbin came up as a suggestion for burning an ISO to a USB drive, but this seemed to suffer the same problem as running dd for Windows with the latest stable ISO.​

By this point I was really beginning to wonder if the ISO somehow was corrupt, and due to the extremely rare hash collisions was still providing a matching hash code (very extreme edge case, but it's also hard to believe so many people have had success with all these other programs, and I'm having so much trouble). So I re-downloaded the ISO, compared the hash value with the expected value, and still had a valid file according to the hash. I then did a binary comparison on the two downloads of the ISO, and they were exactly the same (not a problem with the file...).

I saw some people had mentioned using Rufus, so I decided to give that a shot. Plain and simple, Rufus just works. It noticed an issue with the version of GRUB that Rufus is distributed with and the ISO, and was able to download an older version of GRUB to match the ISO (From my notes of the output: "ISO image uses GRUB2.02~beta2, Rufus has GRUB2.02~beta3 and will try to find and download a compatible core.img file."). Also, while playing with the settings, and trying to burn the ISO to the USB drive, Rufus made the following observation of the ISO, "ISO image is an ISOHybrid image"; this may be related to the issue with the Windows USB/DVD Tool. The installation of FreeNAS onto the 32GB SATA DOM went extremely smooth from this point on. Note: I exited out of the volume creation in the Wizard, to run the Hard-drive Burn-In Tests without ZFS or logging interfering with the tests.

So, now that the OS was installed, and I could get to a prompt, I performed the Hard-drive Burn-In Tests. None of the S.M.A.R.T. Tests (short, conveyance or long) on any of the drives reported any errors, and neither did 'badblocks'. Many thanks to qwertymodo for putting the How-To together, and everyone for helping him to update the post for things like 'badblocks' -b option for larger hard-drives. The 'smartctl' short test took about 2 mins, the conveyance test took about 5 mins, and the long test took about 6 hours. The 'badblocks' destructive read/write '-ws' test took about 60 hours. The long S.M.A.R.T. Test, post 'badblocks' test, also returned with 0 errors successfully on all drives.

So far I've:

1. Assigned an email address for root (to email tests)
2. Configured System->Email settings (with Gmail 2-Factor authentication thanks to Yatti420)
3. Created a RaidZ2 volume with the 6 drives
4. Scheduled S.M.A.R.T. Tests, S.M.A.R.T. Checks and Scrubs (thanks to cyberjock for the Scrub and S.M.A.R.T. Testing Schedule, it's a great starting point)
5. Created Accounts and Groups for everyone on my LAN
6. Created Datasets for different Windows shares
   • + Media
   • + Private
   • + Documents
     
   • + user1
   • + user2
   • + user3
   • + Shared
7. Setup SMB, configured ownership of the shares, and set the security on the shares. Each user has ownership of their Private->Documents->userX share. The Private and Shared Datasets will be backed up, once I get that setup; I don't care about Media, because that will only be movies, music, etc. that I have purchased and can re-download.

I'm currently reading and looking into configuring the CyberPower CP15ooPFCLCD UPS with FreeNAS (yes, it's currently protecting the system's power; it's just not setup to take care of shutting down). From what I've found so far, I'll use the CyberPower CP1000PFCLCD UPS driver, since there isn't one specific to the 1500 model available, and the 1000 model one supposedly works with the 1500 (Source). After that it will be setting up my DNS-323 to run RSync, using FunPlug, and setting up RSync on FreeNAS, so I can backup the Private and Shared Datasets. I'm wondering if this should be done using Periodic Snapshots of the two Datasets, and then using RSync to transfer those to my DNS-323?

So far it's been mostly straight-forward, and I'm looking forward to working through the rest of the setup and testing. If there were anything that could be improved, it would be making the installation USB drive easier and more uniform. Overall though, I'm happy with the decision to use FreeNAS.

 

[wblock]

Writing the image to USB should not be that hard. I've used Win32DiskImager for image files for years. It's possible (but unlikely) that an antivirus program was interfering. Or a virus, come to think of it. UEFI can complicate this also. But you eventually got past it.

 

[Ericloewe]

One thing I'm wondering is if it is possible to mirror the 32GB SATA DOM, containing the OS, with a 32GB SanDisk USB Drive? Or would that slow down the SATA DOM in its attempt to mirror the writes?

Yes to both.

I tried following a number of the ISO to USB Boot drive setups, without much success:

By this point I was really beginning to wonder if the ISO somehow was corrupt, and due to the extremely rare hash collisions was still providing a matching hash code (very extreme edge case, but it's also hard to believe so many people have had success with all these other programs, and I'm having so much trouble). So I re-downloaded the ISO, compared the hash value with the expected value, and still had a valid file according to the hash. I then did a binary comparison on the two downloads of the ISO, and they were exactly the same (not a problem with the file...).

I saw some people had mentioned using Rufus, so I decided to give that a shot. Plain and simple, Rufus just works. It noticed an issue with the version of GRUB that Rufus is distributed with and the ISO, and was able to download an older version of GRUB to match the ISO (From my notes of the output: "ISO image uses GRUB2.02~beta2, Rufus has GRUB2.02~beta3 and will try to find and download a compatible core.img file."). Also, while playing with the settings, and trying to burn the ISO to the USB drive, Rufus made the following observation of the ISO, "ISO image is an ISOHybrid image"; this may be related to the issue with the Windows USB/DVD Tool. The installation of FreeNAS onto the 32GB SATA DOM went extremely smooth from this point on. Note: I exited out of the volume creation in the Wizard, to run the Hard-drive Burn-In Tests without ZFS or logging interfering with the tests.

For future reference, the easy way is to use IPMI and mount the ISO with the iKVM virtual CD drive.

I'm wondering if this should be done using Periodic Snapshots of the two Datasets, and then using RSync to transfer those to my DNS-323?

Sure.

 

[ZeroBit]

Writing the image to USB should not be that hard. I've used Win32DiskImager for image files for years. It's possible (but unlikely) that an antivirus program was interfering. Or a virus, come to think of it. UEFI can complicate this also. But you eventually got past it.

Anti-virus software is a possibility. I use Bit-Defender on the machine where I was trying to create the USB boot drive; it's been pretty aggressive about identifying threats, and has generated some false-positives. Ironically, some of the software I've been coding has been identified by it from time to time; usually the final executables are considered clean though. ;)

 

[ZeroBit]

Yes to both.

For future reference, the easy way is to use IPMI and mount the ISO with the iKVM virtual CD drive.

Sure.

Thank you. I think I'll look into getting a second SATA DOM to mirror the drive, after the new year; maybe wipe the system and try using the ISO via IPMI with the iKVM virtual CD drive, when I setup the mirrored OS drive. Last thing I want to do is lose my data because of an issue with the OS drive and not having the OS backed up.

 

[diedrichg]

The mirrored boot drives is to assist in zero downtime of the server. One boot drive bites the dust and the other takes over. You won't lose the FreeNAS OS until that second boot drive goes unless the first is replaced.

Anywho... yes, mirror the boot drives. What DOES need to be backed up is your server config. This can be done manually in the GUI and/or you can set up a cron job to back it up for you - heck, even email it to you at a certain interval. There is a tutorial by CyberJock to run a script. The config backup could even live on your pool, you would just setup a new FreeNAS install, import your pool, and then import your config and your system would be back up and running in less time than it would take to even make a new installation.

 

[Ericloewe]

Last thing I want to do is lose my data because of an issue with the OS drive and not having the OS backed up.

That would never really happen.

 

[ZeroBit]

The mirrored boot drives is to assist in zero downtime of the server. One boot drive bites the dust and the other takes over. You won't lose the FreeNAS OS until that second boot drive goes unless the first is replaced.

Anywho... yes, mirror the boot drives. What DOES need to be backed up is your server config. This can be done manually in the GUI and/or you can set up a cron job to back it up for you - heck, even email it to you at a certain interval. There is a tutorial by CyberJock to run a script. The config backup could even live on your pool, you would just setup a new FreeNAS install, import your pool, and then import your config and your system would be back up and running in less time than it would take to even make a new installation.

Oh, okay. Thank you for the explanation on how to do that. :) I was clearly thinking about the severity of losing the boot drive in the wrong way. I'll see if I can locate CyberJock's tutorial you've mentioned here.

 

[ZeroBit]

Yes to both.

For future reference, the easy way is to use IPMI and mount the ISO with the iKVM virtual CD drive.

Sure.

So, SuperBiiz finally got the 32GB SATA DOMs back in stock. I ordered one, and it arrived yesterday. This evening, due to my lack of understanding exactly what iKVM is, I spent a few hours trying to get the IPMI GUI's Virtual Media (what I thought iKVM was referring to) to mount the drive. Windows 10 was being a pain, and wouldn't accept the request to the share from IPMI. Then I went back, read some more of the forums, and finally realized I was being dumb. The Remote Console that opens up in Java from the IPMI System summary screen says, "Java iKVM Viewer..." right at the top, and has a "Virtual Media" drop-down menu right at the top.

It took a couple seconds to set the Virtual Storage settings, and then a moment or two to figure out what the Plug-in and Plug-out buttons meant. But yeah, that was SOOOOO much easier than the headache of fighting to get the IPMI GUI's Virtual CD/ROM to work. Thank you.

I finally got my FreeNAS system to boot off the image, and was about to overwrite my existing initial installation, when I went back and re-read the Mirroring the Boot Device section in the manual, just to make sure I would get it right (it's been about a month since I first read that part, and a lot has happened, so it's pretty fuzzy). That's when I realized, I didn't have to wipe the system to mirror the boot device. Yay, for simplicity! I had already figured out how to mount an ISO via iKVM, so wiping the drive and starting over would just be another exercise setting the system up again (actually I have a backup of the server config I could have used, after following CyberJock's tutorial to Backup config file every night automatically). Of course, the exercise might be worth the experience.

Over the holiday break I also set the CyberPower UPS settings, configured it for automatic shutdown after 30 seconds, tested it for power-loss by pulling the power, and everything worked like it was supposed to first time. :)

Next step will be getting RSync setup with my old DNS-323 NAS, via FunPlug. Then I think this box may be ready to start taking on some real data.

 

[NicolasVin100]

Hi!

I'm in the process of buying the parts to my first build and I saw that you bought :

Supermicro X11SSM-F with Samsung M391A2K43BB1-CPB 16GB ECC Unbuffered DDR4-2133 Memory (x2 for a total of 32GB)

I'm also looking to buy this motherboard, but I've read horror stories of problems between RAM (the one that are not on Supermicro Tested list) and this board.

Yours doesn't seem to be on it, so I was wondering if you'd had any trouble with it?

 

[ZeroBit]

Hi!

I'm in the process of buying the parts to my first build and I saw that you bought :

Supermicro X11SSM-F with Samsung M391A2K43BB1-CPB 16GB ECC Unbuffered DDR4-2133 Memory (x2 for a total of 32GB)

I'm also looking to buy this motherboard, but I've read horror stories of problems between RAM (the one that are not on Supermicro Tested list) and this board.

Yours doesn't seem to be on it, so I was wondering if you'd had any trouble with it?

Hello.

According to SuperMicro's memory compatibility site(https://goo.gl/sl4KHI) on the X11SSM-F motherboard, the memory I have is listed as compatible. You can check out the Tested Memory List on the motherboard description page(https://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSM-F.cfm), and it should take you to the same results as the shortened URL there. Make sure to click the 16GB tab.

I read a lot of issues about memory and motherboard compatibilities on a number of different boards when I was first getting started too. So I made sure to buy exactly what they said was compatible (heed the warning, and buy ECC memory).

I like the motherboard well enough, but if I had it to do again, I would look to see if there was one that offered better control over the fans. I bought some Noctua fans, which are nice and quiet (see sig for details), but the board's fan controller in the BMC has issues with setting the power signal too low, and the fans turn off. At this point the fan controller on the motherboard triggers an event, upon which it kicks the fans into high mode, or 100% power. Then the fan controller realizes the temperature doesn't demand 100% power to the fans, knocks the power down (again to a power level too low), the fans shut off, the fan controller triggers the event, etc. So the fans rev between 100% and off until the IPMI logs are full and the fans go to 100% until the logs are manually cleared (the default to 100% power actually makes sense when you think about server hardware costs). The amount of control SuperMicro offers on the BMC, via the IPMI interface, for the fan controller is minimal. Alternatively, in the future I may try some of the Noctua Industrial PPC 3000 PWM fans, but one of my goals is a quiet system that goes in the office/guest bedroom. So, it needs to be quiet enough that people trying to sleep don't notice it usually.

Here's a few threads you should read to get an idea of what you may have to work with, and be prepared to use the script listed in the last one (kinda defeats the purpose of the BMC with the PWM fans, since it is now taking a small extra bit of the processor to run the script, but it substitutes much better control):

* https://forums.freenas.org/index.ph...or-thresholds-with-ipmi-using-ipmiutil.23331/
* https://forums.freenas.org/index.ph...or-thresholds-with-ipmi-using-ipmitool.23571/
* https://forums.freenas.org/index.php?threads/supermicro-fan-control-ipmiutil-or-feeipmi.18377/
* https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/

I wish you all the best with your build, and would love to hear how your system turns out.