Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Build Report: Node 804, X11SSM, Xeon, PLEX

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

xaintly

Cadet
Joined
Aug 7, 2019
Messages
7
Spoilers: Yes it will FreeNAS.

I did some analysis planning what I wanted to build, and these forums were very helpful. There were some things I wish I'd known in advance, so hopefully this will help the next person. All of these parts are well-known and have been previously discussed as being great for FreeNAS so the compatibility won't be a surprise.

Build:
CPU: Intel Xeon E3-1240 V6 (includes a fan)
MB: Supermicro X11SSM-F
RAM: 2 x NEMIX DDR4-2400 REG/ECC SC 16GB
Boot Drive: Sandisk Cruzer Fit 32GB
Storage: 8 x WD Red NAS 10TB
PSU: Seasonic Focus Plus SSR-750FX 750W Gold
Case: Fractal Design Node 804
Fan for HD Cage: Noctua NF-F12 iPPC-3000
CPU Fan: Noctua NH-L9i Low-Profile
SATA Cables: 8x 18" SATA3 cable with 90 degree connector
Additional cable: Rosewill Molex to SATA 1 needed**
Total cost was about $3000 USD; $1300 for the case and the rest for the drives.

** Note, the PSU comes with 8 SATA power connectors and the Node 804 case needs 1 for the included fans.
Since I had 8 HDDs, I used a spare modular SATA power cable from another Seasonic, but you could use a Molex to SATA instead.
You could also plug all the fans straight onto the motherboard and not use the case controller.

Fans:
The Node 804 comes with a 3-speed fan controller and 3 fans (2 for the motherboard compartment, 1 for the drives/PSU compartment).
The controller/fans are powered from a SATA connector and are independent of the motherboard, so you could use the case fans and still plug 5 fans into the motherboard.
The hard drive compartment can get hot with 8 drives, so an extra fan is needed. The Noctua iPPC-3000 is a 4-pin fan and it won't plug into the 3-pin connectors for the case's fan controller. If you plug it into the motherboard, use the FANA connector. I installed the iPPC-3000 as an additional fan in the hard drive compartment, blowing across the drives.
I probably didn't need an additional CPU fan, I didn't realize I was getting one with the CPU.

Cables:
One of the drive bays is a very tight fit with the PSU, so you should probably get at least 4 of the SATA cables with a 90-degree connector.
The other bay has more room, so you could use any kind of cable. 18" was a perfect length, leaving a little slack to tie down with the included velcro in the case.

PSU:
The PSU is overkill, you could use anything with at least 400W I think. However the 650 and 750 come with 2x cables with 4 SATA connectors, which works really well with 8 hard drives. At the time I purchased these, the 750 was cheaper than the 650.

Case:
The case is somewhat unusual. It is the height of a mini-tower but twice as wide, divided into 2 separate compartments. One compartment has the PSU and hard drives, and the other has the motherboard. There is only 1 fan for the hard drives, and it is an exhaust fan blowing out of the case. If you turn it around to blow on the hard drives, then it will suck dust into the case. There is room to mount an additional 7 fans in the case, with 2 of those in the drive compartment. I used one of the drive compartment additional spaces to install the Noctua fan. There is a window in the case, but it is opposite the side that has the USB connector & power button so the window isn't visible in the location where I placed the NAS.

The drives in your case won't be hot-swap, so if you want to do the 1-drive-at-a-time hard drive replacement to upgrade your RAID, it's gonna suck.

RAM:
I'm a little wary of RAM brands I never heard of, but NEMIX ram is made by Hynex supposedly, and people have said it's decent.
Watching the RAM usage, I've occasionally seen it spike to 20gb when running PLEX and other stuff at the same time, so it seems good to get more than 16gb.

HDD:
Of the WD Red drives, 4TB currently has the best price-per-TB, and the next best is 10TB. The drive prices fluctuate from day to day though.

USB boot flash drive:
There were several people who liked Sandisk Cruzer, and 32gb was the cheapest available. They are very small, barely bigger than the USB connector itself. If you want to run from flash drive, you need at least 2 to do the install; 1 will become your boot drive and the other is used to install FreeNAS (after FreeNAS is installed you don't need the 2nd one, so if you have any flash drive laying around that should work I guess?).

Build process:
If you use this exact build, then plug the Noctua fan for hard drives into FANA on the motherboard so you can control it separately from the other motherboard fans.
The rest of the assembly is very easy. The case has lots of room, unlike trying to work in a mini-ITX case.

When adding the drives to the drive cage, write down the serial numbers of each drive. You won't be able to see the serial numbers later without taking drives out, and if a drive fails, FreeNAS tells you what serial number went bad. Save a diagram or drawing of the drive locations, maybe tape it to the inside of the case.

I used Rufus to install the FreeNAS iso onto the flash drive I used for installation.

If you are not used to server products, this motherboard seems slow to boot up. The noctua fan is also much louder than a desktop fan when it spins up to full speed.

I used a VGA connector & USB keyboard to install FreeNAS. You don't need a mouse and you don't need the monitor/kb after the install either.

FreeNAS installed onto the internal flash drive with no problems, I used the UEFI boot. After that, I had to go into the Bios and specifically choose 'UEFI: sandisk cruzer usb key' as the boot device.

FreeNAS setup:
There were no problems running FreeNAS. I configured all the drives as 1 big pool with RAID-Z2. I expect someone will probably tell me why that was a bad idea. Out of 80 TB ("drivemaker terabyte"), there's about 50Tb ("actual terabytes") of usable storage.

Getting the hard drive fan to work:
This was the only complicated bit. As installed, the Noctua fan will run at basically 'idle' speed of 750 RPM. With the 2 fans in the hard drive area (1 pushing and 1 pulling air), the drives will run slightly hot, 38-42 degrees. This is probably fine, but I like my drives to run in the 35-39 range, so I wanted to run the Noctua fan a little faster.

This required me to mess with IPMI. In FreeNAS, you can enable the IPMI service. It will have it's own IP address. You can then issue commands from the FreeBSD command line with 'ipmitool'.

If you plugged the Noctua into FANA, then you can issue these commands:
ipmitool -H ip_address_of_ipmi_interface -U ADMIN -P ADMIN -I lanplus raw 0x30 0x45 0x01 0x04
ipmitool -H ip_address_of_ipmi_interface -U ADMIN -P ADMIN -I lanplus raw 0x30 0x70 0x66 0x01 0x00 0x16
ipmitool -H ip_address_of_ipmi_interface -U ADMIN -P ADMIN -I lanplus raw 0x30 0x70 0x66 0x01 0x01 0x26

These set the fan mode to 'Full' (which allows control over fan speed), then sets the CPU/MB fans to 25% and FANA to 40%
These got the hard drives down to the 32-38 range, though now the CPU fan is permanently set to 25%.
The noctua fan is noisy at 75% and up, but under that and you probably won't hear it.

Here's the discussion on issuing IPMI commands to control the Supermicro fan zones.
You can also install IPMIView from Supermicro on another system. It will let you change the ADMIN password, see all the sensors visually and bring up the KVM so you can access the console of the system and watch it boot up.

Adding features:
Adding users: I created a couple users, one without the ability to login. However, this also disabled password logins via SMB. Re-enabling password logins on that user didn't enable samba; I had to use smbpasswd to assign their samba password.

Windows machine backups: After I created a SMB share and set up users, I installed Veeam Agent (Free product) on the windows PCs I wanted to backup. It looks fairly easy to schedule backups, you can set how often it should create a full backup (the rest will be incrementals) and it will do backups to a SMB share. After it did a backup, I tested a restore of a couple files and it seemed to work. It says it can do bare-metal restores (you create a recovery flash drive for this purpose), but I don't know how well it works.

Installing PLEX; this was as easy as just installing the plugin from the pull-down list of plugins in FreeNAS. I had already copied a bunch of media onto one of the shares, so I had to shut down PLEX, add that share location as a mount point in the PLEX jail and restart the service. Plex costs $5 to run on mobile devices, but the client app for my TV was free. PLEX works really well. I currently have a KODI box plugged into my TV, but I think FreeNAS/PLEX can replace it.

Cloud Sync to AWS: AWS now offers 'deep archive' storage at prices similar to Azure Blob Storage ($0.00099/gb/mo). I already had an AWS account, so I created an S3 bucket and an IAM user for freenas. If this is an interesting topic for someone I can elaborate on IAM policies, etc. With the credentials entered in FreeNAS, I attempted to copy the contents of a share from my location in California to the S3 bucket in the Oregon data center. It was pretty slow, as in less than 1gb/hr transfer rates. I eventually just logged in via SSH and killed the rclone process after it had backed up a few files. While rclone was running, the rest of my network was hosed, almost all my bandwidth ( I have cable internet) was going to the backup process.

Cons:
I didn't get a case with a drive backplane, so no hot-swap drives. I figured hot swap is only helpful for failures, and since I don't expect many failures, I can deal with powering down the server to replace drives.

I didn't know if you could upgrade a RAID in-place, but it turns out that you can; you just replace the drives one-at-a-time until all the drives are bigger (the RAID will use the smallest drive as a base, and if any drives are bigger the extra space will be unused). I'm thinking this is probably the best use-case for hot swap bays. As it is, if I try to use this method, it will be annoying to power down, take apart the case and swap drives.

I had thought I could just backup all the drives to AWS, replace them all at once, then re-download the data. However, my ISP has a monthly data cap, so this is probably not going to work either.

I may end up having to just build another NAS and copy everything over to it when the time comes.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,497
The SATA / power backplanes is one reason I like Lian Li cases. They come without capacitors (though with built-in spots for them). One can retrofit capacitors on them to potentially make them “hot swap” tolerant but I haven’t tested all that. I’d still power down the system.

The iPPC fan is great. I use two to keep the motherboard cool, another just for the cpu (via a 120mm-50mm Adapter). My drives and motherboard trundle along at below 35*C, the only “hot spot” is CPU and HBA.

I’d love to know what your power consumption is seeing that I have a similar number of drives, etc. in my server.
 

xaintly

Cadet
Joined
Aug 7, 2019
Messages
7
I recorded video of boot + 3 minutes, and then another minute while transcoding with PLEX.
Powered off: 3W
While booting: 80W-190W
Normal operation (transcoding didn't change the wattage): 70W (spikes up to 90W sometimes)
 
Top