CelticDubstep
Cadet
- Joined
- May 16, 2020
- Messages
- 7
I'm sure I'm going to get a lot of shaking heads and facepalms, but thought I'd share my little "adventure" with setting up a file server today. I started 12 hours ago. First, the system:
Case: Coolermaster N400
Motherboard: GA-990FXA-UD3 (rev. 4.0)
CPU: AMD FX-8350
RAM: 32 GB (maxed out)
Controller: LSI 9211-8i (Bought it new in IT Mode, but switched to IR Mode when I updated the firmware)
Storage Drives: 8x 12Gb/s 8TB SAS HGST Ultrastar Helium He8 (HUH728080AL4200)
OS Drives: 2x 256GB Samsung 850 Pro's.
NIC Card: Intel PRO/1000 PT Quad Port Gigabit Server Adapter
NIC Onboard: Realtek (Typical Onboard)
My first adventure started close to noon today. I performed a clean install of FreeNAS 11.3 U3.1 and selected the two SSD's for my "boot pool". I setup my drive pool, dataset, groups, users, shares, permissions, etc. I mapped the share on my Windows 10 desktop successfully, and mapped it on my existing Windows Server 2012 R2 system which acts as my File Server (no redundancy, no backups) and Plex Server. It's a 13 year old system (floppy drive included) that has been falling apart piece by piece over the years... A couple of PSU's, a stick of ram, etc. I fired up "FreeFileSync" on the 2012 R2 system and started started the process of copying files over to the FreeNAS System. I was a bit disappointed in the performance as I was only able to get around 550 Mbps (this was using the Intel NIC). I thought perhaps it was my new UniFi switch (it has a pending firmware update) so I pulled out a "dumb" Gigabit switch that I had only used for a couple months and connected the two systems to it, and the switch to the UniFi. No difference. *shrugs*
A bit later... I hear a BIOS Post Beep and look over at the FreeNAS system and was rebooting. Uhh... OK?. I figured it was a fluke, so I logged into the Web GUI and of course it gave the warning that the system shut down unexpectedly. I started the file copy process again and a while later... "beep". Troubleshooting Time. I had bought 4 of those Intel NIC's for a project that ended up not panning out. I didn't know at the time, but these specific PRO/1000 PT's I have (there is another PRO/1000 PT that is "half height" that doesn't have this issue) uses some type of "bridge" that won't work with motherboards with anything higher than V1.0 of PCI Express. However, to my surprise... all 4 of these cards worked on this FX-8350 system, but not any other system I tried. The only PRO/1000 PT I was able to get to work in another system was the half-height model card. So, I pulled the card out and went back to onboard and started process again... and of course, a while later... "beep". I spent some time on google, forums, etc... seems others had the exact same issue with no real solutions. As I had spent hours already, I threw in the towel and decided to try out UnRAID.
Intel NIC Removed, UnRAID was very much a mixed bag. The GUI was nice, provided a lot of information, and I used one of my SSD's as a cache drive (wouldn't let me use two for some reason). I'll have to say that the GUI has a bit of an edge on FreeNAS but not a huge difference. Anyway... I never could get UnRAID to work right... The parity build "paused" and couldn't figure out why until I selected one of my parity disks and in red text, it said there was SMART Errors and to download the report for review... but the report showed no issues and on the same page as the red text, "SMART Status: Healthy" in green. Whaaa??? I rebuilt the array and moved the drives around and no issues. Even with the cache drive, performance was worse than FreeNAS. CPU usage ran rather high, close to 50% nearly the entire time. I eventually got a warning that one of my drives was running warm (46C) and others were close to that as well. I added 3 additional fans to the case and some other minor adjustments and they got down to 29C - 30C. Of course it is now much louder than I would like since all the fans are running at 100%. I ordered a fan speed controller as I have no way to control them now. No crashes, but performance slowly sunk well below 250 Mbps. I was never keen on UnRAID anyway due to the cost and I don't want to run the OS off a flash drive.
I decided to give "Storage Spaces" a try, so I installed Windows Server 2019 and reinstalled the Intel NIC. I enabled onboard RAID and put the two SSD's into RAID 1 (at least it's something) for the OS. I was able to get the drives into a "Storage Pool" in Storage Spaces, but every... single... thing I tried, I could not get a Virtual Disk to create in the pool, even a 250 GB one. WTF?! I did some searching on google, only to read other people having the exact same issue and lots of people trash talking Storage Spaces so I quickly gave up on that idea. I deleted the pool and used diskpart and cleaned all the drives at that point.
A "light" went off at that moment. I had previously ran FreeNAS in Hyper-V without any issues, so I installed the Hyper-V Role and created a VM For FreeNAS. 24GB Ram, 128GB Boot Drive, 4 vCPU. I went into Windows Disk Management and set all 8 drives to "offline" and added all 8 of them to the VM as a direct passthrough. Sure enough, it worked flawlessly. FreeNAS had direct access to the drives and pulled all the info of the drives (SMART Data, serial number, make/model, RPM speed, etc). I repeated the process as I did before and started a file transfer and monitored it.
850-900 Mbps average transfer speed with peaks over 980 Mbps. The slowest I saw was around 700 Mbps but nothing lower. Rock Solid transfer rate and performance and no reboots.
So, this will be the setup I'm going to stick with. Win-Win. The OS Drives are in RAID, I can use snapshots and windows server backup to backup the 128GB boot drive of FreeNAS should anything happen. Since Hyper-V "emulates" a 10 Gb Switch/NIC, I can attach these assorted SATA drives I have laying around with data on them via USB 3.0 and map the drive on the host and I won't be limited by Gigabit Gear trying to transfer files over the network.
Just thought I'd share my "success" story. I know it's not "ideal" but it's working and FreeNAS has direct access to the drives and so far has had the best performance stability.
Case: Coolermaster N400
Motherboard: GA-990FXA-UD3 (rev. 4.0)
CPU: AMD FX-8350
RAM: 32 GB (maxed out)
Controller: LSI 9211-8i (Bought it new in IT Mode, but switched to IR Mode when I updated the firmware)
Storage Drives: 8x 12Gb/s 8TB SAS HGST Ultrastar Helium He8 (HUH728080AL4200)
OS Drives: 2x 256GB Samsung 850 Pro's.
NIC Card: Intel PRO/1000 PT Quad Port Gigabit Server Adapter
NIC Onboard: Realtek (Typical Onboard)
My first adventure started close to noon today. I performed a clean install of FreeNAS 11.3 U3.1 and selected the two SSD's for my "boot pool". I setup my drive pool, dataset, groups, users, shares, permissions, etc. I mapped the share on my Windows 10 desktop successfully, and mapped it on my existing Windows Server 2012 R2 system which acts as my File Server (no redundancy, no backups) and Plex Server. It's a 13 year old system (floppy drive included) that has been falling apart piece by piece over the years... A couple of PSU's, a stick of ram, etc. I fired up "FreeFileSync" on the 2012 R2 system and started started the process of copying files over to the FreeNAS System. I was a bit disappointed in the performance as I was only able to get around 550 Mbps (this was using the Intel NIC). I thought perhaps it was my new UniFi switch (it has a pending firmware update) so I pulled out a "dumb" Gigabit switch that I had only used for a couple months and connected the two systems to it, and the switch to the UniFi. No difference. *shrugs*
A bit later... I hear a BIOS Post Beep and look over at the FreeNAS system and was rebooting. Uhh... OK?. I figured it was a fluke, so I logged into the Web GUI and of course it gave the warning that the system shut down unexpectedly. I started the file copy process again and a while later... "beep". Troubleshooting Time. I had bought 4 of those Intel NIC's for a project that ended up not panning out. I didn't know at the time, but these specific PRO/1000 PT's I have (there is another PRO/1000 PT that is "half height" that doesn't have this issue) uses some type of "bridge" that won't work with motherboards with anything higher than V1.0 of PCI Express. However, to my surprise... all 4 of these cards worked on this FX-8350 system, but not any other system I tried. The only PRO/1000 PT I was able to get to work in another system was the half-height model card. So, I pulled the card out and went back to onboard and started process again... and of course, a while later... "beep". I spent some time on google, forums, etc... seems others had the exact same issue with no real solutions. As I had spent hours already, I threw in the towel and decided to try out UnRAID.
Intel NIC Removed, UnRAID was very much a mixed bag. The GUI was nice, provided a lot of information, and I used one of my SSD's as a cache drive (wouldn't let me use two for some reason). I'll have to say that the GUI has a bit of an edge on FreeNAS but not a huge difference. Anyway... I never could get UnRAID to work right... The parity build "paused" and couldn't figure out why until I selected one of my parity disks and in red text, it said there was SMART Errors and to download the report for review... but the report showed no issues and on the same page as the red text, "SMART Status: Healthy" in green. Whaaa??? I rebuilt the array and moved the drives around and no issues. Even with the cache drive, performance was worse than FreeNAS. CPU usage ran rather high, close to 50% nearly the entire time. I eventually got a warning that one of my drives was running warm (46C) and others were close to that as well. I added 3 additional fans to the case and some other minor adjustments and they got down to 29C - 30C. Of course it is now much louder than I would like since all the fans are running at 100%. I ordered a fan speed controller as I have no way to control them now. No crashes, but performance slowly sunk well below 250 Mbps. I was never keen on UnRAID anyway due to the cost and I don't want to run the OS off a flash drive.
I decided to give "Storage Spaces" a try, so I installed Windows Server 2019 and reinstalled the Intel NIC. I enabled onboard RAID and put the two SSD's into RAID 1 (at least it's something) for the OS. I was able to get the drives into a "Storage Pool" in Storage Spaces, but every... single... thing I tried, I could not get a Virtual Disk to create in the pool, even a 250 GB one. WTF?! I did some searching on google, only to read other people having the exact same issue and lots of people trash talking Storage Spaces so I quickly gave up on that idea. I deleted the pool and used diskpart and cleaned all the drives at that point.
A "light" went off at that moment. I had previously ran FreeNAS in Hyper-V without any issues, so I installed the Hyper-V Role and created a VM For FreeNAS. 24GB Ram, 128GB Boot Drive, 4 vCPU. I went into Windows Disk Management and set all 8 drives to "offline" and added all 8 of them to the VM as a direct passthrough. Sure enough, it worked flawlessly. FreeNAS had direct access to the drives and pulled all the info of the drives (SMART Data, serial number, make/model, RPM speed, etc). I repeated the process as I did before and started a file transfer and monitored it.
850-900 Mbps average transfer speed with peaks over 980 Mbps. The slowest I saw was around 700 Mbps but nothing lower. Rock Solid transfer rate and performance and no reboots.
So, this will be the setup I'm going to stick with. Win-Win. The OS Drives are in RAID, I can use snapshots and windows server backup to backup the 128GB boot drive of FreeNAS should anything happen. Since Hyper-V "emulates" a 10 Gb Switch/NIC, I can attach these assorted SATA drives I have laying around with data on them via USB 3.0 and map the drive on the host and I won't be limited by Gigabit Gear trying to transfer files over the network.
Just thought I'd share my "success" story. I know it's not "ideal" but it's working and FreeNAS has direct access to the drives and so far has had the best performance stability.
