My short lived TrueNAS Experience

seantp

Cadet
Joined
Jan 3, 2021
Messages
3
I'm not necessarily looking for support since I've moved on to a Windows solution for the time being. Just sharing my experience that led me to switching to windows for this project. However, TrueNAS is on to something and I'll be following and testing releases in the future.

Gigabyte C246M-WU4CF
Intel Xeon E-2136
128GB DDR4-2666 ECC
8 12TB WD Red drives
2x 2TB Intel NVME drives
X540-AT2 10GB network card

I started this build with FreeNAS 11.3 and everything installed just fine and worked. However, the one problem I started noticing was copying my 30TB of data to the NAS resulted in periods of transfers dropping completely for a minute or two then starting again. Part of this was being copied from my workstation from NVME drives and the transfer rate was what I expected. 400-500MB/s except for the periods where the transfer rates dropped to zero. Some initial researched just lead me to believe this was just the ram cache filling and the system halting for the disks to catch up. Which made sense, though I thought it was a bit frequent.

However, there was a point when I was copying this data off spinning disks and it did the same thing. That didn't seem right to me. I didn't take the time to run it down and started to test my media with Plex and followed the deduplication stats. I was happy with it, but then started having issues with my workflow. Mainly moving large Bluray files from my workstation to the storage array over the SMB shares I had set up. Not only was I getting the same issues of pauses in file transfers but also errors where the transfer would just straight up fail with vague errors. Again, I did some research and while others were experiencing the same issues there weren't any real solutions provided in any of the threads I found.

After TrueNAS was released I decided to push the upgrade to see if it would resolve these issues. So after 4 days of backing up all the data I pushed the upgrade with zero issues. Then tested again and there was improvement. Moving these large 30- 80GB files worked marginally better. Instead of transfers dropping to zero the transfer rate would drop to about 30-40MB/s second then ramp up to ~150MB/s. At this point the errors had gone away too, but I felt this was too slow. Basically it would spend a lot of time transferring at 30-40MB/s second. Since the hardware requirements seem to be strict I just took it as an issue with my setup or some issue with the software which I didn't' want to spend a lot of time figuring out.

Decided to install Windows Server 2019 (because I can) just to see how it would compare on the same hardware. I used storage spaces and setup a tiered storage with the NVME and spinning discs. Then copied my data back to the system.

Obviously comparing the two is apples to oranges, and I did expect SMB to work better with Windows and it has advantage running on whatever hardware you throw at it.

That said, the experience was night and day. Immediately the transfer rates from spinning disk to spinning disk over the 10G connection was a very constant 200MB/s with zero pauses. Not only that I was able to push applications like qbittorent with it's random disk usage to file destinations directly to the network shares without the random network destination fails I was getting with TrueNAS.

Obviously, there is a MS edge with SMB which makes sense. It's just also a weak spot with TrueNAS from my experience. Not only that but I'm getting much better results with MS's deduplication. 13% vs .03% with TrueNAS.

Since I work in IT and I have the resources at my disposal, this is the solution for me at the moment. I will be playing around with TrueNAS in the future with different hardware and will follow it's progress. Especially with the datacenter converged solutions. That could be killer.
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,975
Deduplication works quite differently with ZFS and is generally not recommended for the average user. The performance hit is tremendous and it would not surprise me if it were the cause of your issues.
 

seantp

Cadet
Joined
Jan 3, 2021
Messages
3
Deduplication works quite differently with ZFS and is generally not recommended for the average user. The performance hit is tremendous and it would not surprise me if it were the cause of your issues.
That's interesting. Is that a process that would show up on the CPU usage graph in the UI. I never really saw any issues with CPU usage.
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,975

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,691
That's interesting. Is that a process that would show up on the CPU usage graph in the UI. I never really saw any issues with CPU usage.

Deduplication doesn't just require CPU, it also uses RAM and when there is not enough RAM, it requires a lot of disk I/O for metadata. While waiting for the metadata, it will be slow to handle read and write requests. The potential solution is a special SSD VDEV that accelerates the metadata access.
 

seantp

Cadet
Joined
Jan 3, 2021
Messages
3
Yes, after reading the thread I do think it was ram. I always wondered why most of my 128GB was always used up in cache even when nothing was really happening. That would explain a lot. Well, now I know for my next build.
 
Top