Best way to skin a cat? (how best to use the drives I have)

[jmcguire525]

First time build and I've been reading, testing, and learning as much as possible but I'm stuck on what to do here... I've got a 6x8tb RaidZ2 volume, 8 total sata ports (2 available), and an m.2 nvme slot. I originally planned to use a ssd for the boot drive, and ssd for a SLOG drive, and an nvme ssd for jails... but I'm not sure that is best.

First off I am trying to maximize my NFS write speed. Without a SLOG drive, sync=standard ~30MBps, sync=disabled ~70MBps. I tested using the nvme drive for SLOG and got ~60MBps standard and ~78MBps disabled... This is writing from a W10 Pro computer so the implementation of NFS on it may be slowing things down as well, if you have any tips for speeding things up let me know.

The storage is almost completely comprised of media, NVR footage, and other non-vital files... losing the array would be a disappointing but having one of two writes fail because of a SLOG drive wouldn't bother me much. That being said, what is my best option?

1. ssd boot drive, nvme jails, ssd SLOG
2. ssd boot, nvme SLOG, ssd jails
3. ssd boot, mirrored nvme SLOG, ssd jails
4. usb boot, mirrored ssd SLOG, nvme jails
5. I also have 64GB ecc ram, should I consider using a ram drive for SLOG?

Also, I realize that endurance is important for the SLOG drive, how important is it for the drive that holds my jails?

 

[tvsjr]

I think you need to do more reading on SLOG in general. First, do you have a huge number of sync writes? If not, why do you need a SLOG at all?

If you do believe you need it, SLOG is all about speed and safety. You need a device that is protected from power loss, and you want the absolute highest speed you can get. A RAM drive for SLOG is a bad idea - you may as well run sync=disabled, as a power failure will have the same result. SLOG also doesn't need to be mirrored. So, if you feel that SLOG is necessary, a single PCIe SSD device is your best option.

As far as endurance for your jails drive... that depends entirely upon what you're running.

 

[jmcguire525]

I think you need to do more reading on SLOG in general. First, do you have a huge number of sync writes? If not, why do you need a SLOG at all?

If you do believe you need it, SLOG is all about speed and safety. You need a device that is protected from power loss, and you want the absolute highest speed you can get. A RAM drive for SLOG is a bad idea - you may as well run sync=disabled, as a power failure will have the same result. SLOG also doesn't need to be mirrored. So, if you feel that SLOG is necessary, a single PCIe SSD device is your best option.

As far as endurance for your jails drive... that depends entirely upon what you're running.

I do need to read up on things more, and have been trying to take in as much information as I can quickly so that I can get the server up and running. I'm probably going overkill on something things but I want the system to be fairly reliable for the next 3-4yrs.

My slow NFS writes (trying to move everything from a single mycloud) is what pushed me towards using a SLOG, its not necessary but I read that with leaving ZIL on the pool incurs more writes and therefore would be wearing out my main pool faster. Even with without a slog I would think the RaidZ2 pool would write faster than ~30MBps, that's something I obviously need to figure out whether its due to the W10 NFS or something else.

Either way, after reading around more today I think I would be best adding a 16GB stick of m.2 Optane memory as my SLOG, its a relatively cheap solution and should benefit me even if I figure out my slow transfer speeds.

A few questions though...

1. Without a SLOG what are normal NFS write speeds for a 6 drive RaidZ2?
2. Is the 16GB Optane drive a good choice for me?
3. I want to minimize the writes to my jails ssd and use a separate ssd for plex transcoding and possibly another ssd for incoming nzb dowloads (final unpack to the main pool). Am I thinking along the right line here assuming that would minimize the wear on my storage volume and jails volume, making it more likely that i'd lose one of the other ssd drives before anything else?

 

[tvsjr]

I can't help you with the jails stuff. Personally, I think a NAS should be left alone to be a good NAS... if you want to run plex, NZB, all that other stuff, do so elsewhere. I have a 3-node, 48-core vSphere cluster for all that other stuff.

Why are you running NFS to your Win10 box, versus SMB shares?

My 6-disk RAIDZ2 pool runs around 360MB/sec for writes, direct from the FreeNAS console, with compression disabled.

 

[jmcguire525]

I can't help you with the jails stuff. Personally, I think a NAS should be left alone to be a good NAS... if you want to run plex, NZB, all that other stuff, do so elsewhere. I have a 3-node, 48-core vSphere cluster for all that other stuff.

Why are you running NFS to your Win10 box, versus SMB shares?

My 6-disk RAIDZ2 pool runs around 360MB/sec for writes, direct from the FreeNAS console, with compression disabled.

I chose NFS mainly for 4k streams using Kodi or Infuse, I've seen people have performance issues with SMB shares and now that I can backup most of my 4K BD's I thought It would be best.

As far as the Jails, I'm also using the server with CableCard tuners for a whole home DVR. My family plans to remotely access it a good bit as well so between that and plex there will be a good bit of writes to the transcode locations depending on their connection/devices direct play ability.

 

[tvsjr]

Properly configured, you should have no issues with running multi-gigabit traffic over SMB, and Windows supports it natively. You also didn't post your full hardware configuration, as required... if you're talking multiple simultaneous 4K streams, plus all of the recording going on, I hope you have a substantially beefy box.

 

[jmcguire525]

Sorry about that... The server has a Xeon E5 2683v3 with 64gb DDR4 ECC ram. I have two GbE ports (as of now I am only using one) reading up on aggregation is next on my list, not sure if I need to set that up in bios or freenas or both.

 

[toadman]

A few questions though...

1. Without a SLOG what are normal NFS write speeds for a 6 drive RaidZ2?
2. Is the 16GB Optane drive a good choice for me?
3. I want to minimize the writes to my jails ssd and use a separate ssd for plex transcoding and possibly another ssd for incoming nzb dowloads (final unpack to the main pool). Am I thinking along the right line here assuming that would minimize the wear on my storage volume and jails volume, making it more likely that i'd lose one of the other ssd drives before anything else?

1) It depends entirely on the NFS client as well as the server. My windows 10 box is slower on NFS that SMB if I use it's native NFS client. The suggestions to use SMB are spot on. I think you'll find it will clearly out perform NFS from windows. And now you can utilize multipath SMB if you have 2 nics in the windows box. You should easily saturate a 1 gbps link. If you go multipath you will get up to 2x that depending on devices involved. No way you'll get that over NFS in my opinion.

2) I highly doubt you need an SLOG. You need to test the speeds with something other than Windows first. The workloads you describe would not be doing a bunch of sync writes as far as I can tell. You should monitor it on the server and see how many sync writes you are doing before deciding you need an SLOG. And i'm 99.9% sure you do not. I would go SMB first per 1) and see what happens.

3) While I understand why you want an SSD for jails (IOPS) you do not need one for downloading or transcoding. I can assure you you'll be fine putting that stuff on the pool. You will not notice any performance delta. It would be a waste of SSD space to use one in my opinion.

 

[jmcguire525]

3) While I understand why you want an SSD for jails (IOPS) you do not need one for downloading or transcoding. I can assure you you'll be fine putting that stuff on the pool. You will not notice any performance delta. It would be a waste of SSD space to use one in my opinion.

What about longevity of the drives in the pool, would it not be better to have all of those transcode writes going to a ssd that is cheaper to replace in the event of a failure even if I'm not seeing a noticeable performance boost?

 

[SweetAndLow]

Nfs on windows is going to always be the bottleneck. Your server can easily do 100MB/s transfers so there is no need for any kind of slog device.

 

[toadman]

What about longevity of the drives in the pool, would it not be better to have all of those transcode writes going to a ssd that is cheaper to replace in the event of a failure even if I'm not seeing a noticeable performance boost?

I doubt that amount of writes would have a noticeable effect on the HDD failure rate. But yes, one can argue SSDs are generally more reliable than HDDs (depending on type and size one can usually write several hundred TB and potentially 1+ PB of data to SSDs, i.e. years worth of data) they are still at a $/GB premium. So I guess it depends on your particular situation really. I was viewing the HDDs as sunk cost and the SSD as additional (and probably unnecessary) cost.

My point was just that you don't need SSD performance. :)