Strategy to increase burst 100GB write speed

[DBordello]

I have a Ubuntu server connected to TrueNAS core via 10Gb/s ethernet utilizing NFS.

The pool is 4 zdevs capable of ~150MB/s writes for a total of ~600MB/s. The TrueNAS server has 32 GB of RAM.

The server writes in bursts of ~ 100GB files from a fast M.2 SSD. My goal is to expedite this write operation (ideally saturate the 10GB/s link; future proof for a 25GB/s link). The files are not very important, so I am not worried about data loss prior to it being written to disk.

A couple quick thoughts:

1) Increase RAM to 128 GB. Would TrueNAS cache the write? I know NFS is supposed to be async, but I haven't been able to confirm this.
2) Add a M.2 SSD SLOG.
3) Configuration change?

I'd love to hear your thoughts!

 

[jgreco]

Increase RAM to 128 GB. Would TrueNAS cache the write?

TrueNAS always caches writes. More RAM may improve write speeds, but not necessarily because of write caching.

I know NFS is supposed to be async, but I haven't been able to confirm this.

Why do you care? Just make it async and be done with it. You already said the data isn't super important.

Add a M.2 SSD SLOG.

I can't imagine what this would be. Perhaps you are unaware that adding ANY SLOG will ALWAYS be slower than the same pool set for async writes. And I can't think of any M.2 SSD's that would be appropriate SLOG, except perhaps for some hard-to-find Optanes.

You should definitely check out the 10 Gig tuning resource I posted recently.

 

[DBordello]

Why do you care? Just make it async and be done with it. You already said the data isn't super important.

I do care, I would prefer async :) However, I am having a hard time confirming my writes are actually async.

I'll check out the 10 Gig tuning resource!

 

[jgreco]

However, I am having a hard time confirming my writes are actually async.

Again, why do you care? Just set it to async. What are you going to do if you somehow manage to confirm that it isn't async even though you've set it async? You pretty much need to trust it unless you're going to go spelunking into the ZFS code. Set it async and just trust that this isn't a compsci homework assignment written by some sophomore college kids.

 

[DBordello]

I don't trust the user, not the developer :)

I don't see an option in the gui to set NFS to async. Can you please point me in the right direction?

 

[jgreco]

I don't trust the user, not the developer :)

I don't see an option in the gui to set NFS to async. Can you please point me in the right direction?

I imagine it's something like

truenas# zfs set poolname/datasetname sync=disabled

 

[DBordello]

I imagine it's something like

truenas# zfs set poolname/datasetname sync=disabled

Fantastic, thanks!

I am going to toss some more ram at the problem and call it a day.