Transfer vs Streaming Performance

[QuantumArc]

Hello, I'm new to the freenas and linux environment, and I've been playing around for the past two months with it all. I run a small post production facility and we just upgraded our storage space with 8 new 10TB WD Gold's. We had been running a mac mini server as our storage array with 8x 4TB drives the past 2 years. I have been pushing for freenas storage, so we built a i7 machine with 16gb ram and a mellanox connectx-2 10gb dual port NIC. We had been able to achieve over 1+GB/s reads and 900MB/s writes when testing the drives before implementing them into the freenas box, the issue was the SMB 3.0 performance in Mac is terrible, only being able to achieve a consistent 300-350MB/s transfers from our windows client. Our Mac clients had no issues, but our primary workstation is on Win10.

I ran a CrystalDiskTest over the network on the WS with the freenas and achieved a 845MB/s Read and 500MB/s write in sequential on the FreeNAS array. When I transfer over the network I can achieve those speeds on files greater than the 16GB. However if I try to stream the files from the array, I don't reach those speeds, only about 200-350MB/s and I'm needing to hit those speeds for the 8K video files I'm trying to stream. Does anyone have any thoughts as to why streaming vs transfer performance is different, looking for some education.

I have already followed the 45drives streaming 5K 10gbnic enhancements guide. Is there anything else I need to do?

 

[kdragon75]

So you didn't bother with ECC in a production environment? I hope your at least did a memtest. Also have you burned in the hard drives using badblocks? What kind of RAIDz did you setup? What do you mean by "Streaming" the files?

Sorry for all the questions, I'm just trying to cover your bases for you. A dead NAS and lost data could seriously impact your business.

 

[Stux]

Macs use sync writes via SMB (I believe)

You either need to disable this in Samba, disable sync on the dataset or get a SLOG. You should be able to test if this is the issue by simply disabling sync on the dataset (temporarily)

You should understand the implications before you make a decision.

 

[QuantumArc]

kdragon75, I had to work with what I had at the time, we're a small shop and experiencing these growing pains. I did not have a CPU or board that supported ECC. I am not very up on the steps to properly setting something like this up. To your other questions, I did not do a memtest or use badblocks. It is in a RaidZ1 setup since capacity and speed are important here. We have other backup methods in place for our data, LTO tape, etc.

What I mean about streaming files, is we get fast transfers, but when accessing the stored data over the network via our editor, it only streams at about 200MB/s and with some of our 4K and 8K files we need that 700-1GB/s speed. Before when we were on the Mac Mini, we were at least getting full 300MB/s avg which did work for a bit of what we were doing. If we're not able to get those high speeds through the network. We'll have to come up with some sort of high capacity speedy cache at each workstation.

stux, i'm not sure what you're talking about. What are the implications of what you're saying to do?

 

[anodos]

Try setting the following auxiliary parameters for your SMB share:

Code:

aio read size = 1
aio write size = 1

 

[HoneyBadger]

stux, i'm not sure what you're talking about. What are the implications of what you're saying to do?

Your Mac clients are now attempting to issue synchronous writes to your SMB share; this means they'll write a chunk of data and then wait to "hear back" from the storage to confirm that it was placed onto the disks and not held in a buffer.

This is fine for things like Hyper-V using an SMB 3.0 share - but for general file use it isn't necessary.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

 

[anodos]

Your Mac clients are now attempting to issue synchronous writes to your SMB share; this means they'll write a chunk of data and then wait to "hear back" from the storage to confirm that it was placed onto the disks and not held in a buffer.

This is fine for things like Hyper-V using an SMB 3.0 share - but for general file use it isn't necessary.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

That's a good point. OS-X SMB clients _always_ send an SMB2 flush request after writing a file (i.e. it sends a request to flush to stable storage). When strict sync is enabled, this results in an fsync. Another thing to love about Apple. This can impact write performance.

 

[HoneyBadger]

Another thing to love about Apple.

"You're holding saving it wrong."

 

[anodos]

"You're holding saving it wrong."

I believe that windows used to do this for FAT32 backed storage. Was an issue for windows 98 clients and samba (the reason why the defaulted to having strict sync off in the first place). So Apple is rocking the SMB like it's 1999.

In theory if you want to honor flush requests from everything but OS-X you could create an include file with macro-magic and make this happen.