Slow write speed on fast hardware. SMB/CIFS bottleneck?

[helloha]

Gotcha.

I've got another 16GB of memory arriving tomorrow. Lets see if that improves it any further.
I'm pretty happy with 700MB/s ish each way so I'll put this into production soon.
Is there anything else you want to try?

Too bad you are on the latest X10 platform. You can get used DDR3 ECC memory for next to nothing these days.
I'm a video editor and use my setup to edit straight from the server. No ssd, just mechanical storage.

If memory price is an issue I would certainly consider swapping the motherboard to X9 to get DDR3. You can get 96GB kits for €200... And like mentioned above ZFS starts to do crazy things if you throw lots of memory at it...

I can get 500-600 MB/s on average whith the pool described below.

Using cheap Chelsio T3 cards and SFP twinax cables!

K.

 

[Rand]

Do you have a link to that cheap memory? :o

 

[jgreco]

Last I looked (recently) 128GB 8x16GB of used 1600 ECC DDR3 for the Supermicros was about $600 ($75/stick) which isn't that much less than the $850 we paid new for that back in 4Q2012.

Meanwhile 32GB 2133 DDR4 sticks (M393A4K40BB0-CPB) have been around $180 with prices slowly falling, which is not that much of a premium over the DDR3.

 

[helloha]

16GB dimms are still expensive, 4GB are dirt cheap but not so interesting unless you have 16 slots. 8GB is the sweet spot for me.

I think memory density is outgrowing disk storage in terms of needs for ZFS is it not? I think you could live on DDR3 for quite some time unless power consumption is an issue. These new X10 systems support TB's of memory and prices and development of hard disks seems to have slowed down in the last 2 years. My 4TB drives almost carry the same price tag as they did two years ago.

When talking price I was referring to used parts from ebay, I regularly see listings like this:

http://www.benl.ebay.be/itm/HP-96GB...994444?hash=item210c57b18c:g:o6IAAOSwxcRW9tEm

http://www.benl.ebay.be/itm/64GB-8x...542180?hash=item3d17eef864:g:rWcAAOSw~oFXFdsw

 

[jgreco]

Well yeah 4GB are cheap but that's because they're slot-stuffing trash. They're cheap because nobody wants them.

You probably don't need dual CPU's for FreeNAS, so that means you really only get 8 slots to work with.

If you fill 8 slots with 4GB sticks you end up with 32GB, which would only be useful if you had a small number of disks but a need for lots of CPU (Plex jail?)

If you fill 8 slots with 8GB sticks you end up with 64GB, which matches a maximal Skylake configuration. At $68 a stick that works out to $544. The cost for 4 x M391A2K43BB1-CPB is around $600. That's not a real big savings.

I don't see tons of value there. I'm certainly fine with buying older generation stuff because the ""improvements"" over time are relatively modest, but I start feeling just a bit nervous as that involves larger amounts of cash spent.

 

[helloha]

Well I would say it depends on how hard you are willing to look for compatible memory.

You could spend $68 on a listing that has "supermicro" in it's title

Or you could look up the part number on the HCL from your motherboard and find it for €25:
http://www.benl.ebay.be/itm/Hynix-8...420145?hash=item25ba3e12b1:g:iLsAAOSwKfVXD8gp

That is €200 for 64GB or almost 1/3 of the price.

 

[jgreco]

I would still view 8GB modules as slot stuffers, and those are 1333 modules anyways. The proper memory for X9 kit is usually considered to be 1600, even if you can technically put DDR3-800, 1066, or 1333 in there.

 

[philhu]

What sort of tuning has been done to it on the 10G side?

The X540 isn't thrilling but can probably be made to work fine.

Can I inflict either pain or pleasure? Try adding the following for tunables.

kern.ipc.soacceptqueue=256
kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.recvspace=4194304
net.inet.tcp.recvbuf_inc=4194304
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendspace=4194304
net.inet.tcp.sendbuf_inc=4194304

Bet it flies. Or crashes. Maybe both. Might also need to stab CIFS with some adrenaline, I don't recall if CIFS tries to manipulate the socket settings itself or not. More memory will help. Shoot for 64GB, 128GB is pricey (~$800) but a nice performance booster.

Will these settings break if I have multiple nics? I do not see these per nic, looks global

I have the ix0/1, em0/1 and igb0/1 in my system

em0/1 is on a different subnet for backups.igb0/1 is not used
ix1 is my main 192.168 line.

Also, where exactly do these values go? in the gui?? Freenas almost-noobie here.

I tuned the PC/windows side
FYI-did dd test shown here, to raidz3 array, got 3.2 GB/s internal to box.
My read from NAS to pc is a steady 400+ mb/s now from raidz3
My write is 140-160 mB/s with spikes to 400 mB/s

Thanks!

 

[philhu]

Will these settings break if I have multiple nics? I do not see these per nic, looks global

I have the ix0/1, em0/1 and igb0/1 in my system

em0/1 is on a different subnet for backups.igb0/1 is not used
ix1 is my main 192.168 line.

Also, where exactly do these values go? in the gui?? Freenas almost-noobie here.

I tuned the PC/windows side
FYI-did dd test shown here, to raidz3 array, got 3.2 GB/s internal to box.
My read from NAS to pc is a steady 400+ mb/s now from raidz3
My write is 140-160 mB/s with spikes to 400 mB/s

Thanks!

Anyone answer this? Those 10gbe tuning numbers, are not card specific, so setting these, will it break my other cards use?

 

[philhu]

ok, bit the bullet, did these tunables on my freenas

Here are the results.....Copying 2+ gig files back and forth. Quad Core win10 machine, FreeNAS on a Supermicro chassis and x8DTN+ mb

Before win optimizations, no Freenas tunables. Everything set to standard, orig 1 g NIc (For control). The 112 listed was pegged, 111-112 for entire copy of file. No wavering

From PC to NAS 112 NAS to PC 80

Using x540-t2 card in NAS, and PC (1 card in each), no optimizations

From PC to NAS 130-140 NAS to PC 300

Optimize win10 using buffer optimizations

From PC to NAS 130-206 NAS to PC 400

Add FreeNAS optimization, here is where it gets wonky

1 specific file, reproduceable, always does this performance
From PC to NAS 616 NAS to PC 300-400

All other files, and I have verified this
From PC to NAS 39-75 NAS to PC 300-400


So I could not believe it, but I have 1 2+G video file, that transfers to NAS at 600+ every time!. All others go well below normal, in the 30-85 range. The only thing I can think of is compression, but I did not turn it on in either place.

I did notice, that I can send 5-10 files as seperate copies, and each gets these numbers, so no one gets huge copy performance, but each gets the same numbers, so the bandwidth is there.

edit-just ran tests with both optimized again and am seeing 90-140 today. Not stellar but better. Whats up?

Any comments on this? I am only user on this home NAS, nothing else is running

 

[Chris Moore]

edit-just ran tests with both optimized again and am seeing 90-140 today. Not stellar but better. Whats up?

Did you ever find an answer to this?