Slow copy from mac pro to FreeNAS

bigjiggity

Dabbler
Joined
Sep 8, 2019
Messages
10
slow copy from mac to freenas
renderTimingPixel.png

########### CONFIG #######################
i5/32GB ram/perc6 hba (passthrough)/usb key boot/240GB x2 (ZIL/L2arc)
Dell MD1000 2TB HGST 7200 SATA x 15
3 zvols, 5 disk per vol, raid-z2, striped, 1 pool
#########################################
created SMB and AFP share and attempted to copy ~666GB of raw photo's over to the share (not at the same time, indipendently tested each protocol). and the copy is insanely slow, ran fine for the first say 15 min. and then fell on it's face. RAM/CPU/L2arc all seem to be good and running as expected... you can see the attached graph screenshot where the disk just falls on it's face, and it's picked back up, but the est. time to completetion went from like 8hr. to over 24hr.

Network seems fine, both systems are plugged into 1GBe ports on same switch. iperf reports good connection bidirectionally.

Client connecting to 192.168.1.100, TCP port 5001
TCP window size: 129 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.1.8 port 59638 connected with 192.168.1.100 port 5001
[ 6] local 192.168.1.8 port 5001 connected with 192.168.1.100 port 26912
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1023 MBytes 858 Mbits/sec
[ 6] 0.0-10.0 sec 800 MBytes 670 Mbits/sec


see graphs for other stats comming from freeNAS. someone with a bigger brain please explain what I am missing.
thanks!
Screen Shot 2019-09-08 at 6.22.28 AM.png

Screen Shot 2019-09-08 at 6.44.36 AM.png

Screen Shot 2019-09-08 at 6.27.26 AM.png

Screen Shot 2019-09-08 at 6.22.45 AM.png
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Left out was the actual transfer speed... what is it? Mac OS can be wildly optimistic re: transfer time. If I'm doing the math right, I get ~8MB/s, which is admittedly very low. The initial speed of ~24MB/s also seems low given the format being used (large media files).

Are the the ZIL and L2ARC on a SSD? If so, what is the type? I wonder because the ZIL is very sensitive to write speeds under sustained use. There is a reason that Intel can charge the wild prices for Optane that it does...

Another advice I have seen here is trying the same transfers with sync=disabled to see what the raw write speed is. If the speed pops up to the level I would expect (i.e. somewhere around 100-120MB/s) then that suggests a configuration / ZIL / etc. issue.

My system with a single VDEV and a Z3 configuration manages about 120-170MB/s with large continuous writes. This is expected as each VDEV pretty much behaves like a single hard drive. So, your 3 VDEV system should be easily capable of saturating a gigabit connection.
 

bigjiggity

Dabbler
Joined
Sep 8, 2019
Messages
10
Left out was the actual transfer speed... what is it? Mac OS can be wildly optimistic re: transfer time. If I'm doing the math right, I get ~8MB/s, which is admittedly very low. The initial speed of ~24MB/s also seems low given the format being used (large media files).

Are the the ZIL and L2ARC on a SSD? If so, what is the type? I wonder because the ZIL is very sensitive to write speeds under sustained use. There is a reason that Intel can charge the wild prices for Optane that it does...

Another advice I have seen here is trying the same transfers with sync=disabled to see what the raw write speed is. If the speed pops up to the level I would expect (i.e. somewhere around 100-120MB/s) then that suggests a configuration / ZIL / etc. issue.

My system with a single VDEV and a Z3 configuration manages about 120-170MB/s with large continuous writes. This is expected as each VDEV pretty much behaves like a single hard drive. So, your 3 VDEV system should be easily capable of saturating a gigabit connection.

the derp is real....
Screen Shot 2019-09-08 at 8.53.39 AM.png

zil/l2arc are both ssd - nothing special though, Sandisk 240GB. - where do i set the sync=disabled setting?
 

bigjiggity

Dabbler
Joined
Sep 8, 2019
Messages
10
These were run from my mac on the mounted volume.

Code:
**Sync Off**

[/Volumes/storage-1:]dd if=/dev/zero of=testfile bs=1024 count=500000
500000+0 records in
500000+0 records out
512000000 bytes transferred in 7.855533 secs (65176991 bytes/sec)
----------------------------------------------------
[/Volumes/storage-1:]dd if=testfile of=/dev/zero bs=1024 count=500000
500000+0 records in
500000+0 records out
512000000 bytes transferred in 6.287553 secs (81430724 bytes/sec)

** Sync Always**
[/Volumes/storage-1:]dd if=testfi/dev/zero of=testfile24 count=500000
500000+0 records in
500000+0 records out
512000000 bytes transferred in 6.475153 secs (79071488 bytes/sec)
----------------------------------------------------
[/Volumes/storage-1:]dd if=/dev/ztestfile of=/dev/zero24 count=500000
500000+0 records in
500000+0 records out
512000000 bytes transferred in 6.239406 secs (82059095 bytes/sec)
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Your flaw is in the testing... all zeroes will not give you a realistic test of disk writes as it will be highly compressible.
 

bigjiggity

Dabbler
Joined
Sep 8, 2019
Messages
10
Your flaw is in the testing... all zeroes will not give you a realistic test of disk writes as it will be highly compressible.


That is true... was a quick and dirty... valid test was moving the original data, and it crawled. 666GB of camera raw files, over 48hrs?! Something is no bueno
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
You might want to try cutting out the SLOG as the first step. Do you have separate SSDs for SLOG and L2ARC?
 

bigjiggity

Dabbler
Joined
Sep 8, 2019
Messages
10
You might want to try cutting out the SLOG as the first step. Do you have separate SSDs for SLOG and L2ARC?

Yep separate ssd’s for each... removing the SLOG... orly? Huh, why? Any kind of caching I would have thought beneficial...
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Not necessarily. It depends in part on the use case as well as the SLOG device itself.

For example, some inexpensive SSDs combine fast, small volatile caches with slower permanent Flash in the back, allowing for “fast” writes until the volatile cache is full. Then things slow down dramatically.

That’s why I asked you what you use as a cache. If it’s not power protected by design (and a UPS doesn’t count) then I would not use it. See the ixSystems discussion re suitable SLOG devices. Servethehome has a great series on what makes a good SLOG and suggested devices also.

Anyhow, I would turn on and turn off sync (disabled vs. always) and then repeat your transfer tests with actual data (let’s say a 5GB collection of representative stuff). See the manual on how to do that for your pool.

I’d retry same with the SLOG enabled and disabled for the “always” case. So three tests total. Time them and report back. I’d also disable the L2ARc completely for all three.
 

William Bravin

Contributor
Joined
Mar 16, 2016
Messages
195
Hello all

I find this discussion pertinent to my issue. I have 2 servers and when i copy from on to the other or when i do an rsync task i normally see speed of 12 to 15 Mbs . Both machines have giga networks and they are connected to the same switch also giga enabled. I have no idea of what slog or l2arc are or what they do or where they are.

Some time I see speeds of 65Mbs but that is rare

What can I do to improve my situation

Thank you
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
There are a number of online resources re: ARC, L2ARC, and the ZIL/SLOG. I'd start with this one, and continue on to this one next.

You're likely to see the fastest speeds transferring when you're dealing with large, continuous files like video. Small files will take longer, presumably due to overhead, directory writes, and so on. Rsync does a lot of directory checking, which is where an L2ARC can come in handy. Per the recommendations of the demi-gods here, I would not install a L2ARC until you have at least 32-64GB of RAM to play with, as L2ARC will also gobble up some ARC RAM in the process.

Per my tests, I found significant benefit to use a L2ARC in conjunction with rsync, most likely due to the extensive directory browsing by rsync and my largely dormant data set. Currently, only metadata is written to my L2ARC, which speeds up directory browsing particularly.

All that said, I'd try out the suggestions laid out above to see where the potential bottleneck is. I'd be very wary of powerline-based ethernet transmissions.
 

William Bravin

Contributor
Joined
Mar 16, 2016
Messages
195
HI, Thank you very much for this response. it will give me a lot to think about.

As you can see from the sig, i have very minimalistic servers. Only on the backup server do i have 24GB ram and i have an additional 1tb wd red installed as a buffer, here freenas is on an SSD. I use this server once a week to act as a redundant server

If i understand correctly what i read on your links,

I would need to
  • make my backup rig my main server,
  • Increase the ram to at least 36GB
  • replace the wd drive with an SSD and then have L2ARC use this SSD.
In addition the 2U server is very loud (this is why i use the ausus server as my main rig. it is much much quieter and i could have it in my office).

Once again many thanks
 
Top