SOLVED Terrible network performance with Reading

[Rainwulf]

This has only just cropped up... i have the freenas box beside me and i was doing some tests, and i started a large file transfer to a machine with an SSD..
I could hear the drives working on the NAS, like seriously hunting. Then i checked my transfer rate:

Can anyone explain this? I honestly dont know whats going on. The machine has been fine since I installed it.

This is with the seagate archive 8tb drives, which have known write issues, but no read issues. This is just.. crazy. The transfer actually STOPS in some cases.

 

[DifferentStrokes]

Do you do regular scrubs? Do you run regular smart tests? If so, any errors?

 

[Rainwulf]

Scrubs are every 35 days, smart tests every month. No errors at all in anything. Its a 92tb pool, with only 16gig of ram, so i have shut it down and put in 32 gig.

 

[Rainwulf]

Doubling the ram just doubled the time it ran "Fast" until it went down to the terrible transfers.

I started a scrub, and its scrubbing at 932meg a second.. what the hell? It can scrub that fast, yet cant maintain a network transfer.

 

[Robert Trevellyan]

32GB of RAM seems a bit light for a 92TB pool. The rule of thumb is 1GB per TB after the 8GB minimum, which is less strict the more RAM you have. 64GB seems like a better match than 32GB.

 

[Rainwulf]

Well it was running on 16gb, and fine for weeks. Its only just recently that its gone down the drain.
The mobo i am running will support more then 32gig, so its the max for now.

 

[Rainwulf]

[root@Riva] ~# zpool status
pool: Riva
state: ONLINE
scan: scrub in progress since Thu Aug 27 11:27:55 2015
488G scanned out of 34.3T at 899M/s, 10h57m to go
0 repaired, 1.39% done
config:

NAME STATE READ WRITE CKSUM
Riva ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/3a8b7536-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3ae5357e-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3b35c807-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3b8c1702-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3be118f1-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3c347927-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3c847940-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3cdab336-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3d2d015b-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3d7d371a-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3dd3ee74-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3e267bfa-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3e79c9d2-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3eccb2f1-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3f1d4f38-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0
gptid/3f741f79-3587-11e5-89e8-001e6777aeb9 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
ada0p2 ONLINE 0 0 0

errors: No known data errors

 

[Robert Trevellyan]

Well it was running on 16gb, and fine for weeks. Its only just recently that its gone down the drain.

This is anecdotally what happens when you don't have enough RAM for your pool size. Everything is fine for a while, and you keep adding data, then you hit a brick wall.

 

[Rainwulf]

But with reading? Writing i can imagine, but reading should be fine, especially considering that it can scrub at nearly 1tb a second.

 

[Robert Trevellyan]

Also, 16 is generally considered too wide for any RAIDZ vdev. Consider splitting that vdev into 2.

 

[Rainwulf]

Bit late for that now haha. I have no alternate storage for the data.

 

[Robert Trevellyan]

Bit late for that now haha. I have no alternate storage for the data.

I suggest you figure out a way to do it now before the hole you've dug gets any deeper.

You might find this article interesting.

 

[Rainwulf]

Mmm. something doesnt feel right. I am running a scrub now, and now all transfers are maxing out the gigabit link.
Me-thinks something is afoot.

I did actually read that article, and yea, i took it onboard, but i decided on the fact that this thing will have very little traffic. Its just local mass storage. I dont need mulitple users or high IOPs.

 

[Darren Myers]

what are the hardware spec's to be specific? As suggested more RAM, and changing the amount of drives per vdev as that can become a slippery slope

 

[Rainwulf]

Xeon 1220V3, Intel S1200BT, 32 gig 1600 DDR3 ECC, crossflashed H200A's as 9611 IT, 16 8tb archive drives.

 

[Darren Myers]

oh boy....you may have 2 issues, 1 being the amount of RAM, and 2 being the fact theyre archive drives... there was a thread somewhere about the speed and overall performance of them and i dont think they preformed well at all

EDIT: https://forums.freenas.org/index.php?threads/seagate-8tb-archive-drive-in-freenas.27740/

 

[Rainwulf]

Hah, i was a part of that thread as im the first person i think who has 16 of these things in an array here :D

They have dismal write performance, but read is apparently perfect. And as the scrub is still happily purring along, making file copies speed up, i might have a few other things to look out for.
I have also been screwing around with VFS modules in samba, so i hope that hasnt broken anything.

 

[Rainwulf]

Is vfs_aio_fork a default VFS module in samba?

 

[SweetAndLow]

You should seriously think about rebuilding this pool. You should split it into two vdevs or 8 drives in raidz2. You should also get 64 if not 96GB or memory.

 

[Rainwulf]

I am thinking about rebuilding, but if you had of read up on my comments, the motherboard i have is limited to 32gig of ram.