[SOLVED] Slow resilver performance in HP microserver Gen8

[m4rv1n]

Hello,
I lost one disk and, after change it, the resilver started.
The starting speed was really poor, about 30MB/s, after a few hours, the speed increased until 55MB/s and finally stabilized about 48MB/s.
The disk, in low level disk format, stay constant at 178MB/s , so the 30MB/s or 50MB/s seemes slow for me.
When resilvering, the storage have not been accessed from network except for console.
With gstat, the new disk is busy between 70% and 96%, the other disk are really idle (every few seconds are busy at 60% for 1 second and after return to 4%).
What I can do to solve this issue?
Thank you.

 

[Darren Myers]

Can you please include your system specs so we can better assist you.

 

[m4rv1n]

HP microserver G8 with G1610T CPU. Only modify RAM and now it have 16GB ECC. The new disk is ST3000VN000 (Seagate NAS 3TB).

 

[enemy85]

Maybe that CPU is your bottleneck?

 

[Starpulkka]

-are you using zfs or ufs?
-if zfs
-is compression set?
-if compression is set what mode is it?
-else set compression on and touch your files by moving (before move double check is all hardware ok)
-you pool is going to be fast as your slovest hdd in it, so look if you have another hdd seek errors or hardtime to keep up..or a bad sata cable..
-is files big or tiny?
-what zfs version?
-what pool version?
-what nas software version?
-what is cpu usage while scrub or resilver?
-how fragmented pool is?
-how full pool is?
-im not see what raidz you are using, are you using raidz3 ?


when you get 500M/s compressed scrub speed you doing ok for a micro hardware.

heres some uncompressed dd speeds on other hp hardware maby you try dd?
https://forums.freenas.org/index.php?threads/hp-n54l-raidz2-performance.26254/#post-189377
maby that CPU is your bottleneck

Edit:
Weird i thought that in freenas 9 lz4 compression is on by default.

Code:

zfs get compression

ok what i try is test some dataset with lz4 compression and if its too much for cpu then i try good ol lzjb
(i personally not yet trust lz4)
and if compression helps then copy data to somewhere safe and recreate pool. Because if you put compression on and move files to that compressed dataset pool just might get way more fragmentation in process. Havend done move operation myself so im not sure does it create a lot fragmentation..
Or if moving data and destroying pool and stuff is too much to setup again, then perhaps next time when you create pool, you member test speeds.

 

[cyberjock]

Your CPU is lower than the minimum we recommend. So I can't say I'm too surprised it is slow. I specifically chose NOT to include your exact CPU for a bunch of reasons... and I knew it would result in unsatisfied end-users.

 

[m4rv1n]

cyberjock, I understand the recommend, but CPU was not really stressed when resilver worked, also if I try to write from samba share, that is very cpu aggressive, the speed is about 85MB/s .

I'm using zfs, 4 x 2TB disk in raidz, 1 dedicated disk for OS (this is slow disk but is only for OS), no compression. FreeNAS-9.3-STABLE-201504152200
All single hard disk are ok.
The file stored in the pool is different size, from few KB to 30 GB. Most space is occupied by big file (>100MB).
The pool is full about 50%, frag 6%.

VolRaidZ: version: 5000

zpool get version
NAME PROPERTY VALUE SOURCE
VolRaidZ version - default
freenas-boot version 28 local

zfs get -r version
NAME PROPERTY VALUE SOURCE
VolRaidZ version 5 -

WRITE TEST (bs=8k count=100k, cpu 52%)
[root@storage] /mnt/VolRaidZ/TEMP_# dd if=/dev/zero of=./zero bs=8k count=100k ;
102400+0 records in
102400+0 records out
838860800 bytes transferred in 2.268126 secs (369847529 bytes/sec)

[root@storage] /mnt/VolRaidZ/TEMP_# dd if=/dev/urandom of=./random bs=8k count=100k ;
102400+0 records in
102400+0 records out
838860800 bytes transferred in 21.984131 secs (38157560 bytes/sec)

READ TEST (CPU 48%)
[root@storage] /mnt/VolRaidZ/TEMP_# dd if=./zero of=/dev/null ;
1638400+0 records in
1638400+0 records out
838860800 bytes transferred in 6.023224 secs (139271059 bytes/sec)

[root@storage] /mnt/VolRaidZ/TEMP_# dd if=./random of=/dev/null ;
1638400+0 records in
1638400+0 records out
838860800 bytes transferred in 5.658469 secs (148248723 bytes/sec)

WRITE TEST (bs=2048k count=10000, cpu 37%)
[root@storage] /mnt/VolRaidZ/TEMP_# dd if=/dev/zero of=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 70.519836 secs (297384696 bytes/sec)

READ TEST (bs=2048k count=10000, cpu 36%)
[root@storage] /mnt/VolRaidZ/TEMP_# dd of=/dev/zero if=ddfile bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 51.685511 secs (405752397 bytes/sec)

What do you mean with "set compression on and touch your files by moving"? If I enable compression now, I will have part of pool uncompressed and part compressed.

 

[enemy85]

You can set compression even on single datasets. It doesn't matter. You should set compression ON (lz4) for the whole pool, even if just new written files would be affected. It would not hurt

 

[m4rv1n]

Ok, I will set compression on, note that the same data on the past freenas was compressed about 3%. But how much this affect resilvering performance? Ther'is a lot of difference between 48MB/s and the other sequential write test's speed.

 

[Starpulkka]

i also edited my previous post
well in my machines lzjb compression did a lot,
think it like that way that you can now put way more data in i/o
and if you test dd speeds in compressed dataset you might notice you push more data

and if you do zero dd test in lz4 you push a tons of data, zeroes compress really welll =P (last one is a joke, but if you try it you notice it)

 

[Starpulkka]

Ok, I will set compression on, note that the same data on the past freenas was compressed about 3%. But how much this affect resilvering performance? Ther'is a lot of difference between 48MB/s and the other sequential write test's speed.

yes that difference, i read it about somwhere in internet why it was like that, but i dont member anymore.
but im now going to sleep, its 23.00 here zzZZ

 

[m4rv1n]

well in my machines lzjb compression did a lot,
think it like that way that you can now put way more data in i/o
and if you test dd speeds in compressed dataset you might notice you push more data

and if you do zero dd test in lz4 you push a tons of data, zeroes compress really welll =P (last one is a joke, but if you try it you notice it)

As I say, the compression is about 3% for the data I use, so I have a lot of data I can not compress. I can enable it but the essential arguments is resilvering performance, other data is ok for me as test show. Did you had seen more I/O in resilvering?
If yes, can you give me some technical motivation why compression speed up resilvering?
Also without resilvering, the speed, compared to disk and dd speed, is really poor. So before enable/disable or other, is important to understand why is slow.

 

[Starpulkka]

yes i did get faster scrub and resilver speeds, and lz4 it does calculate is file worth to compress or not, so it should be win win eitherway

 

[Starpulkka]

Heres some printed data, dont take my word for it.

(does this pdf document work for tecnical motivation?)
https://www.duo.uio.no/bitstream/handle/10852/42152/thesis.pdf?sequence=7&isAllowed=y
search from that pdf document io performance,
or i recommend you to read it fully to late night reading.

havend read it myself, i hope pdf is accurate, going to read it tomorrow
yes i understand that enabling compression might not be the magic bullet you looking for,
but im more worried about your raidz1 setup

edit:2 Well you are telling that theres no slow hdd in pool, so im going to believe you. And drives are in AHCI mode (not sure is it safe to change ide to achi mode on intel board). You now have compression on so it should speed things more but only for new files, (how much more depends on files and your cpu process power)
i have one last thing what i have in mind that you have low power machine so maby you should check cpu MHz does daemon handle cpu MHz corretly or does it throttle (as your speeds vary). If you like you can disable powersavings and force it full speed, and then test does it improve scrub speeds..(i doubt that)
thats all what im going to say. Happy testing =)

 

[m4rv1n]

Hi Starpulkka,
Thank you for suggest.
I compress all dataset (move out and in all files after enable compression) and restart resilver of one disk.
The situation is this:

last pid: 6740; load averages: 0.02, 0.32, 0.51 up 0+00:26:28 21:18:28
44 processes: 1 running, 43 sleeping
CPU: 1.1% user, 0.0% nice, 7.4% system, 0.9% interrupt, 90.6% idle
Mem: 393M Active, 148M Inact, 9083M Wired, 4324K Cache, 2241M Free
ARC: 6912M Total, 6349M MFU, 520M MRU, 162K Anon, 20M Header, 24M Other
Swap: 8192M Total, 8192M Free

pool: VolRaidZ
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Apr 30 21:09:16 2015
66.0G scanned out of 3.56T at 101M/s, 10h4m to go
16.5G resilvered, 1.81% done

dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| fd0
0 0 0 0 0.0 0 0 0.0 0.0| cd0
0 0 0 0 0.0 0 0 0.0 0.0| iso9660/FreeNAS_INSTALL
0 0 0 0 0.0 0 0 0.0 0.0| da0
1 79 79 26965 4.1 0 0 0.0 26.2| da1
1 79 79 26965 2.6 0 0 0.0 20.0| da2
1 79 79 26969 1.9 0 0 0.0 14.3| da3
1 80 0 0 0.0 80 27309 12.2 97.6| da4

So the compression help a lot about speed, from 50M/s to 100M/s is the double . I found strange that the speed on the disk is slow (27309) and the disk is full busy (97.6) with this speed. This is a second disk (Seagate nas HDD 3TB) and the disk is new and tested (also speed). Same thing last time with other disk: on rebuild, the disk was full busy with slow write.

At the end of the resilver :
pool: VolRaidZ
state: ONLINE
scan: resilvered 898G in 26h16m with 0 errors on Fri May 1 23:25:16 2015

So very slow (919552 MB in 94560 sec. = 9,72MB/s).

 

[m4rv1n]

At the end I've changed the controller in microserver from b210i to pci-e cross-flashed lsi-2118it, the scrub speed is now about 300MB/s (max of about 398MB/s). Thank's to all.

 

[enemy85]

So the controller was a HUGE bottleneck in ur system

Inviato dal mio LG-D855 utilizzando Tapatalk