Resource icon

solnet-array-test Discussion Thread

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
This is the discussion thread for the solnet-array-test Resource. Please use the tabs above to navigate to it.
 

SilentStrike

Cadet
Joined
Nov 4, 2017
Messages
9
I have two questions for this...

First, here are my specs.
X9SCL-+F
E3-1240 V2
32GB ECC Ram,
6x 1TB disk
1x 3TB
1x 4TB
1x5TB
lsi 9211-8i
boot disks are 2x 120GB SSD
Running FreeNAS 9.10.2.U6

#1- How long should a run-through approximately take? Are we talking multiple days?
#2- Is it normal that my server reboots if I just do a little something on it while it's happening? I went to check reporting and it just kind of stopped and then initiated a reboot.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175

aadje93

Explorer
Joined
Sep 25, 2015
Messages
60
trying it on 12x WD Red 4TB (live pool) in 2 raidZ2 vdevs 6 disks each;
Code:
solnet-array-test-v2.sh: line 128: 49191 Terminated			  dd if=/dev/${disk} of=/dev/null bs=1048576 2> /tmp/sat.${disk}.err > /tmp/sat.${disk}.out


gives me this error at first, then

Code:
solnet-array-test-v2.sh: line 128: 50320 Terminated			  dd if=/dev/${disk} of=/dev/null bs=1048576 2> /tmp/sat.${disk}.err > /tmp/sat.${disk}.out



its still running now, but i suspec i get this error at every drive :(
 

rio236

Dabbler
Joined
Aug 19, 2016
Messages
38
This is the discussion thread for the solnet-array-test Resource. Please use the tabs above to navigate to it.
I lost connection to the server and so I cannot see the progress report.
I had completed the short test and just begun the long test which should have ended 10/3/2018 but it is still running.
Is there a way that I can connect and see what happening?
If I stop the test, is there a way for me to grab the report?
Thanks
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
You would have had to have used a solution like tmux to be able to disconnect and later connect to the same session.
 

Stavros

Cadet
Joined
May 2, 2015
Messages
3
@jgreco

root@freenas:~ # diff -p solnet-array-test-v2.sh solnet-array-test-v3.sh
*** solnet-array-test-v2.sh 2016-03-14 00:05:31.000000000 +1030
--- solnet-array-test-v3.sh 2018-10-07 12:17:14.842823320 +1030
*************** getdisksize() {
*** 55,62 ****
--- 55,67 ----
disksize=`egrep "^${1}:.* byte sectors.*" < /var/run/dmesg.boot | head -1 | sed -n 's/^[a-z0-9]*: \([0-9]*\)MB .*$/\1/p'`

if [ -z "${disksize}" ]; then
+ disksize=`dmesg | egrep "^${1}:.* byte sectors.*" | head -1 | sed -n 's/^[a-z0-9]*: \([0-9]*\)MB .*$/\1/p'`
+ fi
+
+ if [ -z "${disksize}" ]; then
disksize=0
fi
+
if [ "${disksize}" -eq 0 ]; then
echo "Unable to determine disk ${1} size from dmesg file (not fatal but odd!)" > /dev/tty
fi
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
What are you getting at? A diff with no context doesn't really clue us in on what is going on.
 

Stavros

Cadet
Joined
May 2, 2015
Messages
3
What are you getting at? A diff with no context doesn't really clue us in on what is going on.

Its to resolve the error
Unable to determine disk da16 size from dmesg file (not fatal but odd!)

In my case it's occurring because I plugged the disks in after boot.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Its to resolve the error
Unable to determine disk da16 size from dmesg file (not fatal but odd!)

In my case it's occurring because I plugged the disks in after boot.

... PEBCAK? :smile:
 

SMnasMAN

Contributor
Joined
Dec 2, 2018
Messages
177
does anyone know where we can get this script? (or can they share a copy pls?) the FTP and http links are dead (his entire server/domain do not resolve).
tks

EDIT: looks like the server is back-online, so i was able to pull the script.
wget ftp://ftp.sol.net/incoming/solnet-array-test-v2.sh

(would author be ok with me putting it on google drive or pastebin and posting a perm-link here for the future maybe? -- will not do this until given permission by him tho).
tks
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
No, I have no idea why it wouldn't resolve for you, but we've been on the 'net longer than Google, Pastebin, AOL, etc., and I don't really want to worry about needing to update things in locations that aren't obvious to me, or worse yet, that I don't even have access to update.

I'm not a super-big fan of all these free services because I've heard it all. "Why don't you just put your web pages on MySpace?" "Why bother running your own XMPP server when there's Google Wave or AIM or Yahoo Messenger?" "Why do you run your own email servers when there's Inbox.com?" "Why don't you host your DNS with Granite Canyon or DynDNS?" -- all free services that have shown up to much fanfare, and then either dropped the free service or gone away entirely. Since I do networking and server infrastructure and have been building bits of the Internet for most of my professional life, I don't see anything hard about building and maintaining services.
 

Mastakilla

Patron
Joined
Jul 18, 2019
Messages
202
Hi Jgreco!

Thanks for this script... 2 comments:

1) The grep for disk size doesn't work on all my disks

Code:
Performing initial serial array read (baseline speeds)
Sun Feb  9 18:39:21 CET 2020
Unable to determine disk da3 size from dmesg file (not fatal but odd!)
Unable to determine disk da5 size from dmesg file (not fatal but odd!)
Unable to determine disk da7 size from dmesg file (not fatal but odd!)
Sun Feb  9 18:57:24 CET 2020
Completed: initial serial array read (baseline speeds)

Array's average speed is 224.049 MB/sec per disk

Disk    Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
da0      9537536MB    240    107 ++FAST++
da1      9537536MB    226    101
da2      9537536MB    236    105
da3            0MB    238    106
da4      9537536MB    149     67 --SLOW--
da5            0MB    229    102
da6      9537536MB    234    104
da7            0MB    240    107 ++FAST++

Seems like it sometimes displays multiple disks on 1 line and the script doesn't handle it...
Code:
root@FreeNAS[~]# cat /var/run/dmesg.boot | grep "byte sectors"
ada0: 152627MB (312581808 512 byte sectors)
da0: 9537536MB (2441609216 4096 byte sectors)
da6: 9537536MB (2441609216 4096 byte sectors)
da1: 9537536MB (2441609216 4096 byte sectors)
da2: 9537536MB (2441609216 4096 byte sectors)
da4: 9537536MB (2441609216 4096 byte sectors)
da5: da7: da3: 9537536MB (2441609216 4096 byte sectors)
da5: 9537536MB (2441609216 4096 byte sectors)
da9: 40.000MB/s transfersda7: 9537536MB (2441609216 4096 byte sectors)
da9: 981MB (2009088 512 byte sectors)
da8: 29820MB (61071360 512 byte sectors)
root@FreeNAS[~]#

and here is a fix for that code: replace line 55 with
Code:
disksize=`(dmesg; cat /var/run/dmesg.boot) | egrep "byte sectors" | egrep "^${1}: [0-9]*MB| ${1}: [0-9]*MB" | head -1 | sed "s/^.*${1}/${1}/" | sed -n 's/^[a-z0-9]*: \([0-9]*\)MB .*$/\1/p'`


2) I have 1 disk that is a lot slower (da4), which worries me... Any suggestions?

Code:
Array's average speed is 224.049 MB/sec per disk

Disk    Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
da0      9537536MB    240    107 ++FAST++
da1      9537536MB    226    101
da2      9537536MB    236    105
da3            0MB    238    106
da4      9537536MB    149     67 --SLOW--
da5            0MB    229    102
da6      9537536MB    234    104
da7            0MB    240    107 ++FAST++

Performing initial parallel array read
Sun Feb  9 18:57:24 CET 2020
The disk da0 appears to be 9537536 MB.
Disk is reading at about 240 MB/sec
This suggests that this pass may take around 663 minutes

                   Serial Parall % of
Disk    Disk Size  MB/sec MB/sec Serial
------- ---------- ------ ------ ------
da0      9537536MB    240    242    101
da1      9537536MB    226    227    100
da2      9537536MB    236    238    101
da3            0MB    238    238    100
da4      9537536MB    149    186    125 ++FAST++
da5            0MB    229    229    100
da6      9537536MB    234    234    100
da7            0MB    240    240    100


edit:
Here is an overview of all the speed tests I've done so far...

1581335553804.png


da4 seems to be a lot slower in FreeNAS (according to the solnet-array-test script), a little slower in Aida64, but strangly enough, it was faster when doing a HDD Sentinel Surface Read Test??

Should I consider this as normal variance between the HDDs and ignore this difference? The difference according to solnet-array-test does seem huge to me...

edit2:
After letting the script complete a run, it seems like reading the whole disk evens it out a bit
Code:
Awaiting completion: initial parallel array read
  Mon Feb 10 18:01:12 CET 2020
Completed: initial parallel array read

Disk's average time is 53123 seconds per disk

Disk    Bytes Transferred Seconds %ofAvg
------- ----------------- ------- ------
da0        10000831348736   51721     97
da1        10000831348736   54930    103
da2        10000831348736   52964    100
da3        10000831348736   52785     99
da4        10000831348736   52440     99
da5        10000831348736   54381    102
da6        10000831348736   53401    101
da7        10000831348736   52361     99

Still I find it pretty strange that the initial read test consistently has da4 so much slower than all others... Anyone have an idea why?
 
Last edited:

Mastakilla

Patron
Joined
Jul 18, 2019
Messages
202
How long does the "initial parallel seek-stress array read" normally take? It's been running since 10 February in meantime :p so the estimate of 707 minutes is a little off ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Well if the straightline read time averaged 53k seconds, figure that number times several. Your drive is getting saturated with seeks.
 

Mastakilla

Patron
Joined
Jul 18, 2019
Messages
202
Thanks! It completed shortly after my question :) It took a factor 6-7 longer than the straightline read...

Here is a summary of my findings

1) The grep for disk size doesn't work on all my disks
Just in case you didn't read it yet with all my blabla and edits ;)
bugfix = replace the existing "disksize=" line with the below line
Code:
disksize=`(dmesg; cat /var/run/dmesg.boot) | egrep "byte sectors" | egrep "^${1}: [0-9]*MB| ${1}: [0-9]*MB" | head -1 | sed "s/^.*${1}/${1}/" | sed -n 's/^[a-z0-9]*: \([0-9]*\)MB .*$/\1/p'`


2) Disk speed variances that worried me
I've tried to gather all my "speedtests" so far in the below table
1581810949371.png

da0 seems the strongest. da4 and da5 seem the weakest.

Although da4 consistently scores exceptionally low in the IOstat test, it scores very average in the dd test (and all other tests except perhaps Aida Middle Seq). I'm still not sure what is causing these bad IOstat scores, but as it is scoring average in most other places, I'm hoping it might be ok. Could it perhaps be because I have an active FreeNAS filesystem on all disks (it was 100% unused during the test, but still, maybe something like a patrol-read-equivalent or other maintenance job could cause this?). Maybe I'll retry it once more after destroying the pool/zfs...

da5 scores the weakest in the dd tests and also in Aida random, but not alarmingly low...

edit: doing 1 more final run with the zpool destroyed and FreeNAS rebooted...
So far:
IOstat serial=146MB/sec (still ridiculously slow)
IOstat parallel=241MB/sec (from very slow to very fast)

So although it partly helped, there is still something weird going on with da4
 
Last edited:

Mastakilla

Patron
Joined
Jul 18, 2019
Messages
202
As I suspected my HBA from running too hot and causing these issues, I've replaced it and added a fan on it. This has slightly helped with the above inconsistencies, but not as much as I've hoped...

Results can be found in this thread

@ Jgreco: I think I've also found a bug (no solution this time) with the seek-read test timers (details in the above link as well)
 
Joined
Jan 7, 2015
Messages
1,150
I came to look for an updated version of the script and see if it had been superseded by a different tool yet. I discovered the links are dead.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
No, the links are not dead. However, your web browser might no longer support FTP URL's, so you may need to use an FTP client to retrieve the file, or use FreeBSD's "fetch" command.
 
Top