T7500 - My god-awful machine...

[Robert Thomspon]

OK... so, Im still tackling this god awful machine that runs like molasses in the winter...
Original Build was:
2x Xeon X5667 (3.07GHz/ea)
84GB RAM (9x8 4x4) all ECC Registered... (I ran out of 8GB sticks...)
1x6TB Seagate Baracuda and 5x5TB Seagate Baracuddas (the 5x5 were running on my T3500 just fine)
1x 2 port Intel SATA controller
1x4 port Marvell SATA controller
... ran god awful.

Changed out 1 of the 5TB drives for another 6TB drive as it was showing a lot of what appeared to be queued writes under gstat.
Pulled the mismatched RAM to match Dell's recommended configuration at 8GB in Banks 1,2,3 and in the Riser for 2nd CPU, banks 1,2,3...
Switched out the SATA controller cards for an Intel RAID card flashed to IT mode... (Model RS2WC040) and the remaining 2 drives connected to on board SATA... (Card runs at SATA 3, on board runs at SATA 2... I believe, this effectively makes everything run at SATA 2 speed... ) anyway... there's enough drives that the reduced speed should make little to no difference on performance...

gstat still kicks back with a lot of %busy on the drives (and I am copying data over to them)... but no single drive sticks out like the 1 5TB drive did before being replaced (it would normally sit at 80-140% busy ... now all drives show roughly the same wait time... sometimes spiking individually above 100%... but not constantly)

BUT... it still runs like total crap (though, a bit better crap than OG build... I used to top out around 10MB/s across the network... Im not around 35MB/s on data transfers across an entirely gigabit LAN... )

But worst of all is the performance of the VM machine (I have an ubuntu VM running all my gadgets like SABNZB, Sonarr... etc expect Transmission which runs in a jail behind a VPN. Best example I can offer for comparison is the installation of boot-repair tools from command line (it's a server build, so no GUI... but I do know how long it SHOULD take (my T3500 installs this in about 5 minutes)... this install took about an hour... Even simple things like losing in via SSH seem to lag massively (VM is built with 2 cores, 8GB of RAM... probably a little overkill for what is needed... but wanted to rule out the idea that maybe the VM is running out of resources...)

Im tempted to pull the second CPU (riser) to see if maybe there's an issue with the CPU? im honestly at a loss on what else could be the issue with this things speed/responsiveness...

dd if=testfile of=/dev/zero bs=1024 bytes=50000
kicks back read times of between 58-66 MB/sec

dd if =/dev/zero of=testfile bs=1024 bytes=50000
kicks back write times of between 170-190 MB/sec

Can anyone point me in a direction to test the machine to kick back legit numbers? or where to look for why this machine feels SO bogged down?

Sorry for the novel.

 

[toadman]

What version of FreeNAS?

I think the dd tests you are doing will give "legit" numbers, at least for basic pool speed. However, (1) on the read test you want of=/dev/null. And (2), on the write test you want to make sure the dataset you are writing to has compression set to OFF (otherwise it won't actually write any of that long string of zeros you are sending it).

That said, have you done any isolated network tests to rule that out as a contributing factor? (Like running iperf and making sure the network runs at the expected speed without having to interact with the disk system.)

What topology is your pool? Mirrors, RAIDZ, RAIDZ2? (Mirrors preferred for hosting VMs, if you care about IOPS and VM performance.)

How are you running the VM? (It's not clear... on your FreeNAS system? If yes, under virtualbox, or something else? or on some other system where you are using FreeNAS for storage only?)

 

[Chris Moore]

anyway... there's enough drives that the reduced speed should make little to no difference on performance...

In a single vdev, you get the speed of the slowest drive. So, if you are running at SATA 2 speed on even 1 drive, the whole vdev (and you only have one vdev in the pool) is limited to the speed of that drive. Best performance is to get all the drives on a single controller.

There could be some other hardware in the system that is causing the slowdown. Have you run the integrated Dell Diagnostics? Please give a rundown of all the hardware in the system, even down to the video card.

 

[Robert Thomspon]

What version of freenas?

I think the dd tests you are doing will give "legit" numbers, at least for basic pool speed. However, (1) on the read test you want of=/dev/null. And (2), on the write test you want to make sure the dataset you are writing to has compression set to OFF (otherwise it won't actually write any of that long string of zeros you are sending it).

That said, have you done any isolated network tests to rule that out as a contributing factor? (Like running iperf and making sure the network runs at the expected speed without having to interact with the disk system.)

What topology is your pool? Mirrors, RAIDZ, RAIDZ2? (Mirrors preferred for hosting VMs, if you care about IOPS and VM performance.)

How are you running the VM? (It's not clear... on your freenas system? If yes, under virtualbox, or something else? or on some other system where you are using freenas for storage only?)

Great questions... and recommendation the dd...

Running RAIDZ2.
FreeNAS version 11.1.U2
VM is run under FreeNAS as it's built in VM (VM> Virtual Machine style, as in NOT Docker).
I have never used iperf, but will research its use and report back my findings...

 

[Robert Thomspon]

In a single vdev, you get the speed of the slowest drive. So, if you are running at SATA 2 speed on even 1 drive, the whole vdev (and you only have one vdev in the pool) is limited to the speed of that drive. Best performance is to get all the drives on a single controller.

There could be some other hardware in the system that is causing the slowdown. Have you run the integrated Dell Diagnostics? Please give a rundown of all the hardware in the system, even down to the video card.

I will post a complete rundown in a little bit... I've got stuff copying at the moment and will run the onboard diagnostics. I will post inline with this thread as well as go back and edit the original post to reflect a breakdown of all components...

 

[toadman]

Actually, looking again at your dd tests, I'm not familiar with the "bytes=" switch. I would use a LOT bigger transfer to test your pool, esp given the amount of memory you have. Otherwise the ARC will just cache everything. Alternatively you can set the arc_max to a really low number (say 1GB) and transfer less. But I would transfer 2x your arc_max size. And since I don't know what "bytes=" does (though it seems obvious) I would do (for a 100GB test)...

Write: dd if=/dev/zero of=./testfile bs=1M count=100K
Read: dd if=./testfile of=/dev/null bs=1M

 

[wblock]

dd if=testfile of=/dev/zero bs=1024 bytes=50000

Tiny 1K blocks will give a lot of overhead that might really slow things down. A size of 64K or more should cut down overhead enough to not hide the actual abilities of the system.

 

[Robert Thomspon]

Actually, looking again at your dd tests, I'm not familiar with the "bytes=" switch. I would use a LOT bigger transfer to test your pool, esp given the amount of memory you have. Otherwise the ARC will just cache everything. Alternatively you can set the arc_max to a really low number (say 1GB) and transfer less. But I would transfer 2x your arc_max size. And since I don't know what "bytes=" does (though it seems obvious) I would do (for a 100GB test)...

Write: dd if=/dev/zero of=./testfile bs=1M count=100K
Read: dd if=./testfile of=/dev/null bs=1M

sorry... don't know why I didn't catch that... the "bytes" should be "count" ... I mistyped when carrying over the line :)

 

[Robert Thomspon]

Actually, looking again at your dd tests, I'm not familiar with the "bytes=" switch. I would use a LOT bigger transfer to test your pool, esp given the amount of memory you have. Otherwise the ARC will just cache everything. Alternatively you can set the arc_max to a really low number (say 1GB) and transfer less. But I would transfer 2x your arc_max size. And since I don't know what "bytes=" does (though it seems obvious) I would do (for a 100GB test)...

Write: dd if=/dev/zero of=./testfile bs=1M count=100K
Read: dd if=./testfile of=/dev/null bs=1M

using /dev/null kicks back immediately with 0s across the board (I don't think my build likes it :) )

Code:

% sudo dd if=/dev/null of=./testfile bs=1M count=100000

Password:

0+0 records in

0+0 records out

0 bytes transferred in 0.000012 secs (0 bytes/sec)

% sudo dd if=/dev/zero of=./testfile bs=1M count=100000

100000+0 records in

100000+0 records out

104857600000 bytes transferred in 1983.962423 secs (52852614 bytes/sec)

% sudo dd if=./testfile of=/dev/zero bs=1M count=100000

Password:

100000+0 records in

100000+0 records out

104857600000 bytes transferred in 17.552906 secs (5973802986 bytes/sec)

% 

And iperf is showing:
[164] 0.0-100.0 sec 7337 MBytes 73.4 MBytes/sec.

so... not great on gigabit lan (wired) ... but not garbage either...

 

[Chris Moore]

73.4 MBytes/sec.

That could be part of the problem, but I don't think it is the whole thing. I am not sure how iperf figures out the answer here, but I know that when I move files on my 1Gb network, the speed is usually 113MB/s. Your speed test there is not horrible, but it should be faster.

Have you tried this:
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

 

[Robert Thomspon]

Ok... build is:
Dell T7500...
CPUs are Xeon X5667 (dual w/ riser card)
Memory is 6x8GB SK Hynix w/ cisco branding 2Rx4 PC3L 12800R in MB and riser slots 1,2,3 as per dell instructions.
Intel RS2WC040 RAID controller card (LSI L3-25091-04C) flashed to IT mode with SAS2008
Video card is ATI Firepro V4800 (but just swapped out with ATI HD4890 (creates less heat... machine just happened to come with v4800)

I was going to wait to post this until the Dell Self-Test is done.. but wow.. it takes a LONG time on the memory. Im on hour 4 so far...

 

[Robert Thomspon]

Have you tried this:
https://forums.freenas.org/index.php?resources/solnet-array-test.1/

I have not... but will run it overnight tonight if the self test finishes before I go to bed.

Also... moved the 2 drives that were attached to MB SATA to the add-on SATA card (Asmedia Controller)... Bumped my write speeds up to 80-120MB/sec (still not great... but showing improvement)... iperf tests remain the same (During Dell self test, I went through some cabling and shortened some lines. Specifically FreeNAS to switch and router to switch (both of which were Cat5e, and moved to as short as possible Cat6) all tests run over wired connection...

@Chris Moore running your suggested tool now... I assume I will have to report results in the morning... but it is the only thing running on the machine, VMs shut down, no network calls to the machine...

 

[Robert Thomspon]

Switched SAS controller to an H310 flashed to IT mode... no change in performance.

 

[wblock]

Why are you reading from a write-only device (/dev/null) and writing to a read-only device (/dev/zero)?

Part of the problem is there. Read from /dev/zero, write to /dev/null, but neither will give real-world performance.

 

[Stux]

In a single vdev, you get the speed of the slowest drive

You get the IOPs of the slowest drive. Speed is generally a multiple and is a nebulous term too.

 

[Stux]

Re Iperf. Would suggest running a direct line from the test machine to the test host. No router/switch

Be surprised if a dual X5667 can’t saturate gigabit.

 

[Robert Thomspon]

Why are you reading from a write-only device (/dev/null) and writing to a read-only device (/dev/zero)?

Part of the problem is there. Read from /dev/zero, write to /dev/null, but neither will give real-world performance.

I'm not... that was a suggestion from another member...
Please read the rest of the post and you'll see that his suggestion kicks back immediately without results.

As for read/write speeds, these were just tests to get a feel for what the drives are doing, and pulling out the read/write speeds of the drives themselves... Ended up running much larger (larger than ARC) read/write tests to get more legitimate results.... in theory.

Do you have a better suggestion? I would be happy to try...

 

[Stux]

What's that last line supposed to say? "Be surprised if dual X5667 'can' or 'can't' saturate gigabit?

Can’t. Fixed.

 

[Stux]

I'm not... that was a suggestion from another member...
Please read the rest of the post and you'll see that his suggestion kicks back immediately without results.

As for read/write speeds, these were just tests to get a feel for what the drives are doing, and pulling out the read/write speeds of the drives themselves... Ended up running much larger (larger than ARC) read/write tests to get more legitimate results.... in theory.

Do you have a better suggestion? I would be happy to try...

You misimplemented the suggestion

In dd, if= specifies the Input File and of= specifies the Output File

Input file should be readable, Ie your test file, or /dev/zero
Output file should be writable, ie your test file or /dev/null

Originally you were writing to /dev/zero, which shouldn’t have worked. When told about your mistake you started reading from /dev/null, which also shouldn’t work (and doesn’t)

 

[Robert Thomspon]

You misimplemented the suggestion

In dd, if= specifies the Input File and of= specifies the Output File

Input file should be readable, Ie your test file, or /dev/zero
Output file should be writable, ie your test file or /dev/null

Originally you were writing to /dev/zero, which shouldn’t have worked. When told about your mistake you started reading from /dev/null, which also shouldn’t work (and doesn’t)

except that /dev/zero works... so... And I wasn't "told about my mistake" I was advised to try /null... which I did, and immediately reported back that it didn't work.

I took the /dev/zero from someone who wrote a blurb on testing the speed of the drives....
here:

http://www.benwagner.net/uncategorized/test-readwrite-speed-in-freenas/

And it worked for what I was trying to see... (it gave better/more applicable results when I made the file to read/write 100GB as was suggested by another user on here...)

That being said, wether it's supposed to work or not... it did indeed work for me, and gave me results I could see that seem to match with what I should be seeing...

That being said, I get that you don't like the lines used... but can we move on? or failing the moving on, can you suggest an actual course of action?