Horrible performance on new build...

[Robert Thomspon]

Hey guys,
Went from an old Dell T3500 machine which performed really well on my network, to a Dell T7500 (Dual Xeon, ~100GB of ECC RAM) and went from a RAIDZ to a RAIDZ2 setup from 5 5TB drives to 5x5TB and 1x6TB drive (they are all seagate barracudas... I couldn't find a 5TB drive when I was shopping...) and all run off of IO SATA controller cards (onboard is only SATA 2 speeds, the cards offer SATA 3 speeds)... on running read/write tests with dd, I seem to be getting about 1G/sec write speeds, and about 1.3G/s read speeds)... problem is, VMs seems to run REALLY slow and my network copy speeds are hovering around 25MB/s on a fully gigabit lan... (the t3500 would get me around 90MB/s which was fine by me)

As an indicator, Im on hour two of installing an Ubuntu Server VM with 8GB RAM and 2 CPUs... it's at 37% installed... It makes no sense!!!

The 6TB Drive is brand new... the 5x5TB drives were all run through bad blocks over the weekend, and smartctl long test... all tested within parameters...

Im not really sure what to share with you guys to clue you in on what info you might want... but feel free to ask, and ill be happy to provide whatever I can.

[DISREGARD] Additionally, and this might not even matter... but, I don't have any interfaces listed under my NETWORK > INTERFACES... I've never noted what its supposed to look like... but have been digging through different settings trying to figure out what is going on with the system... (Disregard what is italicized... I checked my other build... it also has nothing in the INTERFACES tab... sorry, I couldn't figure out how to do strikethrough... so went with italics) [DISREGARD]

Any help is appreciated.

 

[Chris Moore]

What is the output of zpool status?
It should look something like this:

Code:

  pool: Emily
 state: ONLINE
  scan: scrub repaired 0 in 0 days 02:11:45 with 0 errors on Fri Feb  9 10:11:46 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		Emily										   ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/1b9f316d-da26-11e7-b781-002590aecc79  ONLINE	   0	 0	 0
			gptid/bbf7a1c8-73ee-11e7-81aa-002590aecc79  ONLINE	   0	 0	 0
			gptid/55f074a3-cdb5-11e7-bdb7-002590aecc79  ONLINE	   0	 0	 0
			gptid/78e8e147-d97d-11e7-b781-002590aecc79  ONLINE	   0	 0	 0
			gptid/78b8bf90-d9b1-11e7-b781-002590aecc79  ONLINE	   0	 0	 0
			gptid/90d74abf-d18c-11e7-bdb7-002590aecc79  ONLINE	   0	 0	 0

errors: No known data errors

PS. What is the clock speed of the CPU in the T7500?

 

[Robert Thomspon]

 

[Robert Thomspon]

3.07 GHz (Screenshot is attached, as is the pool status screenshot) View attachment 23148

View attachment 23148

 

[Robert Thomspon]

Can I remove the 6TB drive and have the array rebuild itself on the 5 5TB drives? Im not 100% sure how the RAIDZ2 works... There's only about 1TB copied back on to the NAS at this point in time.

 

[Robert Thomspon]

[DISREGARD]

OK... so... Looks like one drive is jacking the whole system up (maybe).

stat puts my ada3 usage on average at about 90%, with a low of 75ish and spiking pretty regularly above 120%...

Is there a way to get more info on what the system has assigned as ada3? (I have a hunch that it's going to be the new 6TB drive)... but unsure how to get details about what

FreeNAS assigned each drive
[DISREGARD]

I got details with smartctl -a /dev/ada3

(It appears to be one of my older 5TB drives... will do further testing)

 

[wblock]

The easy way is diskinfo -v ada3.

 

[Robert Thomspon]

So, replaced "busy" drive with another brand new 6TB drive... resilvering now... will post progress as it happens. (Have about 4 hours left on resilvering).

on a random note: Does anyone know why it's called resilvering and not reslivering? seems to me the correct nomenclature would be sliver... as in slivers of data...

 

[Ericloewe]

Slivering? No, "slivers" doesn't make much sense in this context. I'd have preferred "rebuild" or something like that, but Bonwick seems to have been a force for weird naming inside Sun.

 

[Robert Thomspon]

Slivering? No, "slivers" doesn't make much sense in this context. I'd have preferred "rebuild" or something like that, but Bonwick seems to have been a force for weird naming inside Sun.

Why does "slivers" not make sense? the data is written in slivers... Small slivers on each drive... no single drive contains a large enough portion of data for the data to be recognizable (caveat being incredibly small files like docs or whatnot)... and spread out into slivers on the differing drives... Not arguing, just explaining. (Though, to be fair, I think that that isn't accurate either... just what we use as visual representation to explain how data is handled in RAID arrays...)



But yes... rebuild would make more sense then either one...

 

[Ericloewe]

Why does "slivers" not make sense?

Because that's not how mirrors work. So it wouldn't make sense for half of ZFS users.

 

[Robert Thomspon]

You and your silly facts and logic...

 

[wblock]

Why does "slivers" not make sense? the data is written in slivers... Small slivers on each drive...

Actually, those are "stripes". Yet people almost always mess that up, referring to "stripping" or "stripped" disks. Kind of the same way many people think "resilvering" actually is "reslivering".

The moral of this is probably that there is no way to win.

 

[Robert Thomspon]

So... new drive didn't help... If anything, its now running worse (after the resilvering)...

I am transferring across the LAN in the KB/s to this machine... My VM is downloading in SabNzbd at ~2M/sec ... the messed up part... my transmission jail (behind PIA) is downloading at 15M/sec ...I have never seen it over 6M/sec ...

Seriously... in need of suggestions...

 

[gpsguy]

I would try moving the HDD's back to the onboard SATA controller. With spinning rust, SATA 2 should be fine.

Out of curiosity what chipset do your add on cards use? Marvel? If so, this could cause problems.

 

[Robert Thomspon]

2 of the 6 drives are back on the mono SATA... One is indeed Marvell (The other is Asmedia).. BUT, seeing how, in Dell's infinite wisdom, they only built 3 SATA connectors onto this motherboard, I have no choice but to run SATA controllers... they're not fo the performance, just for handling the number of drives I have.

With Marvell being a problem, what SATA controller would you recommend?

And what are the issues with Marvell? (Ill scour and research as soon as I post this... figured you might have concise answers you're willing to volunteer :) )

 

[gpsguy]

We recommend a good HBA flashed to IT mode. Typically you can find one in eBay for $100 USD or less. Look in our resources section for recommended make/model.

I don't remember the Marvell problems off the top of my head. As you said - you can search the forums. I am replying from my cell phone so I can't help you with the search.

 

[Chris Moore]

You and your silly facts and logic...

This is a link to tell you about the source of the term. When the folks at Sun started designing ZFS, they applied the term to all rebuild operations for their own reasons that nobody else may ever fully understand.

Sorry, I somehow forgot to include the link... Here it is:
https://lonesysadmin.net/2012/03/23/why-is-it-called-resilvering/

 

[Chris Moore]

2 of the 6 drives are back on the mono SATA... One is indeed Marvell (The other is Asmedia).. BUT, seeing how, in Dell's infinite wisdom, they only built 3 SATA connectors onto this motherboard, I have no choice but to run SATA controllers... they're not fo the performance, just for handling the number of drives I have.

The Dell T7500 has four SAS ports built into the system board. Why don't you use those?

 

[Chris Moore]

If you are going to buy a controller, I use several of these and have been using them for years. Very reliable in my experience:
https://www.ebay.com/itm/Dell-H310-...ZFS-FreeNAS-unRAID-High-Air-Flow/162834671120
You will need two sets of these cables to go with it:
https://www.ebay.com/itm/Mini-SAS-3...-Breakout-Internal-Cable-3-28-FT/322480834919