Performance on one pool is great- bad on another

[rplucker]

Hey all,
Currently running core 12.0-U8.1 on:

Supermicro X10SRH-CF
E5 1630 V4
128GB ECC
X540 T1

Pool 1:
16x 10TB raidz2

Pool 2:
16x 18TB raidz3

All disks in pool 1 are connected to a supermicro case/backplane, which is connected to the motherboard via a single SFF 8643 cable

Similar with pool 2, but I'm using a separate enclosure for those disks.
Backplane -> 8643 cable -> 8643/8644 adapter -> 8644 external cable -> 8643/8644 adapter -> 8643 cable -> motherboard

read/write performance in pool 1 is fantastic. I consistently get speeds over 1GBps (10000mbps) over the network.
Pool 2 is a different story. Just copying files from pool 1 to pool 2 is much, much slower than anticipated at around 30-50MBps.
Same goes for copying files to/from over the network to pool 2.

What's my next steps here?

 

[Arwen]

Please list all your hardware. Disk controller cards mater, as do back planes and disk models. Then list the output of zpool list & zpool status in code tags.

For example, your Pool 1 could be using a 12Gbps SAS controller on 8 SAS lanes, with a 12Gbps SAS expander back plane.
But you Pool 2 could be using a 6Gbps SAS controller on 4 SAS lanes, with a 6Gbps SAS expander back plane.

Not to mention if Pool 1 was in a PCIe 3.x slot with 8 lanes, but the Pool 2 controller was in a PCIe 2.x slot with 4 lanes through the platform's chipset. So make sure to list which slots the disk controllers are in.


Last, if you are truly using 16 disk with RAID-Z2/3, that can be a problem. Especially if the pools are fullish. Performance is less than ideal with RAID-Zx stripes larger than say 8 to 10 disks. I know, performance seems good on Pool 1, but fragmentation and total amount used make a difference which is where Pool 2 could be.

 

[HoneyBadger]

All disks in pool 1 are connected to a supermicro case/backplane, which is connected to the motherboard via a single SFF 8643 cable

Similar with pool 2, but I'm using a separate enclosure for those disks.
Backplane -> 8643 cable -> 8643/8644 adapter -> 8644 external cable -> 8643/8644 adapter -> 8643 cable -> motherboard

To confirm - you're using the onboard SAS 3008 controller for both pools, with a chain of adapters and brackets for the latter?

What's the total length of the adapter cables to the second pool; have you checked that all connections are secure?

 

[rplucker]

To confirm - you're using the onboard SAS 3008 controller for both pools, with a chain of adapters and brackets for the latter?

What's the total length of the adapter cables to the second pool; have you checked that all connections are secure?

All of that is correct.

Length of the 8644 cable is 1M, + 1.5ft or so for each internal 8643 cable. I was wondering if cable length was the issue.

BUT-- I left for the day right after I posted this, then I came back the next morning and everything is working as expected. I changed *nothing*. No reboots, I made no physical changes, not even checking to make sure cables are secure.

Currently getting 1.1GBps+ read/write on the 2nd pool (both from pool to pool transfers & network transfers) with no issues.

Any idea what's going on & why it started working?

TIA

@Arwen
Each chassis is a supermicro 836BE1C-R1K03B. Unless I'm missing something, all the info on the backplane I see is listed as:
16-Port 3U SAS3 12Gbps single-expander backplane, support up to 16x 3.5-inch SAS3/SATA3 HDD/SSD

Using the onboard sas 3008 controllers on the motherboard.

"vault" is pool 1, "PeterParker" is pool 2.


root@cortana:/ # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT PeterParker 262T 7.37T 255T - - 0% 2% 1.00x ONLINE /mnt boot-pool 3.61T 1.43G 3.61T - - 0% 0% 1.00x ONLINE - flash 14.5T 2.69T 11.8T - - 0% 18% 1.00x ONLINE /mnt vault 145T 135T 10.1T - - 22% 93% 1.00x ONLINE /mnt root@cortana:/ #

root@cortana:/ # zpool status pool: PeterParker state: ONLINE config: NAME STATE READ WRITE CKSUM PeterParker ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 gptid/53e4654c-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/54ac07ff-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5649ed00-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/594f1034-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5a37e24e-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5b410fd2-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5c7bdabd-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5c07e27a-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5b50e415-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/5d37aa5f-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/62d891dc-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/633d2ac4-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/63b07c99-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/63a0dd74-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/6467bc11-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 gptid/65236669-acc4-11ed-8755-a0369fb21da2 ONLINE 0 0 0 errors: No known data errors pool: boot-pool state: ONLINE scan: scrub repaired 0B in 00:00:19 with 0 errors on Mon Feb 20 03:45:20 2023 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 errors: No known data errors pool: flash state: ONLINE scan: scrub repaired 0B in 00:33:27 with 0 errors on Sun Feb 19 00:33:27 2023 config: NAME STATE READ WRITE CKSUM flash ONLINE 0 0 0 gptid/9c830b59-cc23-11ec-ac6a-a0369fb21da2 ONLINE 0 0 0 gptid/9c8b54c9-cc23-11ec-ac6a-a0369fb21da2 ONLINE 0 0 0 errors: No known data errors pool: vault state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: resilvered 106G in 00:32:55 with 0 errors on Tue Feb 14 18:30:37 2023 config: NAME STATE READ WRITE CKSUM vault ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gptid/95318a95-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/95bda82e-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/963d207a-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/96be06ac-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/9734c0d7-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/aa0d6029-4c38-11e9-a7a9-a0369fb21da2 ONLINE 0 0 0 gptid/982e12b9-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/99e7c5ca-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 0 gptid/56b5da7c-9686-11ed-843b-a0369fb21da2 ONLINE 0 0 0 gptid/9aed80a9-797e-11e8-a1d8-a0369fb21da2 ONLINE 0 0 1 gptid/243630d6-f746-11ec-81cc-a0369fb21da2 ONLINE 0 0 0 gptid/7f2a2915-dee2-11e9-9c18-a0369fb21da2 ONLINE 0 0 0 gptid/f5efc59b-17be-11ea-b6a1-a0369fb21da2 ONLINE 0 0 0 gptid/d41ffb19-2b3b-11ea-9185-a0369fb21da2 ONLINE 0 0 0 gptid/e2bf4e7d-c191-11ec-9193-a0369fb21da2 ONLINE 0 0 0 gptid/31df5d27-7d26-11eb-bace-a0369fb21da2 ONLINE 0 0 0 errors: No known data errors

 

[rplucker]

Update #2:

After seeing that read/write performance is what I'm expecting, I started moving about 14TB of files over. Everything ran at the speeds that I was anticipating for a mix of large & small files (350-600MB/s), but after 20 minutes or so, it's slowed down to a crawl again and is currently sitting at 27-40MB/s.

 

[Jessep]

Thermal throttling?

 

[HoneyBadger]

How's the airflow situation over your NIC and HBA (I'm aware it's embedded as a motherboard item) - slowdown after that kind of period could imply an overheating component. Normally the ZFS write throttle will make itself known earlier than that.

 

[rplucker]

Airflow seems to be good. All 5 case fans are spinning and there's no obstructions inside or out.
Ambient air is 69-71 degrees. CPU sitting at 38 degrees under 40% load.
I dont think I can monitor the hba directly, but I'm checking to see if we have an infrared thermometer.

TLDR I don't see any issues there

Interesting development though, I tried the following:
-Delete pool 2 & create 2x 8 disk vdevs, each in raidz2, for a single pool across the 2 vdevs. Performance stayed the same. Good momentarily, then slowed down to the same crawling speeds.
-Delete pool 2 & stripe all disks (I know its bad, just testing). Performance seems to be spot on. So far I've copied 3TB over the course of an hour at 300-600MB/s and it seems to be going steady.

 

[Arwen]

Your SAS controller looks fine as does the SAS expander. Not sure about the length of cable. SAS supports >1m but with the adapters and internal lengths, that might introduce problems. Except I would have thought the problems would be constant.

---

I would have said your pool 2 "PeterParker" RAID-Z3 vDev was too wide at 16 disks. But, you ran a test with with 2 x 8 disk RAID-Z2 vDevs, and they had the same result.

---

I vaguely recall seeing thread here in forums where a user had 1 disk causing slow downs. Whence replaced, all was good. This reminds me, perhaps a SMR drive got slipped into the bunch. Re-reading this thread, I see we forgot to ask vendor & model.

What are your disk manufacturer(s) & model(s)?

Further, you could use zpool iostat -v PeterParker to see if you can detect any specific disk slow downs. Remember, a write is not complete until all disks in a RAID-Zx stripe are written and acknowledged. So yes, a single disk can slow an entire pool down.

---

One other idea I had, you could possibly use a SAS 12Gbps PCIe card with 8e to the external disk tray. That would remove the 2nd pool's traffic from the internal SAS controller. Plus hit 2 other potential issues, shortening the cable(s), and potentially adding 4 more SAS lanes for your external disk chassis. (And maybe adding another 4 more SAS lanes to the internal SAS expander.)

SAS does not load share SAS lanes properly. In fact, it could be considered brain dead. (If I understand it, which I may not.) So if too much traffic needed the same SAS lane, it would be a bottle neck.

---

Anyway, a few things to think about.

 

[rplucker]

Your SAS controller looks fine as does the SAS expander. Not sure about the length of cable. SAS supports >1m but with the adapters and internal lengths, that might introduce problems. Except I would have thought the problems would be constant.

---

I would have said your pool 2 "PeterParker" RAID-Z3 vDev was too wide at 16 disks. But, you ran a test with with 2 x 8 disk RAID-Z2 vDevs, and they had the same result.

---

I vaguely recall seeing thread here in forums where a user had 1 disk causing slow downs. Whence replaced, all was good. This reminds me, perhaps a SMR drive got slipped into the bunch. Re-reading this thread, I see we forgot to ask vendor & model.

What are your disk manufacturer(s) & model(s)?

Further, you could use zpool iostat -v PeterParker to see if you can detect any specific disk slow downs. Remember, a write is not complete until all disks in a RAID-Zx stripe are written and acknowledged. So yes, a single disk can slow an entire pool down.

---

One other idea I had, you could possibly use a SAS 12Gbps PCIe card with 8e to the external disk tray. That would remove the 2nd pool's traffic from the internal SAS controller. Plus hit 2 other potential issues, shortening the cable(s), and potentially adding 4 more SAS lanes for your external disk chassis. (And maybe adding another 4 more SAS lanes to the internal SAS expander.)

SAS does not load share SAS lanes properly. In fact, it could be considered brain dead. (If I understand it, which I may not.) So if too much traffic needed the same SAS lane, it would be a bottle neck.

---

Anyway, a few things to think about.

I'm running all identical seagate ironwolf pro 18tb in pool 2 (peter parker) and a mix of 10tb hgst deskstar nas and WD red plus in pool 1 (vault).

I'll run some more tests tomorrow & update with the results of zpool iostat -v PeterParker

Very much appreciated!

 

[Arwen]

With those models, I don't see any problems that I know of, (like SMR...).

 

[rplucker]

Deleted the array & configured as raidz2.
Copied a bunch of files over and it did the same thing- good speeds to start with, then slowed down a crawl. After it slowed, here's the results of zpool iostat -v PeterParker. All disks seem to be consistent. Also ran this at the start when speeds were good & had similar results, but all disks were at a consistent write of ~28M.

root@cortana:~ # zpool iostat -v PeterParker capacity operations bandwidth pool alloc free read write read write ---------------------------------------------- ----- ----- ----- ----- ----- ----- PeterParker 1.88T 260T 1 2.00K 8.39K 91.4M raidz2 1.88T 260T 1 2.00K 8.39K 91.4M gptid/bf0b4489-b23c-11ed-8755-a0369fb21da2 - - 0 104 535 5.72M gptid/bf1a9603-b23c-11ed-8755-a0369fb21da2 - - 0 152 529 5.71M gptid/bf2a96c9-b23c-11ed-8755-a0369fb21da2 - - 0 151 531 5.71M gptid/bea5b169-b23c-11ed-8755-a0369fb21da2 - - 0 113 552 5.72M gptid/c3ce6199-b23c-11ed-8755-a0369fb21da2 - - 0 152 534 5.71M gptid/c48dfa19-b23c-11ed-8755-a0369fb21da2 - - 0 125 542 5.71M gptid/c4f5eff4-b23c-11ed-8755-a0369fb21da2 - - 0 105 530 5.72M gptid/c56c19c7-b23c-11ed-8755-a0369fb21da2 - - 0 96 539 5.72M gptid/cb4dd659-b23c-11ed-8755-a0369fb21da2 - - 0 102 530 5.72M gptid/ce24babb-b23c-11ed-8755-a0369fb21da2 - - 0 108 527 5.72M gptid/cccddd31-b23c-11ed-8755-a0369fb21da2 - - 0 153 569 5.71M gptid/ccbdab88-b23c-11ed-8755-a0369fb21da2 - - 0 151 538 5.71M gptid/ce14f22e-b23c-11ed-8755-a0369fb21da2 - - 0 110 521 5.72M gptid/ccece935-b23c-11ed-8755-a0369fb21da2 - - 0 119 539 5.72M gptid/cee03052-b23c-11ed-8755-a0369fb21da2 - - 0 151 528 5.71M gptid/cf4612e3-b23c-11ed-8755-a0369fb21da2 - - 0 152 537 5.71M ---------------------------------------------- ----- ----- ----- ----- ----- -----

 

[rplucker]

@Arwen Since you mentioned using a separate hba,

I already have a 9220-8i laying around that I'm not using.

If I purchase a 8643 to mini sas cable & use this card (it mode) for the external connection to the 2nd array, while keeping the onboard connection for the first array, does anyone see any issues with that?

I see it's limited to 6gbps (I'm assuming that's 6gbps per card, but a single mini sas port on it can support a 6gbps connection?), which isn't a huge deal. A couple hundred MB/s transfer speed would be acceptable as this is for long term storage.

Or do I need something like a 9300 8i?

 

[Arwen]

Looking at the pool 2 IO stats, their does not seem to be any slow disk. They all look to be similar enough.


You could try the 9220-8i even though it is 6Gbps, (per SAS lane). Ideally you would wire up all 8 external SAS lanes to your external enclosure. (If the external enclosure supports it...)

Other than that, I am not sure I can help further.

 

[rplucker]

Looking at the pool 2 IO stats, their does not seem to be any slow disk. They all look to be similar enough.


You could try the 9220-8i even though it is 6Gbps, (per SAS lane). Ideally you would wire up all 8 external SAS lanes to your external enclosure. (If the external enclosure supports it...)

Other than that, I am not sure I can help further.

Got it, thank you for your help

 

[Arwen]

Your welcome.

Continue posting if you need further help. I might not respond if I have nothing to add.

 

[rplucker]

Well, I swapped in the 9220-8i and it did the same thing.
I took a look at system reporting and after a few minutes I discovered that the disks in the 2nd enclosure were hitting temps of 60c+.......

Turns out in my rush to get this system up and running, I hadn't connected the chassis fans.

Fan controller on the way. You live and you learn...