My NAS hardware and tunables are below, and it's a good build. In theory the pool should be blazingly fast, but it isn't.
A first bottleneck is that it crawls (comparatively speaking) on sequential writes, even locally from the CLI. I think I know why, and I'd like to reduce the bottleneck, but I'm not sure if I'm facing a config issue or hardware issue, and how to determine the optimal place to improve it.
Until recently I was getting a consistent 1GB/sec read and write across the LAN with Samba from SSD (about 970 - 990MB/sec). So I know the hardware can do 1GB/sec including LAN+CIFS overhead on large parallel sequential writes. As far as I know, I've only upgraded the mirrors from 2 way (previously) to 3 way (as they are now), and upgraded 11.0-U4 to 11.1. I've never got my Samba speed much above the values below, since then.
What I'd expect is that writes are as fast as a very fast NVMe SSD, due to a good ZIL, until txg's are full or it has to flush, but at worst, not much worse than single HDD speed for seq write after that (100-200 MB/sec?), and reads should be as fast as any read split across 12 HDDs can be (all data can be pulled from 3 mirrors and the file is striped across 4 vdevs since the pool isn't very full), certainly over 1GB/sec for large files.
I don't know if this is the "right" way to test, but anyhow. I created a 20GB malloc-backed ramdisk containing an 8GB random file (this was to ensure a full 8GB was written each time, and because /dev/random is CPU intensive). I copied this onto both my dedup'ed dataset and my non-deduped dataset in the same pool, 3 times each with different block sizes of 4k, 32k and 1m, making 6 tests total. I deleted and recreated the random file before each copy, to ensure any caching effects were similar and didn't favour the 2nd or subsequent access. The output was ... dramatic? ... to say the least:
So clearly dedup processing is the first major issue. I use dedup because I get about 1.3x saving from compression and about 4.5x saving from dedup (the system was specced with - hopefully! - enough RAM and a fast CPU specifically to make dedup practical after experimenting with data saving benefits). But I never expected it to add 3 minutes to a 10-15 second write.
Read speed is not very good for the hardware either. For example, reading back a 12GB Blu-ray rip (not previously accessed/cached) from the non-dedup dataset:
565MB/s without dedup is what I'd expect from a single 3-way mirror, but this pool has been made up of 4 striped 2-then-3-way mirrors since the start, so it should be a lot faster than that. No idea.
There might be other bottlenecks. For example:
Basically, this rig should be able to do a lot better, and 40-50MB/s isn't really okay. I can probably improve it by tuning or selective upgrades.
What can I do to either improve what I have, or pin down the upgrades needed for better performance?
My detailed hw and tunables are below.
Hardware spec:
I use mirrored HDD vdevs for my pool, with good hardware, fast CPU, 10G Chelsio, fast ZIL + L2ARC and plenty of RAM (detailed spec below). The system has been configured through the GUI only (original clean install 9.10.2) and doesn't run any services except Samba, iSCSI (idle/no connection) and SSH, no VMs, no extensions.
A first bottleneck is that it crawls (comparatively speaking) on sequential writes, even locally from the CLI. I think I know why, and I'd like to reduce the bottleneck, but I'm not sure if I'm facing a config issue or hardware issue, and how to determine the optimal place to improve it.
Until recently I was getting a consistent 1GB/sec read and write across the LAN with Samba from SSD (about 970 - 990MB/sec). So I know the hardware can do 1GB/sec including LAN+CIFS overhead on large parallel sequential writes. As far as I know, I've only upgraded the mirrors from 2 way (previously) to 3 way (as they are now), and upgraded 11.0-U4 to 11.1. I've never got my Samba speed much above the values below, since then.
What I'd expect is that writes are as fast as a very fast NVMe SSD, due to a good ZIL, until txg's are full or it has to flush, but at worst, not much worse than single HDD speed for seq write after that (100-200 MB/sec?), and reads should be as fast as any read split across 12 HDDs can be (all data can be pulled from 3 mirrors and the file is striped across 4 vdevs since the pool isn't very full), certainly over 1GB/sec for large files.
I don't know if this is the "right" way to test, but anyhow. I created a 20GB malloc-backed ramdisk containing an 8GB random file (this was to ensure a full 8GB was written each time, and because /dev/random is CPU intensive). I copied this onto both my dedup'ed dataset and my non-deduped dataset in the same pool, 3 times each with different block sizes of 4k, 32k and 1m, making 6 tests total. I deleted and recreated the random file before each copy, to ensure any caching effects were similar and didn't favour the 2nd or subsequent access. The output was ... dramatic? ... to say the least:
So clearly dedup processing is the first major issue. I use dedup because I get about 1.3x saving from compression and about 4.5x saving from dedup (the system was specced with - hopefully! - enough RAM and a fast CPU specifically to make dedup practical after experimenting with data saving benefits). But I never expected it to add 3 minutes to a 10-15 second write.
Read speed is not very good for the hardware either. For example, reading back a 12GB Blu-ray rip (not previously accessed/cached) from the non-dedup dataset:
565MB/s without dedup is what I'd expect from a single 3-way mirror, but this pool has been made up of 4 striped 2-then-3-way mirrors since the start, so it should be a lot faster than that. No idea.
There might be other bottlenecks. For example:
- I don't know if my PCIe-2 9211's can handle 8 HDDs at full speed, or 4 per port, so I have split the 8 disks across 2 HBAs, but it hasn't helped. Do I need a next-gen 3008 card like a 9300/9305/9311 or is this unlikely to be a bottleneck?
- Do I need more RAM (it can't hurt but it's expensive) or a better CPU?
- Should I change dedup algorithm on the dedup dataset? (I'm not sure how to check what I'm using but could well be SHA256 not fletcher+verify)
- ZIL has a top-end 400GB NVMe card to play with but doesn't seem to be making great advantage of it, and its txg size is probably tiny by comparison to the card size. Should I retune the ZIL sysctls to make better use of the ZIL card? Or will this be great up to 400MB and timeout for an hour flushing afterwards? :)
- Worth noting, I get occasional timeouts across the LAN on CIFS + iSCSI, but not sure how to diagnose what causes them. This can happen even when copying one large (sequential?) file from a single client. Maybe that's a separate question.
Basically, this rig should be able to do a lot better, and 40-50MB/s isn't really okay. I can probably improve it by tuning or selective upgrades.
What can I do to either improve what I have, or pin down the upgrades needed for better performance?
My detailed hw and tunables are below.
Hardware spec:
I use mirrored HDD vdevs for my pool, with good hardware, fast CPU, 10G Chelsio, fast ZIL + L2ARC and plenty of RAM (detailed spec below). The system has been configured through the GUI only (original clean install 9.10.2) and doesn't run any services except Samba, iSCSI (idle/no connection) and SSH, no VMs, no extensions.
- Hardware: Supermicro X10 series, Xeon 1620-v4 (quad core 3.5GHz to ensure good performance on single threaded tasks like Samba), 96GB ECC 2400, 10G T420-CR Chelsio NIC, two LSI 9211's with latest fw
- Disks:
Pool: vdevs are all 3 way mirrors configured as (3x 6TB) + (3 x 6TB) + (3x 8TB) + (3x 4TB) 7200 Enterprise drives, total formatted capacity ~ 22TB, used ~ 14TB. The HDDs are about 4 SAS and 8 SATA.
Boot: 2 mirrored Intel SSDs.
ZIL: Intel P3700 NVMe (PCie gen.3 slot)
L2ARC: Samsung 250GB NVMe (PCie gen.3 slot) - HDD/HBA connections: The HBAs are gen2 and in gen 2 + gen 3 slots. 8 of the HDDs (including all the SAS drives) are connected to the 9211s, and the rest are on the baseboard Intel chipset connectors. I also split the 9211-connected disks from 8 on one HBA to 4 each on 2 HBAs in case the older 2008 chips were a bottleneck.
