Lol. Ok. Ive been down this road. I’m trying to give you a fix. Keep suffering and wasting time. Doesn’t bother me. you can’t easily try it though.
OK -- so... I will be doing some testing this weekend, and hope to prove that we are just pushing data into the array too damn fast for the 3-disk write bandwidth we have.
So -- if I wanted to build a disk array for bulk storage, how many drives would I need to absorb that much IO?
Thinking about a disk shelf with a bunch of 4tb drives in it, all mirrored, etc. Is this doable or am I stuck with either slowing down the IO or converting to all flash (not an option due to cost)?
OK folks, here's the update, and it's not good:
I have 6 drives in a 3-way striped mirror, so I have the write performance of 3 drives.
- 12Gb SAS means max transfer of 1.5GB/s
- 1.5GB/s * 3 drives = 4.5 GB/s
- 4.5GB in a 1s TXG * 2 TXGs = 10GB
Tested again. Marginal improvement, still fails.
So I thought, "OK, 12GB/s is MAX transfer, not likely these drives will achieve that in actual write speed." so I dropped the SLOG to 5GB.
So -- if I wanted to build a disk array for bulk storage, how many drives would I need to absorb that much IO?
Thinking about a disk shelf with a bunch of 4tb drives in it, all mirrored, etc. Is this doable or am I stuck with either slowing down the IO or converting to all flash (not an option due to cost)?
you need guaranteed performance for iSCSI, or else iSCSI freaks out and resets connections and everything goes to hell.
sync=always
from the ZFS side.I'm stealing this metaphor. This is mine now.But at the end of the day, trying to aim a 40GbE at six hard drives is like trying to water a potted plant with a firehose.
I'm stealing this metaphor. This is mine now.
I would be looking to add 12 more vDevs (24 disks in striped mirrors) to that tier 2 pool. Supermicro has several JBOD enclosures to accomplish adding additional spinners to your server. I don't know what your budget looks like but I would get the 45 disk capacity JBOD so you can grow as needed.I was thinking about adding 12 more disks, to effectively triple the IOPS we have now.
When we were beating the hell out of the array, we saw both the read and write performance to the disks drop dramatically. We were getting read rates of 20-40M, and write rates of 100-200.
When I am doing write-only, I see numbers closer to 500-600.
I believe we are suffering from a lack of IOPS, which is what was suggested earlier in this thread. I know that I can't ever really expect to saturate 40Gb onto disks, but I was thinking that we might be able to mitigate the problem by adding a bunch more disks in an external disk shelf. I was thinking about adding 12 more disks, to effectively triple the IOPS we have now.
This is part of the reason that the "use NFS" advice holds some water; it's sending the writes as synchronous from vSphere, whereas the VMware iSCSI initiator sends the writes asynchronously which is why you need to forcesync=always
from the ZFS side.
I'm stealing this metaphor. This is mine now.
But at the end of the day, trying to aim a 40GbE at six hard drives is like trying to water a potted plant with a firehose.
For 40 gigabits? I doubt it. 10? should be. 20? maybe.We are using 8TB drives now, and that's what I would use in the disk shelf I am thinking about. The added Vdevs would increase our capacity by huge amounts and allow us to not fill them.
The question is, would 12 more disks in striped mirrror added to the existing 6 be enough?
For 40 gigabits? I doubt it. 10? should be. 20? maybe.
i sit corrected..:)10Gbps? With nine HDD vdevs? Very, very, very sketchy. It totally depends on what you're actually expecting out of the thing. I would think you'd probably be fine at 1Gbps, except that I know I could break it if given a free hand to place a torturous workload on it. If you aren't popping surprise stressy write workloads on it that break the ZFS write throttle, 10Gbps could be fine, but I guarantee it to be breakable if you get even moderately aggressive.
The hypothetical worst case situation here is that a single vdev might only be able to handle 150 IOPS of 512 byte I/O's. That is 77KBytes/sec or 614Kbps. To sustain 40Gbps, you would need 65146 vdevs, or ~128,000 hard drives. As an expectation-setting exercise, I hope you get the point being made.
In practice, of course, it shouldn't be anywhere near that bad. So let's calculate the most optimistic case, just for comparison. A single modern HDD should be able to sustain 150MByte/sec, or 1.2Gbit/sec, sequential write speeds. So you would need 34 vdevs, or 68 mirrored hard drives, or a hundred three-way mirrored hard drives, in order to be able to have a chance of being able to fully saturate a 40Gbps. Anything less is simply not going to be able to keep up.
You're better off thinking about a disk shelf with a bunch of larger hard drives in it, and then not using most of the space. One of the biggest write accelerators for ZFS is when it can easily find contiguous free space.
as jgreco said above in the first quoted section..no you simply do not have enough hardware to keep up. you are going to need quite a few more vdevs to have a prayer.We are using 8TB drives now, and that's what I would use in the disk shelf I am thinking about. The added Vdevs would increase our capacity by huge amounts and allow us to not fill them.
The question is, would 12 more disks in striped mirrror added to the existing 6 be enough?
I know this is a Coke vs Pepsi question, but would switching to NFS help since it's not "block" storage?
I know this is a Coke vs Pepsi question, but would switching to NFS help since it's not "block" storage?