mediahound
Dabbler
- Joined
- Mar 11, 2013
- Messages
- 15
I'm trying to design up a system or two around certain "expected performance goals" and am trying to understand how much freeNAS and ZFS would actually be using in the background so that I don't inadvertently have a bottleneck.
Example, I currently have two separate systems i'm playing with the idea of:
- Slower data capture system. Only needs to maintain about 15megabytes/second transfers but has to maintain that under ALL conditions 24/7 - meaning while rebuilding drives and doing all other forms of housekeeping. Dropping below this means lost data. I don't want to overengineer, just make sure I can hit this goal. If the overhead isn't much I might even be using USB 2.0 drives or hubs or ports due to cost savings and not needing to do anything but this minimum rate.
- Performance/processing system. This I want to be able to maintain 100megabytes/second transfer rate but only when in use. If a drive drops and something has to rebuild it's acceptable performance degrades, it just means slower nonrealtime video processing and similar.
IOPS doesn't matter much in either, it's not a webserver or database or something needing throwing alot of spools at. Mostly multichannel media writing and reading.
To both of these, i'm trying to figure out how much additional performance overhead to design in... for instance considering that a typical hard disk anymore exceeds 60MB/sec at the slowest end of the drive, and 120MB/sec at the beginning, I would think that just two drive stripe sets should manage my 100MB/sec performance system, and be ridiculously unneeded in the data capture system, but I could be mistaken/wondering what happens if like two drives are dying and having data migrated to replacements at the same time.
I'm also curious if there are any other issues that would make drive performance not scale linearily. Ie - if I wanted 200MB/sec instead, I would assume four drive stripes should achieve that if I assume minimum transfers are still 60MB/sec of likely drives to use. Much over 200MB/sec I can't see the need for, because my main goal is to saturate a gigabit channel primary connection, while having a 2nd channel running full speed backup of other parts of the array or being used for higher speed housekeeping.
Could someone help me understand what ZFS actually needs to be doing in the background, how demanding it is, how often its doing it and similar? Between recovering from bad drives, scrubbing the disks looking for corrupt data, silvering new disks, writing out space allocation requests (if it gets 1 bit of a file demanding 50gigabytes contiguous does ZFS just pre-allocate it out as one block at max speed?), writing to various logs or journals, and all other housekeeping...
Example, I currently have two separate systems i'm playing with the idea of:
- Slower data capture system. Only needs to maintain about 15megabytes/second transfers but has to maintain that under ALL conditions 24/7 - meaning while rebuilding drives and doing all other forms of housekeeping. Dropping below this means lost data. I don't want to overengineer, just make sure I can hit this goal. If the overhead isn't much I might even be using USB 2.0 drives or hubs or ports due to cost savings and not needing to do anything but this minimum rate.
- Performance/processing system. This I want to be able to maintain 100megabytes/second transfer rate but only when in use. If a drive drops and something has to rebuild it's acceptable performance degrades, it just means slower nonrealtime video processing and similar.
IOPS doesn't matter much in either, it's not a webserver or database or something needing throwing alot of spools at. Mostly multichannel media writing and reading.
To both of these, i'm trying to figure out how much additional performance overhead to design in... for instance considering that a typical hard disk anymore exceeds 60MB/sec at the slowest end of the drive, and 120MB/sec at the beginning, I would think that just two drive stripe sets should manage my 100MB/sec performance system, and be ridiculously unneeded in the data capture system, but I could be mistaken/wondering what happens if like two drives are dying and having data migrated to replacements at the same time.
I'm also curious if there are any other issues that would make drive performance not scale linearily. Ie - if I wanted 200MB/sec instead, I would assume four drive stripes should achieve that if I assume minimum transfers are still 60MB/sec of likely drives to use. Much over 200MB/sec I can't see the need for, because my main goal is to saturate a gigabit channel primary connection, while having a 2nd channel running full speed backup of other parts of the array or being used for higher speed housekeeping.
Could someone help me understand what ZFS actually needs to be doing in the background, how demanding it is, how often its doing it and similar? Between recovering from bad drives, scrubbing the disks looking for corrupt data, silvering new disks, writing out space allocation requests (if it gets 1 bit of a file demanding 50gigabytes contiguous does ZFS just pre-allocate it out as one block at max speed?), writing to various logs or journals, and all other housekeeping...
