Arc Hit Ratio is at 100%?

[jgreco]

yes it was, thank you for all that information. when i used to be a system engineer i used NetAPP filers so i never had to worry about hitting the limits , it just worked and worked beautifully. but that was a FAS6040 (with 10gigE) ...
but now that i am using horrible hardware i never thought that iSCSI would be hit that hard and perform this badly .. sigh, need to rethink my plans and approach now that i have to work with less than idle hardware.

Really, anyone coming from a WAFL environment ought to be able to cope ... NetApp is the poster child for poor performance due to its CoW implementation. You remediate through adding more spindles, cache, etc., adjusting the filer's design to accommodate the workload.

For example, for the new VM storage box we're building here, we need kinda decent write performance and we want very good read performance, so I've started out with a 64GB box with 256GB L2ARC and then am adding three-wide mirror vdevs to it. The chassis has 24 2.5" bays so with 7 2TB vdevs, that's a 14TB pool and three warm spares. You don't want to heavily use a ZFS pool doing SAN storage, so that's maybe 5-7TB usable space. But I'm fully aware I might need to boost to 128GB of RAM to get the sort of performance I want. With three disks in each vdev, though, even random reads should be fairly awesome.

 

[jgreco]

and to increase that, i would need to add more drives/memory? or just memory?

Get rid of the RAIDZ1 and go with mirrored vdevs. Also make sure you've picked a reasonable blocksize, otherwise ZFS will be hurting. Then add more memory. Memory is probably hurting you less than the RAIDZ1, but your setup is all kinds of "bring on the pain."

yes it is, it starts really fast .. i think i saw the first 2 seconds hit 950MByte/sec then start slowing down.... 10 seconds later it goes down to 12MByte/sec and stays there :( so frustrating.

Yup. Your first few seconds, the transaction group in memory fills, starts getting written out, then another transaction group starts to fill, and that all happens fast-as-ye-can-cram-bits-down-the-net, then comes the reality hangover when that second transaction group needs to be committed and it has to sit pending the first group's completed write, and iSCSI basically comes to a crawl.

The problem with a RAIDZ1 vdev is that it has, at best, a single disk's worth of IOPS, and if it isn't set up well, then you can be squandering even that.

 

[yis]

Get rid of the RAIDZ1 and go with mirrored vdevs. Also make sure you've picked a reasonable blocksize, otherwise ZFS will be hurting. Then add more memory. Memory is probably hurting you less than the RAIDZ1, but your setup is all kinds of "bring on the pain."



Yup. Your first few seconds, the transaction group in memory fills, starts getting written out, then another transaction group starts to fill, and that all happens fast-as-ye-can-cram-bits-down-the-net, then comes the reality hangover when that second transaction group needs to be committed and it has to sit pending the first group's completed write, and iSCSI basically comes to a crawl.

The problem with a RAIDZ1 vdev is that it has, at best, a single disk's worth of IOPS, and if it isn't set up well, then you can be squandering even that.

time to do some reading... i will follow those PDF/powerpoint and will let you guys know if i have any question .. because i have no idea how to remove RAIDZ1 and have mirrored vdevs :)
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

 

[yis]

Get rid of the RAIDZ1 and go with mirrored vdevs. Also make sure you've picked a reasonable blocksize, otherwise ZFS will be hurting. Then add more memory. Memory is probably hurting you less than the RAIDZ1, but your setup is all kinds of "bring on the pain."



Yup. Your first few seconds, the transaction group in memory fills, starts getting written out, then another transaction group starts to fill, and that all happens fast-as-ye-can-cram-bits-down-the-net, then comes the reality hangover when that second transaction group needs to be committed and it has to sit pending the first group's completed write, and iSCSI basically comes to a crawl.

The problem with a RAIDZ1 vdev is that it has, at best, a single disk's worth of IOPS, and if it isn't set up well, then you can be squandering even that.

Hey,
quick question, so i destroyed my old volume and i am creating a new one.. just to confirm, i currently have 3 * 3TiB drives and the total usable space after i change it mirror is going to be 2.7TiB ? :O
should i get another 3TiB drive now instead of just creating a volume? or can i just create the volume and add the additional drive later?

 

[Bidule0hm]

Use only 2 drives, put them in a mirror.

Then when you have a fourth drive, add it with the third drive in a mirror striped with the first mirror.

Read the cyberjock's guide if it's not clear (link in his sig or my sig) ;)

 

[Bidule0hm]

Sorry, double post. Forum bug?

 

[yis]

Use only 2 drives, put them in a mirror.

Then when you have a fourth drive, add it with the third drive in a mirror striped with the first mirror.

Read the cyberjock's guide if it's not clear (link in his sig or my sig) ;)

AH! thats what it was, i did read it in the powerpoint but i didnt totally understand what he meant .. now it make sense .. .
If i added a fourth drive after i had the 3, and something happens to that fourth then the entire volume fails. -- sometimes i am not a total retard.

 

[Bidule0hm]

Yes, don't use 3 drives and then add one drive or you'll be in big trouble... :)

 

[yis]

ok memory upgraded to 32gb and added a fourth hard drive.. going to create the mirrored vdevs.. will update later on the performance .

 

[yis]

lol... same results... ISCSI suck sweaty balls. heh

 

[Rick Arman]

lol... same results... ISCSI suck sweaty balls. heh

With iSCSI it seems like RAM is king. you'd want at least 64gb of RAM as well as fast read cache for even decent performance. Not to mention keeping the amount of usable capacity under 50%.

 

[SweetAndLow]

With iSCSI it seems like RAM is king. you'd want at least 64gb of RAM as well as fast read cache for even decent performance. Not to mention keeping the amount of usable capacity under 50%.

Whoa! No need to bring back a thread that is over 2 years old.