With Ssd zil and l2arc would nfs provide better throughput than iscsi?

[ror]

jgreco,

I get your point about doing one's due diligence and read the docs. And I agree with you.

Now, I am just curious about the default iSCSI config of Solaris based OS in terms of honoring sync write. Of course, I can and WILL test this too but just wondering if you guys have already had experience with it and wouldn't mind sharing.

 

[jgreco]

Me? No. Because quite frankly I don't sit here memorizing what the defaults for various products are. There are too many products and too many versions of them, and this is a trite setting. I remember big things ... things like "Nexenta is a frickin' train wreck."

But don't let that discourage you. Feel free to experiment and report back.

 

[pbucher]

Based on some comments iSCSI

Now, I am just curious about the default iSCSI config of Solaris based OS in terms of honoring sync write. Of course, I can and WILL test this too but just wondering if you guys have already had experience with it and wouldn't mind sharing.

Based on some comments iSCSI might be better implemented on Solaris & friends and hence there isn't a need to tweak the sync setting, then again FreeBSD 10 has a brand new iSCSI module so the current one in FreeBSD 8/9 is probably not the best. Testing is the only sure way to discover how it all really works.

General hint: anything that turns in good performance against a ZFS based SAN/NAS with ESXi and you don't have fairly expensive SLOG device(or you don't see writes hitting the SLOG) is cheating in some way and you are open to data corruption of some sort. It's just the design of ESXi & ZFS, when you want maximum safety you need some expensive hardware if you want any kind of performance. That's not a knock against ESXi or ZFS, in fact it's a compliment that their design allows the kind of data protection you can get out of the box.

 

[jgreco]

General hint: anything that turns in good performance against a ZFS based SAN/NAS with ESXi and you don't have fairly expensive SLOG device

Just curious, would you label my (ab)use of a mainboard's built-in LSI SAS2208 controller along with some run of the mill hard disk to be "fairly expensive SLOG device" or does that qualify as cheating? From my perspective the controller was "free" but the BBU module for it did admittedly cost about $200.

Any word from the front lines on how vSphere 5.5 is?

 

[ror]

Based on some comments iSCSI


Based on some comments iSCSI might be better implemented on Solaris & friends and hence there isn't a need to tweak the sync setting, then again FreeBSD 10 has a brand new iSCSI module so the current one in FreeBSD 8/9 is probably not the best. Testing is the only sure way to discover how it all really works.

General hint: anything that turns in good performance against a ZFS based SAN/NAS with ESXi and you don't have fairly expensive SLOG device(or you don't see writes hitting the SLOG) is cheating in some way and you are open to data corruption of some sort. It's just the design of ESXi & ZFS, when you want maximum safety you need some expensive hardware if you want any kind of performance. That's not a knock against ESXi or ZFS, in fact it's a compliment that their design allows the kind of data protection you can get out of the box.

I hear you as from my extensive testing ( many months), no matter how I slice and dice it, it looks like I need a super fast ZIL device if I want ESXI and ZFS (nfs or iscsi) to perform with data security in mind. So, I am waiting for the Intel DC S3700 100GB (with super capacitors) to arrive tomorrow and hopefully I see some good number.

One other question. So, let's say I have a fast ZIL working. Should I go for RaidZ(1 or 2) or Mirror? Since the ZIL is going to help with IOPS, wouldn't it mask out the slow write of a RAIDZ vdev? I am trying to find the best balance in terms of cost and peformance by using a combo of expensive fast ZIL device to mask out the cheap SATA disks in RAIDZ.x

Thanks!

 

[jgreco]

I wouldn't fixate on "cheap SATA disks". Any spinny rust is slow by default. You have your choice between butt slow and slightly less slow and only modestly slow.

But RAIDZ is never a winner for speed if you are interested in IOPS. A good discussion of this is at http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance

You need to remember that the SLOG device is not in the write path; it is not a write cache. It merely means that the pool doesn't have to be responsible for hosting the in-pool ZIL, and this allows ZFS to aggregate writes and treat the pool in about the same way it would if sync hadn't been requested. That's cool but it doesn't accelerate writes beyond what a RAIDZ pool would have been capable of.

For best performance in an environment requiring redundancy, mirror drives (pairs or trios) together into vdevs, and then stripe the resulting vdevs. ZFS will do a competent job of balancing I/O's. It can actually get kind of fast.

 

[pbucher]

Just curious, would you label my (ab)use of a mainboard's built-in LSI SAS2208 controller along with some run of the mill hard disk to be "fairly expensive SLOG device" or does that qualify as cheating? From my perspective the controller was "free" but the BBU module for it did admittedly cost about $200. :)

Any word from the front lines on how vSphere 5.5 is?

I wouldn't say it's an "fairly expensive" setup but certainly not a cheap one. Now I'm going to have to move that up on my list of things to try and I was just gearing up to use my hardware RAID cards to build a Linux mirrored SAN(using drbd), that might have to wait until I try this out.

On 5.5 I'm guessing is a few weeks out, not much time left until the end of the year so they can't wait too long.

On pool configs: When I was building one of my main SANs I initially did 8 mirrored pairs and when I didn't get enough bang out of it I tried a pool of 3 RAID1Z devs(with 5 drives in each) and found that for simple raw I/O the RAIDZ pool edged out the mirrored pairs. I do agree however that for IOPS that the mirrored pairs should win. Since my work load doesn't generate a ton of IOPs I went for the disk space of the RAIDZ devs.

Good summary on what a SLOG can and can not do. I should some time redo my benchmarks now that I know more about ZFS and I've worked out the kinks of my virtualized SAN.

 

[ror]

Hi guys, hope you don't mind me posting a link to Hard Forum about my test result with the Intel DC S3700 100GB as ZIL device. The result is exactly as what you would expect with regards to sync write in ESXi environment.

There are screen shots to prove! :)

http://hardforum.com/showpost.php?p=1040226516&postcount=5398

 

[ror]

On pool configs: When I was building one of my main SANs I initially did 8 mirrored pairs and when I didn't get enough bang out of it I tried a pool of 3 RAID1Z devs(with 5 drives in each) and found that for simple raw I/O the RAIDZ pool edged out the mirrored pairs. I do agree however that for IOPS that the mirrored pairs should win. Since my work load doesn't generate a ton of IOPs I went for the disk space of the RAIDZ devs.

I plan to test this for my type of work load which is Veeam replication/backup for Disaster Recovery at a co-location. mirror vs raidz (several 3-disks vdevs to help with IOPS). If it's a bit lower in IOPS and time, then it might be worth it to gain the capacity.

Now, I plan to use 4TB disks, would Raidz be okay in terms of re-silvering time and/or data safety?

Also, would another decent SSD help with L2ARC in ESXi environment?

 

[pbucher]

I plan to test this for my type of work load which is Veeam replication/backup for Disaster Recovery at a co-location. mirror vs raidz (several 3-disks vdevs to help with IOPS). If it's a bit lower in IOPS and time, then it might be worth it to gain the capacity.

Now, I plan to use 4TB disks, would Raidz be okay in terms of re-silvering time and/or data safety?

Also, would another decent SSD help with L2ARC in ESXi environment?

I use veeam & NFS to backup a ESXi box all the time to a RAID-Z setup.

Anyways raidz does fairly good at re-silvering, the key thing is not to make 15 disk vdev. On the L2ARC I'd through RAM at the problem first if it appears to need more performance, also L2ARC on disk eats RAM anyways because it needs to keep track in RAM what's on the disk. Unless you have something really read intensive I'd skip the SSD for L2ARC.

Post back your experience on mirror vs RAIDZ for your veeam job I'll be interested to hear the comparison.

 

[cyberjock]

Hi guys, hope you don't mind me posting a link to Hard Forum about my test result with the Intel DC S3700 100GB as ZIL device. The result is exactly as what you would expect with regards to sync write in ESXi environment.

There are screen shots to prove! :)

http://hardforum.com/showpost.php?p=1040226516&postcount=5398

ror,

Thanks for the testing. My 2 cents... the benchmarks mean nothing because there's no guarantee that any of those tests even remotely resemble the actual workload you can expect to see. But, there's one tidbit that IS valuable...(especially the bold, in my opinion).

- In real test, veeam replication to ZFS datastore, with the S3700 as ZIL device, it is only a couple of minutes slower in completing the replication than when sync=disabled. Without a ZIL device, replication completed several hours slower.

Conclusion:
S3700 is a very good ZIL device to help with sync write peformance and data security in ESXi, NFS, and iSCSI environment.

I'd worry less about the benchmark results and worry more about actual real world performance. We've had MANY people that run all sorts of benchmarks and get really good speeds, but then try to run 2 machines over iSCSI and the roof comes crashing down. They are clearly far too dependent on benchmarks to tell you what is faster and what is slower. What you really need to do is work on making it faster when its "too slow". What is "too slow"? Well, that's when its too slow for what you need it to be. My answer is vague, and its deliberately so.

Glad its all working for you though. :)