Bad random write performance

[dasfliege]

I finally managed to setup a TrueNAS as my new homelab ESX datstore these says. My setup is as follows:
- Dual-Xeon E5-2690 v3 @ 2.60GHz
- 260GB RAM
- LSI SAS controller
- JBOD connected by 6Gb SAS
- 24x 400GB IBM Enterprise SSD
- lz4 compression, no dedup
- NFS4.1 share connected by 10Gbit ethernet to ESX hosts

Even though IOPS are okay, especially the random write througput seems pretty low for this setup.
Unfortunately the reporting section of my trueNAS doesn't show anything. Just empty graphs, which makes it quite difficult to locate a bottleneck. CPU and memory is pretty idle even during benchmarking.

Is there any simple tweak i may forgot which could explain this performance? Do you guys have any idea why my reports section doesn't shows any graphs?

Best thanks in advance!

 

[NugentS]

How is the pool setup, how many vdevs etc

 

[sretalla]

a TrueNAS

Which version?

 

[dasfliege]

Which version?

12.0-U7

 

[dasfliege]

How is the pool setup, how many vdevs etc

I guess its a single vdev Raid 6. I've just added all the disks and selected the raid level. What would be best practice for 24 SSD disks?

 

[Samuel Tai]

Any RAIDZ* vdev should only be at most 6-8 drives wide. For straight IOPS, you should consider an 8-way stripe of 3-way mirrors.

 

[dasfliege]

Any RAIDZ* vdev should only be at most 6-8 drives wide. For straight IOPS, you should consider an 8-way stripe of 3-way mirrors.

That would mean i will only have 1/3 usable space? I don't need max performance. I rather go with the largest possible capacity that still provides a decent read AND write performance. Two paritiy disk are good enough, as i have instant replacement for them in case of a failure.

 

[Samuel Tai]

Then consider a 4-way stripe of 6-way RAIDZ2s. This will provide a good balance of IOPS, capacity, and data safety.

 

[jgreco]

single vdev Raid 6

There's no such thing in ZFS. You might have meant RAIDZ2, but that is not the same thing.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to translate terms on the reader's side when reading posts...

www.truenas.com

That would mean i will only have 1/3 usable space? I don't need max performance. I rather go with the largest possible capacity that still provides a decent read AND write performance.

That would be two-way mirrors. RAIDZ is not likely to give you the amount of space you think; parity utilization does not work the same as it does on RAID5 or RAID6. Performance also sucks.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[dasfliege]

Yes, RAIDZ2 is the term i was searching for

Sounds like it's quite crucial to understand ZFS specific behavior in order to choose the right configuration for everyones special needs and requirements? Is there any guide showing some example configurations and how they compare to each other in terms of capacity, performance and redundancy? And most important: How i have to configure them correctly in TrueNAS?

 

[jgreco]

guide showing some example configurations and how they compare to each other in terms of capacity, performance and redundancy?

Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!

Slideshow explaining VDev, zpool, ZIL and L2ARC and other newbie mistakes! I've put together a Powerpoint presentation(and PDF) that gives some useful info for newbies to FreeNAS. I decided to create this slideshow because in the last 5 months I've been on this forum I've seen a lot of people...

www.truenas.com

ZFS Primer

ZFS is an advanced, modern filesystem that was specifically designed to provide features not available in traditional UNIX filesystems. It was originally developed at Sun with the intent to open source the filesystem so that it could be ported to other operating systems. After the Oracle...

www.truenas.com

And most important: How i have to configure them correctly in TrueNAS?

Storage in ZFS really breaks down to only two types. Mirrors, which excel at everything, but generally provide less usable space than RAIDZ, or RAIDZ, which excels at storing long sequentially written files with no overwrites (i.e. ISO files great, VMDK files not great).

That's the BIG decision. Once within, you've already ended up definitely beyond halfway to the decision-making finish line, which is why we focus on pool design as a huge consideration.

From there, if you're just storing your ISO's, a trite task for both mirrors and RAIDZ, you might well be "done", while if you are doing database of VMDK storage, you need more specialist knowledge, as in what I talk about in the block storage article.

 

[dasfliege]

Wow, this is a great presentation. Thanks for the link!
But it is written there, that ZFS based storage may not be an ideal solution for ESX datastores unless you are willing to spend hours to tweak it. Is this still a valid statement?

I've ended up trying the above mentioned configuration with "4-way stripe of 6-way RAIDZ2s". But performance is even worse then on a single 24-disk RaidZ2.

If i am not willing to spend hours to tweak it and not willing to loose half or even 2/3 of my available capacity by utilizing a mirror configuration instead of RaidZ1 or 2, it seems that i may have to find an alternative to ZFS based storage?

 

[jgreco]

ZFS based storage may not be an ideal solution for ESX datastores unless you are willing to spend hours to tweak it. Is this still a valid statement?

That's a naive statement, I'm guessing from someone who tried to under-resource it. ZFS eats gobs of resources, but if you follow the recommendations in the block storage article I linked to, it should fly with minimal screwing around.

 

[dasfliege]

The statement is from the powerpoint presenatation you have shared

I also went through you article about block-storage. Even though im using NFS which isn't block, i guess that these recommendation still applies as the underlying filesystem is block?

I checked several different configuration. Even 8-way stripe over 3-way mirrors doesn't give me better performance then a single 24-disk RaidZ2. But as i read in your article, a simple disk benchmark may don't give the results that are crucial for VM-storage as VM-stroage heavily relies on parallel disk operations. I may give it a try executing benchmarks from different VMs at the same time. If this will result in more or less the same performance as on a single VM, i guess that my setup is fine.

Performance should not be a problem in my case. I run TrueNAS software native on a IBM server with dual xeon CPU and 260GB RAM. Guess that should be enough to get decent performance.

 

[jgreco]

The statement is from the powerpoint presenatation you have shared

... in response to you asking about pool design.

Yes, I know. I can throw shade at people while simultaneously acknowledging that the overall material isn't bad. That's why I even explained it as likely being "someone who tried to under-resource it". I did give you the link to my "path to success" article first, which specifically mentions gobs of RAM. I gave you the other two presentations in response to your question about pool design; they should be taken in that light. It's a mistake to take things out of the context in which they're offered.

Most of the posters here are hobbyists or SOHO users, and may not be willing to throw a half of a terabyte of RAM and dozens of hard drives at it in an attempt to make a credible VM storage platform. I believe Cyberjock wrote his pool slideshow at a time before he worked for iX, and had probably seen lots of frustrated posters on the forums trying to do these things on systems that were far too small.

Even though im using NFS which isn't block, i guess that these recommendation still applies as the underlying filesystem is block?

How is that not block? You think it's not block just because the protocol isn't ENFORCING block? That's not the definition of block storage.

The fundamental problem we're looking at is that with a CoW filesystem, if you write a large sequential file (and let's define it as being written contiguously for the sake of discussion), and then overwrite a block in the middle of that file, you get a NEW block allocated elsewhere on the filesystem. This means that you no longer have a fully contiguous file, which means that when you access it sequentially, you get a performance blip in the middle of it where the drive seeks back and forth.

Now once you've overwritten a hundred thousand blocks, because you wrote files, updated file atimes, ran OS updates, whatever, you now have a significantly noncontiguous file.

And it really doesn't matter if you were accessing this via iSCSI on a zvol, or iSCSI on a file, or NFS on a file. It's the act of overwriting that is causing the issue, and which is why you may want to mitigate that. The protocol used is a distraction.