ISCSI (ESXI 5.5) awfully slow after Upgrade 9.2.1.3->9.2.1.6RC2

[Thomymaster]

Hi All

Yesterday i upgraded my FreeNAS Box to 9.2.1.6RC2. After this i noticed that the ISCSI performance is awfully slow (i use ESXI 5.5 to mount my Datastore which is zVol based).
I get 5MB/s write compared to approx. 40MB/s ion 9.2.1.3

Nothing was changed (there are no tuneables active) my system specs are:
Build FreeNAS-9.2.1.6-RC2-72b8479-x64
Platform Intel(R) Xeon(R) CPU E5345 @ 2.33GHz
Memory
16362MB

 

[Thomymaster]

I use the user-space ISCSI target (istgt) and like i said, nothing has changed :(

 

[dlavigne]

It shouldn't be slower...

Does running this command improve things? sysctl vfs.zfs.vol.mode=1

If not, are you willing to try the experimental kernel iSCSI to see if that is better?

 

[dlavigne]

To modify my post above, it appears that that sysctl only affects pools imported after that sysctl change.

However, you can create a Tunable called vfs.zfs.vol.mode with a value of 1. That will require a reboot as tunables are only read at boot time.

 

[Thomymaster]

OK but if i get it right when i set the tuneable i have to export and reimport the pool afterwards. How can i do this via GUI/CLI?

So this issue is propably caused by the way how the HDDS are handled in 9.2.1.6RC2 (https://www.mail-archive.com/svn-src-all@freebsd.org/msg82617.html)

 

[mav@]

If you set it as tunable, then reboot is required and sufficient. If you try to set it as sysctl without reboot -- this should be theoretically possible, but you'd have to stop all services accessing the pool, export pool, import pool, start services ... I would better reboot. ;)

Yes, you mentioned right commit message for this tunable, though it is not yet clear whether it is the cause of your slowdown, or, if so, why. So more data from you would be highly appreciated.

 

[Thomymaster]

Yes, applying the tuneable did the trick, now it is as fast as before (which is still not very fast). :)

Whereas gstat showed this when suspending a virtual machine (and this took approx 30 minutes)

0 548 37 224 0.1 511 3352 0.3 90.7| zvol/zpool1/zvol1

0 548 37 224 0.1 511 65352 0.3 13.7| zvol/zpool1/zvol1

So theres 65MB/s write performance compared to 3 MB/s before.

And i don't see theses messages regarding increased I/O anymore ("Events" in the vSphere client).

 

[mav@]

I am sorry, what are these numbers? Unless I missing something, without the tunable set, on FreeNAS 9.2.1.6 you should not see ZVOLs in gstat at all, only in `iostat` or `systat`. And how did you managed to get so equal other numbers there? ;)

 

[mav@]

Could you also tell a bit about your pool? What is the pool configuration? Do you have any ZIL?

 

[mav@]

I've found a bug in new ZVOL code. Please try next nightly build without that tunable set and tell us what you get. I would expect much better performance.

 

[Thomymaster]

@mav

Yeah ill try the nightly tomorrow and report the results, thanks for submitting the bug (https://bugs.freenas.org/issues/5425) :)

My ZFS consists of striped-mirrors (4x1,5TB Seagate Barracuda 7200.11 S-ATA2 HDDs), i have a ZIL (because its always present, but no SLOG) but no L2ARC.

With the above numbers from "gstat" (and the zvol shows up there because i set the tuneable and the HDDs are then handled via GEOM i guess) i just wanted to point out that the performance has increased with the tuneable beeing set

 

[Thomymaster]

OK sorry i cant test as it is a productive system :(

 

[mav@]

It is your system and your call. Though I am somewhat disappointed. :( Any way thanks for the report.

 

[Thomymaster]

Ill test it with my virtual FreeNAS box then :)

 

[mav@]

Not sure what can it give. I don't trust benchmarks run inside virtual environments. I don't think there can be any implications to find except performance. Those are really interesting to see, but in real environment, where you have something to compare to.

 

[Thomymaster]

I agree, but the company cannot afford a fileserver downtime of an hour or so. I can try with the nightly after working hours this evening maybe.

 

[Thomymaster]

Hi

I tested with 9.2.1.6-RELEASE and it seems to be working without the tuneable now :)

 

[mav@]

Cool. Thanks. Any performance differences?

 

[cyberjock]

I agree, but the company cannot afford a fileserver downtime of an hour or so. I can try with the nightly after working hours this evening maybe.

If so you should be purchasing a HA setup with TrueNAS from iXsystems....

 

[Thomymaster]

Hi

Yeah like i said above i had the same performance as with 9.2.1.6RC2 (65MB/s). In the meantime 9.2.1.6-RELEASE is out, so the fix should be included there as well.