istgt[1880] : istgt_lu_disk.c Error ZFS / Freenas help needed..

[yoyojazz]

Hi there,

I hope someone can help me?

I have been using freenas on a small HP Proliant Microserver box. I have provisioned it as ISCSI to my ESXi and been running VMs on it on its own seperate network. For months this has been solid power up and running multiple VM's.
This was version 8.0.

Just recently I tried to power it up and start up just one VM and the VM would not start from the Central datastore.
I could see all the files in the datastore and browse and create new files.

However when i tried to start up a VM it would freeze and on occasion it would start up the VM but really slow.....on checking freenas

istgt[1888]: isgt_lu_disk.c:6300:isgt_lu_disk_lbexecute_same@ ***error*** lu_disk_write() failed

There is no massive load for these discs I had them under a massive load before but with one VM powering up now it would either fail or power up after a long time.

I looking into this and my GUESS is

The microserver has 4x 2TB drives configured as ZFS2 i think I have about 4.5tb storage...

I presented this as a file extent and NOT as a Device extent.... and created the ISCSI for it.

The file extent has grown massively i think occupying more than 80% of the total storage and I read this can affect ZFS scrubbing? Am i right or totally wrong here?

When i try and delete the contents of hte datastore to bring it down ... it became obvious that the file extent had grown to > 80% and will not shrink down on the physical discs.

I need my data and to try and bring it back

I was on Freenas 8.0.3 but upgraded to 8.3 and still no change?

Can any ZFS/freenas expert advice me please?


Many thanks again for all your help and for taking the time to read this mail.

Cheers

Jazz

 

[cyberjock]

If you look around you'll see that long term performance is killed on ZFS with iscsi. There's a bunch of threads discussing this around. jgreco has a boatload of info about this topic on the forum.

 

[yoyojazz]

Thanks for the reply noobsauce80.

I have started to check out jgreco postings for any fixes.

If fixing the issue is not possible ..the data is more important to me can anyone suggest the easiest way of getting this data off or put the ZFS or iscsi in a state where i can migrate the VM's storage over?

I was thinking via V-Sphere of adding some SAN storage as a datastore and copying this data of the freenas storage to the SAN storage and getting it off there. Is it possible to do in the current state or would the isgt errors make this fail?

Thanks again noobsauce80

 

[yoyojazz]

Thanks for the reply noobsauce80.

I have started to check out jgreco postings for any fixes.

If fixing the issue is not possible ..the data is more important to me can anyone suggest the easiest way of getting this data off or put the ZFS or iscsi in a state where i can migrate the VM's storage over?

I was thinking via V-Sphere of adding some SAN storage as a datastore and copying this data of the freenas storage to the SAN storage and getting it off there. Is it possible to do in the current state or would the isgt errors make this fail?

Thanks again noobsauce80

 

[cyberjock]

I'm not sure. I would just add a hard drive to the machine that is using iscsi and see if you can copy the data to the hard drive.

 

[yoyojazz]

Good idea but the HP microserver has 4 x 2TB hard drives so no empty space :(
I only thought of adding a SAN datastore to vsphere and transfer the Vms over.

I wanted to add a CIFS or NFS connection to the volume but I am scared that as it is already attached to the iscsi as a file extent i would cause some damage and not be able to get the data off it.

Any advice as to how to get the data off fast and without losing any data?

Thanks everyone

 

[jgreco]

See bug 1531 for a bunch of performance issues and tunable workarounds.

If your pool is already more than 80% full, you are in a bad position, because I don't think vmfs supports shrinking.

ZFS is probably fine for iSCSI if you address the issues in 1531, but one of the significant caveats is that even on a STANDARD ZFS filesystem, you want to maintain probably no more than 80% full. With ZFS and iSCSI, due to the fact that the system is going to be doing COW writes anywhere you twiddle data, my feeling is that you get into sucky territory as you move past about 60% full. So you buy disks twice as big as what you need and go on with your life and ZFS will probably treat you pretty well. The 60% isn't magic, it's just an amount that means that there is a plentiful supply of empty spots to choose from, so that file fragmentation - while never ideal - might not be quite as bad.

The istgt errors you are seeing are what I would commonly associate with a slowly responding iSCSI target. That COULD be caused by lots of writes. If you have some OS that does crazy amounts of I/O while booting (Redmond, I'm lookin' at you) then look at 1531 for tuning tips that may actually decrease performance while increasing responsiveness. But if it is because your file extent is already just so scrambled and fragmented, your best bet is to see if you can shut down the VM('s) on the iSCSI disk and migrate them somewhere.

So I have also talked about avoiding ZFS and using UFS for iSCSI with ESXi and FreeNAS. I won't repeat the discussion. The ZFS-lovers here probably aren't fond of the suggestion, but it bears consideration. There are benefits and downsides to it in any case.

 

[yoyojazz]

jgreco sir you are an absolute star and a gent. Thank you so much for the advice and help.
The explanation has certainly made it allot easier to get a basic understanding for what is going and I now have options to investigate. starting with bug 1531!
With the bother i am going through right now..... i will be sure to go UFS with iscsi in the future)

Cheers again

Jazz