iSCSI Volume corrupting after being attached to host

[Austin Downing]

I setup a FreeNAS unit for backups and I have found that my data is as far as I can tell corrupting for no reason. I have created a 6 disk RAIDZ2 array and am using a iSCSI link to my backup server using a Microsoft iSCSI software link into our VM. I am using my dedicated iSCSI switches to connect our FreeNAS to our clusters of ESXi hosts.

I have no errors on the GUI but on the console it is showing many errors that say.

istgt_lu_disk.c:6168:istgt_lu_disk_exec, lu_disk_bread() failed.


As of now all of my backups have disappeared out of my windows explorer except for one backup. After I do a disk health check I find lots of corrupted files inside of our volume.

 

[jgreco]

istgt_lu_disk.c:6168:istgt_lu_disk_exec, lu_disk_bread() failed.

That sounds bad. What do "zpool list" and "zpool status" say? And why are you connecting ESXi to this? That just sounds like asking for trouble. What sort of filesystem are you building on the volume?

 

[Austin Downing]

I am not connecting ESXi to this. I connecting it directly to the VM using the Microsoft Software iSCSI initiator. Formatted using NTFS I already deleted the pool and recreated it after everything corrupted. I will get you zpool list and zpool stats. Here is what my console looked like previously.

 

[Austin Downing]

Bump as I don't know how to fix this still.

 

[jgreco]

I think we were waiting for you to report "zpool list" and "zpool stats".

I hope you are not trying to attach multiple clients to an iSCSI disk using NTFS. It isn't a cluster-aware filesystem and it won't work, and in fact will fail spectacularly in a manner similar to what you describe.

Also, ZFS and iSCSI are odd bedfellows, and you kind of need to go to a little extra work to make sure that responsiveness is sufficient that your initiator software doesn't freak out and drop connections.

 

[Austin Downing]

I hope you are not trying to attach multiple clients to an iSCSI disk using NTFS. It isn't a cluster-aware filesystem and it won't work, and in fact will fail spectacularly in a manner similar to what you describe.

Wow I feel stupid, as a IT professional this is something I should have thought of, I did have a second host connected as a way to monitor the files but everything always showed up as blank. Derp. Let me recreate the system without multiple hosts connected and see how it responds.

 

[jgreco]

Wow I feel stupid, as a IT professional this is something I should have thought of, I did have a second host connected as a way to monitor the files but everything always showed up as blank. Derp. Let me recreate the system without multiple hosts connected and see how it responds.

B^D "seen dat before"... but doesn't explain the errors on console, which is more reminiscent of bug 1531 ("zfs responsiveness tuning exercise")

 

[Austin Downing]

Let me run full backups of my entire system and we see if we get more errors, if I do I will get you those logs you need. As I am new to FreeNAS do you mind guiding me through getting "zpool list" and "zpool stats"? Looks like I have some information to give some people as they copied my setup to use in their offices.

 

[Austin Downing]

Just as a update, after removing the second iSCSI connection all of my errors and data corruption stopped occuring. Thank you so much for your help guys!

 

[jgreco]

Very happy to hear it. Fortunately you acquired that bit of knowledge at a relatively low cost. We've had a few others come in here, angrily, having lost much more than just some backups.

 

[Austin Downing]

Very happy to hear it. Fortunately you acquired that bit of knowledge at a relatively low cost. We've had a few others come in here, angrily, having lost much more than just some backups.

I can feel for them, at the same time inside of a business I would have a really hard time using FreeNAS for anything but a backup system. The support I get from our SAN vendors can be invaluable at times.

 

[jgreco]

Yes, and there are some of us who avoid vendors for that very reason... I prefer to be able to inspect the code, and it means that I can often solve the problem myself, given sufficient time and motivation. Vendor "support" is often a euphemism to "cannot get any insight into what is going on inside vendor's magic black box, so you have to pay them to have their wizards solve the problem."

This turns out to be an interesting exercise down in the value-priced end of the NAS spectrum, because the manufacturers are not making sufficient profit from the NAS to provide free support, and they don't really offer reasonable paid support options either.

If you actually require support, iXsystems can provide it, and my guess is that you can even buy a TrueNAS from them and it'll still be cheaper than what you can get a Netgear ReadyDATA system for.

But really, a SAN vendor wouldn't have saved you from the mistake you made. That's the point I was making; people *do* make the mistake of multiply mounting an NTFS filesystem on a SAN block device, and this generally leads to data loss. It is something I consider to be strictly operator error, not the fault of FreeNAS.

 

[Austin Downing]

I agree this was my fault and I have learned, still it would have been nice to be able to call a vendor and explain to them what is going on, and have them help me. I am not saying anything bad about FreeNAS, in fact I love the product. I just believe that a production enviroment should be for the most part industry standard so that when a problem is had I can call a vendor to get a problem fix.

 

[cyberjock]

You have something that is roughly the same. You even used it. It's the forum :D

If having a voice communcation with a vendor was important to you, you could have called iXsystems. I'm sure that for a fee(or maybe free if they know what mistake you made without even looking at your system) they could inspect your system and tell you of whatever problem you have either now or in the future.