<div class="bbWrapper">Hi,<br />
<br />
I'm running a fairly large FreeNAS installation and exporting 3 iSCSI LUNs to a Windows Server 2016. (1 LUN is exported to a Linux server via iSCSI, I don't believe this is having issues)<br />
I am using MPIO on all LUNs, connected via 2x 10GbE<br />
<br />
LUNs are sized as follows:<br />
a) 45TB<br />
b) 2TB<br />
c) 85TB<br />
<br />
Recently, I've been starting to get issues where Windows will not let me browse certain folders. The exact error message is:<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>X:\ is not accessible.
The request could not be performed because of an I/O device error.</pre></div><br />
<br />
<br />
This problem seems to come and go, if I revisit the folder at a later time it may work. For example right now as I type this, it's working fine - all LUNs and all folders are working with no issue. However a few hours ago without me touching anything, I received the I/O errors from Windows.<br />
If I offline the disk in Disk Manager and online it again, things work fine for a while.<br />
It's important to note that this system has worked flawlessly for over a year now at the very least, it's almost as if overnight the issues started occurring.<br />
<br />
Additionally, I occasionally see the following errors in the Windows event log:<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>The system failed to flush data to the transaction log. Corruption may occur in VolumeId: X:, DeviceName: \Device\HarddiskVolumeX.
(The I/O device reported an I/O error.)</pre></div><br />
<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>Disk X has crossed a capacity utilization threshold and used Y bytes. When the threshold was crossed, the pool had Z bytes of remaining capacity.</pre></div><br />
<br />
<br />
A ZFS Scrub comes up fine, a Windows chkdsk /R also comes up fine with no errors.<br />
<br />
This FreeNAS system was previously running 9.10, I've upgraded it to 11.1 recently however the issue persists.<br />
It's running on Supermicro hardware with the following:<br />
<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>2x Xeon E5-2620 v3
128GB RAM
Intel 10GbE SFP+ NICs
Raw space: 282TB Used, 53TB Available (84%)
11x RAIDZ2 vdevs containing 6 drives in each (HGST HUS726060AL4210 A7J0 - 7200 SAS)</pre></div><br />
<br />
I'm hoping someone may be able to point me in the right direction.<br />
I have a feeling this is less FreeNAS related and more hitting Windows NTFS limits (although I did check and I'm below the maximums). I know switching to the ZFS filesystem and using FreeNAS to expose shares could probably work better in my case, but my trade off would be losing the 2x 10GbE MPIO if I did that (last I saw, SMB3 multi-channel was still experimental)<br />
I also know of the 80% performance degradation, and the 50% capacity utilization when using iSCSI LUNs, but from my understanding that would just affect performance and not give me errors such as what I'm experiencing?</div>
<div class="bbWrapper">Where to start...could be any number of things here. Start simple...do you have a scheduled SMART test on the FreeNAS box? If so, do all drives report healthy? It sounds like your pool may be getting pretty full so fragmentation issues may occur which could manifest into something like this. Any other errors on the Windows box that may point to iSCSI timeout issues? It may also be useful for us to see the output (pasted in code blocks) of the following commands from FreeNAS:<br />
zpool status<br />
zpool list</div>
<div class="bbWrapper"><blockquote data-attributes="member: 81946" data-quote="Antix" data-source="post: 429713"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=429713"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-429713">Antix said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
I also know of the 80% performance degradation, and the 50% capacity utilization when using iSCSI LUNs, but from my understanding that would just affect performance and not give me errors such as what I'm experiencing?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>These problems are going to go hand-in-hand. With block-level storage, the OS is going to be very unforgiving if there is any latency or delay in response. If ZFS is super bogged down, and taking "forever" to complete the writes or reads, Windows will think the device has dropped and is no longer accessible, giving you weird errors like you're seeing.</div>
<div class="bbWrapper"><blockquote data-attributes="member: 81946" data-quote="Antix" data-source="post: 429922"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=429922"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-429922">Antix said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
<code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>XXXXXXXXXX-zpool 336T 283T 53.3T - 31% 84% 1.00x ONLINE /mnt</kbd></code>
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
I would bet dollars to donuts that this is your problem: 84% utilization is insanely high for ZFS, much less ZFS block storage.<br />
<br />
Furthermore, even though you've got 11 vdevs, they are all RAIDZ2, so I'm surprised you have enough I/O without a SLOG. I'll be honest though, I don't have much experience with ZFS at that scale other than stripped mirrors.</div>
<div class="bbWrapper">Thank you for your input, random I/O has been generally fairly good - this system stores backups so I don't need insanely high IO<br />
<br />
What are your thoughts on migrating the LUNs to a zvol and presenting them over SMB, rather than iSCSI?<br />
I'll also keep in mind that I can add extra disks but at this scale the 80% target starts to hurt so I don't know how sustainable that might be</div>
<div class="bbWrapper"><blockquote data-attributes="member: 81946" data-quote="Antix" data-source="post: 430008"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=430008"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-430008">Antix said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
What are your thoughts on migrating the LUNs to a zvol and presenting them over SMB, rather than iSCSI?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
That doesn't make any sense. A ZFS volume is just a dataset that represent a block device. If you want to access the data over SMB, you just create a regular dataset and move the data to it. Are you not using zvol's right now as the extent type for your iSCSI lun's? Can you post a screen shot of iSCSI extents from the FreeNAS gui?</div>
<div class="bbWrapper">A zvol is block storage, so I'm assuming you mean a filesystem dataset. If you put the data directly on the filesystem and present via SMB, you should have better performance.<br />
<br />
The reason behind the utilization targets is because ZFS is a copy-on-write file system (<a href="https://pthree.org/2012/12/14/zfs-administration-part-ix-copy-on-write/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://pthree.org/2012/12/14/zfs-administration-part-ix-copy-on-write/</a>). This means that, any time data changes, ZFS copies the entire block to a new block, and then marks the old block as free space. This prevents the write-hole problem. However, FreeNAS also uses a tree of cryptographic hashes to verify data. When data changes, all the nodes in this tree also change. And they must be copied and written. These two factors combine to heavily fragment the file system. And the best solution to combat fragmentation is to leave a bunch of open space on the drives.</div>
