zpool not degraded, but certain files cause transfers to hang

[averyfreeman]

Hello,

I have a pool on a general file server with 6 x 2TB drives configured in a three-group mirror that are very old. The system, for all intents and purposes, appears to be working normally. The pool does not report as degraded.

However, I have been trying to move certain files using rsync and every time it gets to a certain folder it hangs. It's always transferring the same, or near the same (same folder), file every time it hangs.

Also du command hangs when gets to folder.

I have scrubbed the pool several times now and it does not appear to improve the behavior. I also have tested the drives individually using smartmontools and also taken them all out and individually tested each surface using HDDtools on a wintel machine. They all come back basically OK (just old).

Does anyone have any ideas for what I should do to be able to finish backing up my files?

PS: I created the pool with Corral but I was using it in 11.1-U4, right now I have it in a FreeBSD VM because it was making FreeNAS hang at boot.

 

[Dice]

Clues might be found here:
https://forums.freenas.org/index.ph...ng-from-freenas-corral-to-freenas-9-10-11.36/

(love your profile pic)

 

[Ericloewe]

Have you reset the aclmode to something supported?

 

[averyfreeman]

I don't see why files wouldn't rsync as root regardless of acl (?).

Anyway, I gave up and destroyed the pool since I had already backed up enough files to be comfortable - I am trying RaidZ2 now. I was able to copy all the files off my backup server - Doesn't appear to have any issues.

Oh I should probably mention, I think it was actually the enclosure. I don't think it started that way, but then I had some issues with the enclosure that made it way, way worse:

I do NOT recommend iStarUSA trayless hot swap drive racks - when I was testing, one of the two I have (the 3-drive 2x5.25 model) the door broke off in my hand under very little force - the metal hinge literally came apart and it is irreparable!

I couldn't get the drives fully back in without the door because of how the mechanism works, so I pulled all the drives out and connected them manually, and my server is laying open with a bunch of drives piled on top of it waiting for a SuperMicro drive rack to arrive in the mail...

I have three types of hot swap cages - the ones in a SuperMicro SC826TQ 2U case, one Rosewill RSV-SATA-Cage-34, and two iStarUSA BPN-DE230SS - one of which is going in the garbage. Neither of the other two chassis has given me any issue. I actually like the Rosewill slightly better than SM but it's close so I went with the one that had 5x drives in the same 3x 5.25 bay space (will still have to hard-mount one drive - grumble).

Anyway, thanks everyone for your help.

 

[parpar]

Have you checked on what signal commands are hanging if they access the folder?
Try a ps -axHl -O lwp and check what is in the MWCHAN field, it may be related to my issue

 

[averyfreeman]

Have you checked on what signal commands are hanging if they access the folder?
Try a ps -axHl -O lwp and check what is in the MWCHAN field, it may be related to my issue

That's a good idea. Unfortunately (fortunately?) the pool is back online and working fine. I honestly gave up after a couple round of smart checks and just destroyed the pool, made a new pool and restored from backup (this time Z2 instead of 3-vdev mirror). I am going to have to put all this stuff in my toolchest to use if it happens again, though (wouldn't surprise me).

Someone on FreeBSD forum also recommended using top in another console during the time the rsync takes a dump. They're both great ideas. I have switched to FreeBSD since I couldn't load the pool initially in FreeNAS, but I made it with FreeNAS so I figured coming back here for help couldn't hurt. This is a great forum, everybody has good ideas.

I wonder if dtrace could be used, also? Always heard so much about it on SmartOS forums (which I tried but gave up on due to bugs, bugs, bugs and now just use ESXi like a normal person ;) ).

 

[Ericloewe]

I wonder if dtrace could be used, also?

Probably, but the learning curve is on the steep side.