mv command crashed my TrueNAS 13

[Kevo]

I decided it was time to get rid of AFP and put my Time Machine share on SMB. I didn't realize how TrueNAS lays it out at first. But I ended up in the situation where my original backup was at the root of my dataset and my new backup was inside a child dataset named for my user. I thought no problem I'll just mv my old backup into the new dataset.

After many hours of copying data my server locked up with a bunch of errors on the console and was unresponsive. Ended up having to reboot.

The errors were like

pid xxxx (httpd), ... was killed: failed to reclaim memory

I have no idea how many there were, but I was unable to access the server on the console, by web, or by ssh.

I rebooted and it came up fine from what I can tell.

I am still left with the need to move my backup. I'm wondering what might be a better option besides mv. Only thing I can think of is rsync. I hope it would not crash my server like mv did, but does anyone else have any other suggestions?

 

[mav@]

I'd ask you to create a ticket attaching debug archive. But since system didn't crash, it is unlikely we have kernel dump, unless you've used NMI button to reset it. So at most we may have some logs.

`mv` between datasets/filesystems is effectively a "copy and delete". Other than that you could use local ZFS replication, that would also keep all the file attributes.

 

[Kevo]

Are the dumps still found in /data/crash? If so, then I think you are right. Whatever happened it didn't actually crash, just killed everything and became useless.

I didn't think about trying to zfs replicate. There were two backup images in the old dataset, so I'm not sure if I could only replicate one directory to the new dataset. I'll look that up for future reference.

I did go ahead and start up rsync. There were some files from the mv in the new dataset after reboot so rsync made sense to me. I've been monitoring it while it runs and it seems to be working without any memory increase so I expect this to complete without error.

 

[anodos]

Are the dumps still found in /data/crash? If so, then I think you are right. Whatever happened it didn't actually crash, just killed everything and became useless.

I didn't think about trying to zfs replicate. There were two backup images in the old dataset, so I'm not sure if I could only replicate one directory to the new dataset. I'll look that up for future reference.

I did go ahead and start up rsync. There were some files from the mv in the new dataset after reboot so rsync made sense to me. I've been monitoring it while it runs and it seems to be working without any memory increase so I expect this to complete without error.

Were you doing the `mv` operation through the web shell or an SSH session? Generally speaking there should be no need to move data around when switching from AFP to SMB.

 

[Kevo]

It was an SSH session. I turned off the AFP share and setup the same dataset as an SMB share using the timemachine preset. When I went to do my first backup it created a new dataset for my user and a new backupbundle in that directory on the server. That's when I figured out it was setup a bit differently than the old one which was a single dataset that housed multiple backupbundles for multiple users. I also noticed there was no quota option like there was with the AFP timemachine share and found that it set the quota on the new user dataset.

So I thought, no problem I'll just move my old backupbundle to the new dataset. I wasn't thinking it would take ages because it was just a mv command, but it was datasets and not directories despite how it seems, so then I thought well, it might take a whole day but still no big deal.

Anyway it failed in a novel way which I've never seen before on any unix like system and now I am waiting for rsync to finish moving the backupbundle to see if this actually ends up working out or not.

Rsync seems to have setup the transfer using about 160MB of memory and has not increased at all for hours so I'm pretty confident this transfer will succeed. Just not sure how the Mac will react on the next backup. Hopefully things weren't broken during the original attempt.

 

[Kevo]

Rsync completed and a new timemachine backup also completed successfully so it looks like everything was salvaged with no lasting repercussions.