disk pool throwing alert

[samarium]

or even being updated after @media snapshot was taken

 

[georgelza]

Those 2 files.

Was that the full list from the rsync output?

If only those 2 files are different, perhaps there's some kind of problem with them... I'm just assuming that they may be added while the zfs send was in progress (or any time after the snapshot was taken).

<Was that the full list from the rsync output?>
No, the files are still scrolling past.

lots and lots and lots of jelly tots flying ;)

G

 

[sretalla]

No, the files are still scrolling past.

lots and lots and lots of jelly tots flying ;)

OK, so have you been doing anything on the server that would cause files to be touched? like some kind of media library scan?

 

[georgelza]

the plex might have scanned the directory, looking for new files... but it would not have touched the files/modified the files in any way.

G

 

[samarium]

Nice assumption, but have you checked the modify dates, the sizes, and maybe the checksums on some of the files? Easy on the destination, and on the current source. Pity you did a modify pass with the destination, a dry run would have been easier to find out what actually happened.
Checking the files in @media will be more difficult, you would have to look in the hidden snapshot directory of the data set, ie poolMountPoint/dataset/.zfs/media should show all the files at time point in time of the snapshot

 

[georgelza]

checked, last touch date is 21 March.

samarium, I know nothing of ZFS, I'm going on advise given me here...

G

 

[samarium]

Where as I've been using zfs send/recv and rsync for years, so I expect it to work like @sretalla.

So did you check insde the .../.zfs/media snapshot like I suggested?

Code:

f="G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4"
snapd=/mnt/tank/media/.zfs/snapshot/media
filed=/mnt/tank/media
ls -l "$filed/$f" "$snapd/$f"
md5sum "$filed/$f" "$snapd/$f"
zfs list -o name,guid,creation tank/media@media bunker/media@media

What does what show?

 

[georgelza]

Remember there is well over a thousand files being listed....

f="G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4"
snapd=/mnt/tank/media/.zfs/snapshot/media
filed=/mnt/tank/media
ls -l "$filed/$f" "$snapd/$f"

-rwxrwxr-x 1 apps apps 1516921455 Dec 14 2022 '/mnt/tank/media/.zfs/snapshot/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4'
-rwxrwxr-x 1 apps apps 1516921455 Dec 14 2022 '/mnt/tank/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4'
md5sum "$filed/$f" "$snapd/$f"
aead4110f1a28270f9e685ab3647a0ab /mnt/tank/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4
aead4110f1a28270f9e685ab3647a0ab /mnt/tank/media/.zfs/snapshot/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4
#

 

[samarium]

Updated slightly please run:

Code:

f="G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4"
snapd=/mnt/tank/media/.zfs/snapshot/media
filed=/mnt/tank/media
bunkd=/mnt/bunker/media
ls -l "$filed/$f" "$snapd/$f" "$bunkd/$f"
md5sum "$filed/$f" "$snapd/$f" "$bunkd/$f"
zfs list -o name,guid,creation tank/media@media bunker/media@media

Also for the rsync you said the files were streaming past, which I expect means fast, which is not what I expect when the files are 1GB+ each, I expect at least a blip of two. What is probably most important with that is the statistics at the end of the rsync saying what the speed up was because if didn't have to copy so much of the data? That is what I expect out of rsync -auv, like this:

Code:

# rsync -auv . /tmp/a/
sending incremental file list
.bash_history

sent 305,431 bytes  received 113 bytes  611,088.00 bytes/sec
total size is 16,077,789  speedup is 52.62

Although I don't see a bunch of files flowing past either, so interesting. I wonder if what you saw was metadata or permissions being updated rather than files being copied.

 

[georgelza]

I have a screen like this, refresh with a new line sometimes once a minute, sometimes every couple of seconds.
My issue, if these lines imply a delta between source and target then WHY.
I created the snapshot, I then send the snapshot as per above commands.
there should be zero delta between source and target.

Create snapshot
/usr/sbin/zfs snapshot -r tank/media@media

Copy snapshot to target
tmux new-session -d -s zfscopy '/usr/sbin/zfs send -R tank/media@media | pv | zfs recv -Fv bunker/media'
--> took 12 hours

Upsert
tmux new-session -d -s rsync 'rsync -auv /mnt/tank/media /mnt/bunker/media'

 

[georgelza]

ls -l "$filed/$f" "$snapd/$f" "$bunkd/$f"

-rwxrwxr-x 1 apps apps 1516921455 Dec 14 2022 '/mnt/bunker/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4'
-rwxrwxr-x 1 apps apps 1516921455 Dec 14 2022 '/mnt/tank/media/.zfs/snapshot/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4'
-rwxrwxr-x 1 apps apps 1516921455 Dec 14 2022 '/mnt/tank/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4'

Only the above 3 lines returned here.

md5sum "$filed/$f" "$snapd/$f" "$bunkd/$f"

aead4110f1a28270f9e685ab3647a0ab /mnt/tank/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4
aead4110f1a28270f9e685ab3647a0ab /mnt/tank/media/.zfs/snapshot/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4
aead4110f1a28270f9e685ab3647a0ab /mnt/bunker/media/G-movies/DaveShaw/Dave Not Coming Back (1080p) - Documentary, Adventure, Heroes.mp4

zfs list -o name,guid,creation tank/media@media bunker/media@media
NAME GUID CREATION
bunker/media@media 14200738407499650151 Wed Aug 9 16:47 2023
tank/media@media 14200738407499650151 Wed Aug 9 16:47 2023

 

[samarium]

Modify times and checksums all the same. Looks identical to me, or course ACLS could be different, and maybe ctime too.

Snapshots are identical, even same guid, so I think the snapshots are identical and zfs snapshots are readonly, can't change.

So i can only suggest than next time you want to do a rsync check that you use rsync -anv not -auv and then you have much better chance of understanding what is different, and also pay attention to the statistics at the end of the rsync run.

Metadata / permissions is the best thought I have for the sync run, I don't think you were copying files. Wonder what @sretalla has to say?

 

[georgelza]

so why is: tmux new-session -d -s rsync 'rsync -auv /mnt/tank/media /mnt/bunker/media'

showing bucket loads of files streaming pass. NOTE: this process is again like Tuesday night filling up the diskpool, I'm north of 10TB atm, where I should not be passing 9TiB

nervous about deleting source media set located on tank.

so you rather suggest using, looks like I'm going to have to redo this again... (for security sake will even shut Plex down this time, as it is the only think access tank/media )

tmux new-session -d -s rsync 'rsync -anv /mnt/tank/media /mnt/bunker/media'

G

 

[samarium]

auv is actually updating the files somehow, I don't know why or what.

anv won't update it just shows what would be done

To see what is actually happening you would need to find the rsync debug options.

I don't know why you are using rsync after an zfs send recv, I would do another snap shot and then do another send recv, but an incremental one.

If auv is actually copying data I don't know why. Maybe u is the problem. I usually use aAhHxXS if I want ACLS.

If rsync is actually copying data then it will be banking up against the snapshot so you will have double data, so maybe that explains the data usage you are seeing. I would kill the rsync.

 

[samarium]

You might want to rollback the bunker/media to the @media snapshot to undo all the extra copying. Although if you do and incremental snapshot replication then the extra will get rolled back anyway.

 

[samarium]

Code:

zfs rollback bunker/media@media

will rollback to just after completion of the snapshot, freeup any extra space used by the rsync.

after that run rsync -anv and see what the stats say at the end, n won't actually do anything so safe to run

might be a good idea to check before you rollback if there is a /mnt/bunker/media/media folder with stuff in it? if so that probably means rsync command specification error.

 

[georgelza]

I've decided to delete the bunker/media dataset.
rebuilding it atm, we can then say what now, how to verify it is all there, and if not how to bring it in sync, but at 12hour for 6Tib, this is adding up, I need to get this completed.

Understand, I don't know zfs so I asked how and was given commands and steps. sure there are other ways...
my issue is I did the zfscopy, I then ran a compare to make sure it's all there and immediately had allot of files listed as not there.
I was then told to do the 2nd rsync command, to determine the delay and only copy required files, which then ended inflating the space used even more until it ran out of space on the diskpool.

G

 

[georgelza]

... say again, this is easy commands for you, unknown for me...

I don't know why you are using rsync after an zfs send recv, I would do another snap shot and then do another send recv, but an incremental one

why would we do another snapshot, the source has not changed ?
we were trying to make sure the target = source which => ended as false.

G

 

[samarium]

the trouble with the compare you are running in that it is not a compare but an sync. compare would use -n.

you say easy for me to say because I use the commands, but I still had to lookup the manual for rsync because rsync has a lot of options, you can do the same.

rsync is also very picky about the path arguments. I nearly always use rsync opts src/ dst/ because I know exactly what what will do and I won't get any extra directories created, which I might with src and/or dst, which is maybe what you did, any maybe what is chewing up your disk.

also you can see the command intermediate outputs and we can't so when normally we might see something anomalous and stop we can only do that if you bring the output to our attention.

lots of output from an rsync where I expected almost nothing would be a good reason to stop for me and investigate further.

 

[georgelza]

trying to understand. how is this applicable : rsync is also very picky about the path arguments. I nearly always use rsync opts src/ dst/ because

I told it (as per commands given to me) to take everything from /tank/media and move it to bunker, thus creating bunker/media

once this was completed, I asked for a command to do a compare, to verify everything was copied, and used the commands given, which resulted in allot of files reported as not in sync... and the wheels came off.

This is still what I want to do... I want everything from tank/media to be in bunker/media so that I can change my SMB and NFS shares to point to bunker/media.

G