Scale 22.12.0 Size on disk is different between 2 VDEV

[UltraHKR]

Hello everyone, here something that got me a little stumped.

I have a ZFS VDEV that I originally setup in dec/21 (V3700-JBOD) [On available TN-Scale 22.02-RC.1-1 or 1-2], and a new ZFS VDEV setup last week (SM836-Chassis).
Both are running on the same VM (TN-Scale v22.12.0), but the "size on disk" are totally different values.
NOTE: I'm using the same model drives and firmware on both VDEV's

V3700-JBOD

• 2x RAID-Z2 (12x 1.2TB 10K RPM SAS drives)
• LSI 9207-4i4e PCIe passthrough to VM

SM836-Chassis

• 1x RAID-Z2 (8x 1.2TB 10K RPM SAS drives)
• Disk Passthrough from Proxmox to VM (LSI 9211-8i is the HBA)
• command in proxmox vm config:
  virtio1: /dev/disk/by-id/scsi-xxx00c500xxx86xxx,aio=threads,backup=0,cache=writethrough,serial=xxxxxxxxxx,size=1172112984K

zfs get all output V3700-JBOD
V3700-JBOD - Pastebin.com

zfs get all output SM836-Chassis
SM836-Chassis - Pastebin.com

If I copy (or rsync) via SSH a file from V3700 to SM836 this happens:
V3700-JBOD (As reported by Win file properties dialog)

• Size: 544 MB (571.252.651 bytes)
• Size on disk: 545 MB (571.718.656 bytes)

SM836-Chassis (As reported by Win file properties dialog)

• Size: 544 MB (571.252.651 bytes)
• Size on disk: 581 MB (609.694.720 bytes)

This is bad because I tried rsync'ing a folder of 2.88TB and it ballooned in size to 3.08TB which is a lot 200+GB...........
NOTE: zdb -c doesnt show SM836-Chassis as a listed 'device', but zpool does........ Something is going on..........

Test #2
So I deleted every file & snapshot on SM836-Chassis. V3700 is unchanged from 64K recordsize, SM836 is changed to 128k.
Copied a new file thru SMB share and this happened form V3700 to SM836:

• V3700 size: 937 MB (982.902.457 bytes)
• V3700 size on disk: 937 MB (982.865.920 bytes)
• SM836 size: 937 MB (982.902.457 bytes)
• SM836 size on disk: 999 MB (1.048.172.032 bytes)

The file grew from 62MB or 6.6%

Test #3
Another example i copied the same file (a rar compressed file) from my PC to each VDEV. V3700 is unchanged from 64K recordsize, SM836 is still at 128k.

• NTFS Size: 158 MB (166.319.308 bytes)
• NTFS Size on disk: 158 MB (166.322.176 bytes)
• V3700 size: 158 MB (166.319.308 bytes)
• V3700 size on disk: 158 MB (166.542.848 bytes)
• SM836 size: 158 MB (166.319.308 bytes)
• SM836 size on disk: 169 MB (177.612.288 bytes)

The file grew from 11MB or 6.6%

Test #4 -- File copied via SMB "drag and drop" from V3700 to SM836, size reported by du -h
V3700 is unchanged from 64K recordsize, SM836 is still at 128k.

Code:

root@truenas[/mnt/SM836-Chassis/ServerFolders]# du -h /mnt/V3700-JBOD/ServerFolders/Test.File1.mp4
724M    /mnt/V3700-JBOD/ServerFolders/Test.File1.mp4
root@truenas[/mnt/SM836-Chassis/ServerFolders]# du -h ./Test.File1.mp4
772M    ./Test.File1.mp4

 

[UltraHKR]

Test #5

Tested on a VM using TN-Core 13.0-U3.1 (trying to keep the same config)
Setup mostly the same 8x32GB HDD's in RAID-Z2 as similar as possible VDEV config as V3700 or SM836.

• NTFS Size: 158 MB (166.319.308 bytes)
• NTFS Size on disk: 158 MB (166.322.176 bytes)
• rz2test size: 158 MB (166.319.308 bytes)
• rz2test size on disk: 169 MB (177.734.656 bytes)

Again the file grew from 11MB or 6.6%

Also this version hangs at shutdown......

 

[UltraHKR]

Test #6

Tested on a VM using TN-Scale 22.02-RC.1-1 (reused TN-Core VM but formatted the disks)
Setup mostly the same 8x32GB HDD's in RAID-Z2 as similar as possible VDEV config as V3700 or SM836.

• NTFS Size: 158 MB (166.319.308 bytes)
• NTFS Size on disk: 158 MB (166.322.176 bytes)
• rz2test size: 158 MB (166.319.308 bytes)
• rz2test size on disk: 169 MB (177.612.288 bytes)

Again the file grew from 11MB or 6.6%

The question is why I'm seeing this or is my FS knowledge so flawed that I'm missing something obvious...