Upgraded drives and 4K performance took a dump?

[eroji]

I have 12 5TB Toshiba 7200RPM drives in stripe+mirror and iSCSI to ESXi via X520-DA2s. I just upgraded 4 of the drives to the 5TB from 4TB HGSTs. Right after I completed the upgrade and rebooted FreeNAS, and ran a quick benchmark test. To my surprise, the 4K and 4KQD32 read/write performance took a steep drop. 4KQD32 performance dropped all the way down to ~10MB/s for both read and write. Am I missing something? Even if it's ZFS load balancing to the new, larger drives compared to the originals, it still seems way low.

 

[m0nkey_]

I'm just guessing here, but it sounds like you might have a disk sector misalignment. You'd need to provide more information about your hardware in order to get a proper response.

Check the ashift setting on your pool

Code:

# zdb -C yourpoolname | grep ashift
ashift=9

That's 512 (2^9). ashift=12 is 4096 (2^12).

Reference: https://forums.freenas.org/index.php?threads/zfs-and-4k.9936/#post-43664

 

[eroji]

Yes, looks like that is indeed the problem. Not entirely sure how it became like this since the pool now has all identical drives.

Code:

[root@freenas] ~# zdb | grep ashift
            ashift: 12
            ashift: 12
            ashift: 9
            ashift: 12
            ashift: 12
            ashift: 12
            ashift: 12

Drive info from dmesg.boot

Code:

    da0 at mps0 bus 0 scbus0 target 8 lun 0
    da0: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da0: Serial Number         74D3K4ICFS9A
    da0: 600.000MB/s transfers
    da0: Command Queueing enabled
    da0: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da1 at mps0 bus 0 scbus0 target 9 lun 0
    da1: <ATA TOSHIBA MD04ACA5 FP1A> Fixed Direct Access SCSI-6 device
    da1: Serial Number         54O2K1XQFS9A
    da1: 600.000MB/s transfers
    da1: Command Queueing enabled
    da1: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da2 at mps0 bus 0 scbus0 target 11 lun 0
    da2: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da2: Serial Number         84P1K4MVFS9A
    da2: 600.000MB/s transfers
    da2: Command Queueing enabled
    da2: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da3 at mps0 bus 0 scbus0 target 12 lun 0
    da3: <ATA TOSHIBA MD04ACA5 FP1A> Fixed Direct Access SCSI-6 device
    da3: Serial Number         649FK2WCFS9A
    da3: 600.000MB/s transfers
    da3: Command Queueing enabled
    da3: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da4 at mps0 bus 0 scbus0 target 14 lun 0
    da4: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da4: Serial Number         74D3K4IGFS9A
    da4: 600.000MB/s transfers
    da4: Command Queueing enabled
    da4: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da5 at mps0 bus 0 scbus0 target 17 lun 0
    da5: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da5: Serial Number         84P1K4MOFS9A
    da5: 600.000MB/s transfers
    da5: Command Queueing enabled
    da5: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da6 at mps0 bus 0 scbus0 target 21 lun 0
    da6: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da6: Serial Number         955AK57IFS9A
    da6: 600.000MB/s transfers
    da6: Command Queueing enabled
    da6: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da11 at mps0 bus 0 scbus0 target 26 lun 0
    da11: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da11: Serial Number         Y5N9K731FS9A
    da11: 600.000MB/s transfers
    da11: Command Queueing enabled
    da11: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da7 at mps0 bus 0 scbus0 target 22 lun 0
    da7: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da7: Serial Number         95GAK5LRFS9A
    da7: 600.000MB/s transfers
    da7: Command Queueing enabled
    da7: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da8 at mps0 bus 0 scbus0 target 23 lun 0
    da8: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da8: Serial Number         Y5NAK6R9FS9A
    da8: 600.000MB/s transfers
    da8: Command Queueing enabled
    da8: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da9 at mps0 bus 0 scbus0 target 24 lun 0
    da9: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da9: Serial Number         Y5REKBRZFS9A
    da9: 600.000MB/s transfers
    da9: Command Queueing enabled
    da9: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)
    da10 at mps0 bus 0 scbus0 target 25 lun 0
    da10: <ATA TOSHIBA MD04ACA5 FP2A> Fixed Direct Access SCSI-6 device
    da10: Serial Number         Y5RDKB6JFS9A
    da10: 600.000MB/s transfers
    da10: Command Queueing enabled
    da10: 4769307MB (9767541168 512 byte sectors: 255H 63S/T 608001C)