Existing/Transferred disks not recognized as multipath-disks in SAS-enclosure

wolfman

Dabbler
Joined
Apr 11, 2018
Messages
13
I migrated one of my TrueNAS systems (12.0-U2) this week from a Sumpermicro Server to a Dell 730xd with a Xyratex HB-1235 SAS Enclosure. I tested the Dell/Xyratex-System beforehand with some empty disks (to add as an additional vdev to the existing pool afterwards). Everything went fine and all disks in the Xyratex-Enclosure were recognized as multipath-disks.

So far so good. When i transferred the already used disks from the old system to the new system, TrueNAS booted up and mounted the pool just fine. Accessing the pool is not an issue at all - NFS/iSCSI access works as exptected. But the following issue arose: the used disks transferred from the Supermicro Server to the Xyratex-Enclosure are not recognized as multipath-disks - contrary to new/empty disks (sgdisk --zap-all) added to the enclosure!

The real issue here however is, that apparently the second path of the used disks are showing up as unused disks in the disk-list! And i am allowed by the TrueNAS frontend to create a new pool/vdev with these (second path) disks, which undoubtedly would result in data-loss! :eek:

Is there something to do to let TrueNAS recognize those disks as multipath disks? Disclaimer: This is the first time for me, working with multipath at all. I didn't even expect the disks to pop-up as multipath diks prior to testing the new system with empty disks!
What might work ist pull one disk after another from the enclosure; wipe it; re-add it to the enclosure and let it recognize as multipath-disk; re-add it to the vdev and wait for the re-silvering. o_O

All discs concerned are of the 4 TB SATA Western Digital WD4000FYYZ / WD4002FYYZ / WD4003FRYZ variety with LSI SAS-SATA Interposer boards in the external enclosure.
The data pool consists of 15 4 TB drives (3 vdevs with 5 discs in raid-z2). There is an additonal NVMe L2ARC and 2 Micron SSDs for ZIL, all housed in the Dell 730xd directly. So no issue with those three.

Specs Supermicro
  • Supermicro RI1316+ case with 16 3,5" HDD Slots
  • intel Xeon E5-2609v2
  • 48GB ECC RAM
  • LSI HBA (IT-Mode)
Specs Dell & Xyratex
  • 730xd case with 12 3,5" HDD Slots
  • 2x intel Xeon E5-2630v3
  • 128GB ECC RAM
  • Dell Perc HBA330 IT-Mode for internal disks
  • LSI SAS9200-8e for SAS Enclosure
  • Xyratex HB-1235 SAS Enclosure with 12 3,5" HDD Slots
    • 2x SAS Controller
    • attached with TWO SFF-8088 cables
    • all disks in the enclosure are equipped with an LSI SAS-SATA Interposer board
Any help is highly appreciated.
 

wolfman

Dabbler
Joined
Apr 11, 2018
Messages
13
So i am currently going down the path of pulling each disk that is not recognized as a multipath-disk and re-adding them. But resilvering the 4 TB disks takes extremely long! I am currently at ~98% of resilvering one 4 TB disk and the process is already going on for about 52 hours!
  • The first 20% to 25% of resilvering wen't extremely fast; It maybe took 30 minutes to reach that percentage.
  • Then gstat -p showed around 4 MBps writes on the disk for the next two days.
  • In the last 10% to 15% the writes finally picked up with peaks of around 50 MBps, but also with longer periods of arund 3 to 4 MBps.
Is this normal? I mean, it's only a 4 TB disk! Is the fragmentation of the pool an issue? The system is under load, but far from anything i would consider even close to "heavy"!

Code:
last pid: 71026;  load averages:  0.21,  0.24,  0.24                                                                          up 7+21:14:37  14:50:23
2016 threads:  33 running, 1921 sleeping, 62 waiting
CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
Mem: 172M Active, 2845M Inact, 685M Laundry, 112G Wired, 8926M Free
ARC: 105G Total, 80G MFU, 22G MRU, 78M Anon, 1597M Header, 58M Other
     98G Compressed, 197G Uncompressed, 2.00:1 Ratio
Swap: 10G Total, 10G Free


Code:
root@nas-02:~ # zpool status ggmtank01
  pool: ggmtank01
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Feb 27 10:40:28 2021
        17.9T scanned at 101M/s, 17.6T issued at 99.0M/s, 18.0T total
        1.06T resilvered, 97.79% done, 01:09:59 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        ggmtank01                                       ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/071d138c-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/07d35682-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/ef627048-743e-11eb-8d93-e4434bb19fe0  ONLINE       0     0     0
            gptid/09419f0f-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/11a18e03-12b8-11eb-b67f-000743400660  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/73340a80-5449-11e9-b326-000743400660  ONLINE       0     0     0
            gptid/39c37fc2-d9d5-11e9-b449-000743400660  ONLINE       0     0     0
            gptid/dc3acdac-60b8-11eb-9ad3-000743400660  ONLINE       0     0     0
            gptid/bcd813cc-4144-11eb-9ad3-000743400660  ONLINE       0     0     0
            gptid/3ffd3cdc-7440-11eb-8d93-e4434bb19fe0  ONLINE       0     0     0
          raidz2-2                                      ONLINE       0     0     0
            gptid/c071d681-743c-11eb-8d93-e4434bb19fe0  ONLINE       0     0     0
            gptid/0f702ce5-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/d1bdee26-78df-11eb-bdc3-e4434bb19fe0  ONLINE       0     0     0  (resilvering)
            gptid/10fc2434-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/11b390ef-9644-11e8-8380-000743400660  ONLINE       0     0     0
        logs
          mirror-3                                      ONLINE       0     0     0
            gptid/123e1981-9644-11e8-8380-000743400660  ONLINE       0     0     0
            gptid/12b0bdb1-9644-11e8-8380-000743400660  ONLINE       0     0     0
        cache
          gptid/f4918c31-ff0f-11e9-b449-000743400660    ONLINE       0     0     0

errors: No known data errors


Code:
root@nas-02:~ # zpool list -v ggmtank01
NAME                                             SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
ggmtank01                                       54.4T  18.0T  36.4T        -         -    34%    33%  1.00x    ONLINE  /mnt
  raidz2                                        18.1T  6.09T  12.0T        -         -    34%  33.6%      -  ONLINE
    gptid/071d138c-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/07d35682-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/ef627048-743e-11eb-8d93-e4434bb19fe0      -      -      -        -         -      -      -      -  ONLINE
    gptid/09419f0f-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/11a18e03-12b8-11eb-b67f-000743400660      -      -      -        -         -      -      -      -  ONLINE
  raidz2                                        18.1T  6.08T  12.0T        -         -    34%  33.5%      -  ONLINE
    gptid/73340a80-5449-11e9-b326-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/39c37fc2-d9d5-11e9-b449-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/dc3acdac-60b8-11eb-9ad3-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/bcd813cc-4144-11eb-9ad3-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/3ffd3cdc-7440-11eb-8d93-e4434bb19fe0      -      -      -        -         -      -      -      -  ONLINE
  raidz2                                        18.1T  5.84T  12.3T        -         -    34%  32.2%      -  ONLINE
    gptid/c071d681-743c-11eb-8d93-e4434bb19fe0      -      -      -        -         -      -      -      -  ONLINE
    gptid/0f702ce5-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/d1bdee26-78df-11eb-bdc3-e4434bb19fe0      -      -      -        -         -      -      -      -  ONLINE
    gptid/10fc2434-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/11b390ef-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
logs                                                -      -      -        -         -      -      -      -  -
  mirror                                         220G  3.72M   220G        -         -     0%  0.00%      -  ONLINE
    gptid/123e1981-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
    gptid/12b0bdb1-9644-11e8-8380-000743400660      -      -      -        -         -      -      -      -  ONLINE
cache                                               -      -      -        -         -      -      -      -  -
  gptid/f4918c31-ff0f-11e9-b449-000743400660     466G   384G  81.5G        -         -     0%  82.5%      -  ONLINE


I found some older entries mentioning to change the following Tunabeles (from it's default values 3000 and 2) to improve resilver speed. But i am not sure if changing these values will do anything - at least they had no effect while the resilvering-process was already running. And sysctl -a doesn't even show vfs.zfs.resilver_delay on TrueNAS-12.0-U2 anymore.

Code:
vfs.zfs.resilver_min_time_ms=5000
vfs.zfs.resilver_delay=0


Edit:
Code:
resilvered 1.11T in 2 days 05:10:21 with 0 errors on Mon Mar  1 15:50:49 2021
 
Last edited:

LarchOye

Cadet
Joined
Nov 19, 2019
Messages
2
The issue probably has something to do with the fact that you are using SATA disks. SATA dors not have multi-path, only SAS disks have the dual-controller capability.
 
Top