multipath disks with no multipath in the system?

[artlessknave]

had a power outage/brownout (UPS has been acting up so there was no UPS in place). now truenas has decided a bunch of my disks are multipath? there is no multipath in this system, only single disks connected to 1 HBA. it appears its forced standalone disks to be multipath, causing the pools to be degraded?

this one is new to me.

I have backups but WTF? I have never worked with multipath before since....I don't HAVE multiple paths....

I do have SAS disks in these pools.

Show : glabel status

Code:

nas1% zpool status -v
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0B in 00:03:16 with 0 errors on Fri Mar  8 03:48:16 2024
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            ada0p2    ONLINE       0     0     0
            ada1p2    ONLINE       0     0     0

errors: No known data errors

  pool: jls
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q
  scan: scrub repaired 0B in 00:16:17 with 0 errors on Sat Mar  2 00:16:23 2024
config:

        NAME                                            STATE     READ WRITE CKSUM
        jls                                             DEGRADED     0     0     0
          raidz2-0                                      DEGRADED     0     0     0
            8096301518953900130                         UNAVAIL      0     0     0  was /dev/gptid/f8117e4a-01f6-11ec-be8a-0007430ad7be
            gptid/e98ca213-01ea-11ec-be8a-0007430ad7be  ONLINE       0     0     0
            3423086017492985714                         UNAVAIL      0     0     0  was /dev/gptid/b57d2174-894f-11ec-b797-00074328bdc0
            gptid/91cb4880-8962-11ec-b797-00074328bdc0  ONLINE       0     0     0

errors: No known data errors

  pool: prod
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: resilvered 198M in 00:05:26 with 0 errors on Sun Mar 10 12:28:36 2024
config:

        NAME                                            STATE     READ WRITE CKSUM
        prod                                            DEGRADED     0     0     0
          raidz3-0                                      DEGRADED     0     0     0
            gptid/f9e16b72-a30e-11ea-bb3f-0007430ad7be  ONLINE       0     0     0
            gptid/0fdd85d9-843c-11ee-8405-00074328bdc0  ONLINE       0     0     2
            gptid/58467e37-f2d3-11ed-865b-00074328bdc0  ONLINE       0     0     0
            gptid/879e906c-a237-11ea-b3eb-0007430ad7be  ONLINE       0     0     0
            13532310063243849484                        UNAVAIL      0     0     0  was /dev/multipath/disk3
            gptid/08b44e3b-a1e6-11ea-b3eb-0007430ad7be  ONLINE       0     0     0
            gptid/64d60986-8515-11ee-8405-00074328bdc0  ONLINE       0     0     0
            6137983326915288124                         UNAVAIL      0     0     0  was /dev/multipath/disk1
            gptid/7fc4bf77-b24d-11ea-9283-0007430a55ce  ONLINE       0     0     0
            gptid/ba528ef1-b630-11ea-9283-0007430a55ce  ONLINE       0     0     0
            gptid/42f022fc-b5ce-11ea-9283-0007430a55ce  ONLINE       0     0     0

errors: No known data errors

Show : glabel status

Code:

nas1% glabel status
                                      Name  Status  Components
gptid/59ed825e-8a30-11ec-a94b-00074328bdc0     N/A  da0p2
gptid/5a1cb2d6-8a30-11ec-a94b-00074328bdc0     N/A  da2p2
gptid/91cb4880-8962-11ec-b797-00074328bdc0     N/A  da6p2
gptid/e98ca213-01ea-11ec-be8a-0007430ad7be     N/A  da4p2
gptid/de1590fd-0f5f-11e9-bad5-0cc47ae2abe8     N/A  ada0p1
gptid/15eddd19-a382-11ea-8e6d-0007430ad7be     N/A  da14p2
gptid/42f022fc-b5ce-11ea-9283-0007430a55ce     N/A  da25p2
gptid/08b44e3b-a1e6-11ea-b3eb-0007430ad7be     N/A  da17p2
gptid/879e906c-a237-11ea-b3eb-0007430ad7be     N/A  da24p2
gptid/f9e16b72-a30e-11ea-bb3f-0007430ad7be     N/A  da26p2
gptid/60dd5728-0f62-11e9-bad5-0cc47ae2abe8     N/A  ada1p1
gptid/7fc4bf77-b24d-11ea-9283-0007430a55ce     N/A  da5p2
gptid/5a1e432a-8a30-11ec-a94b-00074328bdc0     N/A  da1p2
gptid/5a060a8e-8a30-11ec-a94b-00074328bdc0     N/A  da3p2
gptid/58467e37-f2d3-11ed-865b-00074328bdc0     N/A  da22p2
gptid/0fdd85d9-843c-11ee-8405-00074328bdc0     N/A  da12p2
gptid/64d60986-8515-11ee-8405-00074328bdc0     N/A  da20p2
gptid/ba528ef1-b630-11ea-9283-0007430a55ce     N/A  da18p2
gptid/7fb5e5ce-b24d-11ea-9283-0007430a55ce     N/A  da5p1
gptid/15c16431-a382-11ea-8e6d-0007430ad7be     N/A  da14p1
gptid/e9754772-01ea-11ec-be8a-0007430ad7be     N/A  da11p1
gptid/91b64ec8-8962-11ec-b797-00074328bdc0     N/A  da15p1
gptid/59eb4567-8a30-11ec-a94b-00074328bdc0     N/A  da2p1
gptid/59d210e7-8a30-11ec-a94b-00074328bdc0     N/A  da0p1

Show : disklist.pl

Code:

partition  label                                       zpool         device  disk                    size  type  serial                  rpm  sas-location     multipath        multipath-mode  multipath-state                               
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------                               
da14p2     gptid/15eddd19-a382-11ea-8e6d-0007430ad7be                da14    ATA WDC WD40EFRX-68N    4000  HDD   WDWCC7K6YERNP3         5400  SAS3224(0):2#3                                                                                  
ada0p2     gptid/de21cf37-0f5f-11e9-bad5-0cc47ae2abe8  freenas-boot  ada0    KINGSTON SA400S37120G    120  SSD   50026B768263D1C2          0                                                                                                  
ada1p2     gptid/60e9894c-0f62-11e9-bad5-0cc47ae2abe8  freenas-boot  ada1    KINGSTON SV300S37A120G   120  SSD   50026B775408DFC9          0                                                                                                  
da11p2     gptid/e98ca213-01ea-11ec-be8a-0007430ad7be  jls           da11    SEAGATE ST300MM0006      300  HDD   S0K3K2BA              10000  SAS3224(0):3#0                                                                                  
da15p2     gptid/91cb4880-8962-11ec-b797-00074328bdc0  jls           da15    SEAGATE ST300MM0006      300  HDD   S0K1MB3Q              10000  SAS3224(0):3#4                                                                                  
da12p2     gptid/0fdd85d9-843c-11ee-8405-00074328bdc0  prod          da12    ATA ST8000NE001-2M71    8001  HDD   WSD9W0RJ               7200  SAS3224(0):2#1                                                                                  
da13p2     gptid/7fc4bf77-b24d-11ea-9283-0007430a55ce  prod          da13    SEAGATE ST6000NM0105    6001  HDD   ZAD4HZVB0000C840ED6D   7200  SAS3224(0):3#2                                                                                  
da17p2     gptid/08b44e3b-a1e6-11ea-b3eb-0007430ad7be  prod          da17    ATA WDC WD40EFRX-68N    4000  HDD   WDWCC7K5HF3UTT         5400  SAS3224(0):2#6                                                                                  
da18p2     gptid/ba528ef1-b630-11ea-9283-0007430a55ce  prod          da18    ATA ST8000VN0022-2EL    8001  HDD   ZA1HFA8M               7200  SAS3224(0):2#7                                                                                  
da20p2     gptid/64d60986-8515-11ee-8405-00074328bdc0  prod          da20    ATA ST8000NE001-2M71    8001  HDD   WSD9W0SF               7200  SAS3224(0):2#9                                                                                  
da22p2     gptid/58467e37-f2d3-11ed-865b-00074328bdc0  prod          da22    ATA ST8000VN004-3CP1    8001  HDD   WWZ03KV6               7200  SAS3224(0):2#11                                                                                 
da24p2     gptid/879e906c-a237-11ea-b3eb-0007430ad7be  prod          da24    ATA WDC WD40EFRX-68N    4000  HDD   WDWCC7K1URKSDU         5400  SAS3224(0):2#13                                                                                 
da25p2     gptid/42f022fc-b5ce-11ea-9283-0007430a55ce  prod          da25    ATA WDC WD80EFAX-68K    8001  HDD   VAGVRU0L               5400  SAS3224(0):2#14                                                                                 
da26p2     gptid/f9e16b72-a30e-11ea-bb3f-0007430ad7be  prod          da26    ATA WDC WD40EFRX-68N    4000  HDD   WDWCC7K6JX44FA         5400  SAS3224(0):2#15                                                                                 
da0p2      gptid/59ed825e-8a30-11ec-a94b-00074328bdc0  ztgl          da0     ATA KINGSTON SA400S3     960  SSD   50026B778445F7DF          0  SAS3224(0):1#12                                                                                 
da1p2      gptid/5a1e432a-8a30-11ec-a94b-00074328bdc0  ztgl          da1     ATA KINGSTON SA400S3     960  SSD   50026B778451215C          0  SAS3224(0):1#13                                                                                 
da2p2      gptid/5a1cb2d6-8a30-11ec-a94b-00074328bdc0  ztgl          da2     ATA KINGSTON SA400S3     960  SSD   50026B778445F768          0  SAS3224(0):1#14                                                                                 
da3p2      gptid/5a060a8e-8a30-11ec-a94b-00074328bdc0  ztgl          da3     ATA KINGSTON SA400S3     960  SSD   50026B7784512153          0  SAS3224(0):1#15                                                                                 
                                                                     da16    SEAGATE ST6000NM0105    6001  HDD   ZAD4RX4X0000C84322A3   7200  SAS3224(0):3#5   multipath/disk1  ACTIVE          OPTIMAL                                       
                                                                     da7     SEAGATE ST6000NM0105    6001  HDD   ZAD4RX4X0000C84322A3   7200  SAS3224(0):3#5   multipath/disk1  PASSIVE         OPTIMAL                                       
                                                                     da19    SEAGATE ST300MM0006      300  HDD   S0K440QL              10000  SAS3224(0):3#8   multipath/disk2  ACTIVE          OPTIMAL                                       
                                                                     da8     SEAGATE ST300MM0006      300  HDD   S0K440QL              10000  SAS3224(0):3#8   multipath/disk2  PASSIVE         OPTIMAL                                       
                                                                     da21    SEAGATE ST6000NM0105    6001  HDD   ZAD4FFY80000C840EF12   7200  SAS3224(0):3#10  multipath/disk3  ACTIVE          OPTIMAL                                       
                                                                     da9     SEAGATE ST6000NM0105    6001  HDD   ZAD4FFY80000C840EF12   7200  SAS3224(0):3#10  multipath/disk3  PASSIVE         OPTIMAL                                       
                                                                     da23    SEAGATE ST300MM0006      300  HDD   S0K4TBW80000K601NSY9  10500  SAS3224(0):3#12  multipath/disk4  ACTIVE          OPTIMAL                                       
                                                                     da10    SEAGATE ST300MM0006      300  HDD   S0K4TBW80000K601NSY9  10500  SAS3224(0):3#12  multipath/disk4  PASSIVE         OPTIMAL        

Show : gmultipath list

Code:

                                                                                                                                                                                                                    
Geom name: disk3                                                                                                                                                                                                                               
Type: AUTOMATIC                                                                                                                                                                                                                                
Mode: Active/Passive                                                                                                                                                                                                                           
UUID: d5a0c552-deb4-11ee-a56d-00074328bdc0                                                                                                                                                                                                     
State: OPTIMAL                                                                                                                                                                                                                                 
Providers:                                                                                                                                                                                                                                     
1. Name: multipath/disk3                                                                                                                                                                                                                       
   Mediasize: 6001175121920 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r0w0e0                                                                                                                                                                                                                                
   State: OPTIMAL                                                                                                                                                                                                                              
Consumers:                                                                                                                                                                                                                                     
1. Name: da21                                                                                                                                                                                                                                  
   Mediasize: 6001175126016 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: ACTIVE                                                                                                                                                                                                                               
2. Name: da9                                                                                                                                                                                                                                   
   Mediasize: 6001175126016 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: PASSIVE                                                                                                                                                                                                                              
                                                                                                                                                                                                                                               
Geom name: disk1                                                                                                                                                                                                                               
Type: AUTOMATIC                                                                                                                                                                                                                                
Mode: Active/Passive                                                                                                                                                                                                                           
UUID: d58379c2-deb4-11ee-a56d-00074328bdc0                                                                                                                                                                                                     
State: OPTIMAL                                                                                                                                                                                                                                 
Providers:                                                                                                                                                                                                                                     
1. Name: multipath/disk1                                                                                                                                                                                                                       
   Mediasize: 6001175121920 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r0w0e0                                                                                                                                                                                                                                
   State: OPTIMAL                                                                                                                                                                                                                              
Consumers:                                                                                                                                                                                                                                     
1. Name: da16                                                                                                                                                                                                                                  
   Mediasize: 6001175126016 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: ACTIVE                                                                                                                                                                                                                               
2. Name: da7                                                                                                                                                                                                                                   
   Mediasize: 6001175126016 (5.5T)                                                                                                                                                                                                             
   Sectorsize: 4096                                                                                                                                                                                                                            
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: PASSIVE                                                                                                                                                                                                                              
                                                                                                                                                                                                                                               
Geom name: disk4                                                                                                                                                                                                                               
Type: AUTOMATIC                                                                                                                                                                                                                                
Mode: Active/Passive                                                                                                                                                                                                                           
UUID: d5b210da-deb4-11ee-a56d-00074328bdc0                                                                                                                                                                                                     
State: OPTIMAL                                                                                                                                                                                                                                 
Providers:                                                                                                                                                                                                                                     
1. Name: multipath/disk4                                                                                                                                                                                                                       
   Mediasize: 299999999488 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r0w0e0                                                                                                                                                                                                                                
   State: OPTIMAL                                                                                                                                                                                                                              
Consumers:                                                                                                                                                                                                                                     
1. Name: da23                                                                                                                                                                                                                                  
   Mediasize: 300000000000 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: ACTIVE                                                                                                                                                                                                                               
2. Name: da10                                                                                                                                                                                                                                  
   Mediasize: 300000000000 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: PASSIVE                                                                                                                                                                                                                              
                                                                                                                                                                                                                                               
Geom name: disk2                                                                                                                                                                                                                               
Type: AUTOMATIC                                                                                                                                                                                                                                
Mode: Active/Passive                                                                                                                                                                                                                           
UUID: d59106cf-deb4-11ee-a56d-00074328bdc0                                                                                                                                                                                                     
State: OPTIMAL                                                                                                                                                                                                                                 
Providers:                                                                                                                                                                                                                                     
1. Name: multipath/disk2                                                                                                                                                                                                                       
   Mediasize: 299999999488 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r0w0e0                                                                                                                                                                                                                                
   State: OPTIMAL                                                                                                                                                                                                                              
Consumers:                                                                                                                                                                                                                                     
1. Name: da19                                                                                                                                                                                                                                  
   Mediasize: 300000000000 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: ACTIVE                                                                                                                                                                                                                               
2. Name: da8                                                                                                                                                                                                                                   
   Mediasize: 300000000000 (279G)                                                                                                                                                                                                              
   Sectorsize: 512                                                                                                                                                                                                                             
   Mode: r1w1e1                                                                                                                                                                                                                                
   State: PASSIVE                                                                                                                                                                                                                              
                            

 

[artlessknave]

well, I was able to use gmultipath destroy disk1/2/3/4 to destroy the invalid multipaths, and my pool appears to have healed.

there was something causing a ton of timeout console messages for awhile after that but that also seems to have stopped.

also used gpart recover to hopefully repair the messed up gpt secondary table...im assuming that got messed up when it decided to make them invalid multipath configs.

very weird. I'm still not sure what caused it, i wouldn't have expected power instability to cause THAT.

 

[joeschmuck]

That is one strange problem that should never happen. It is such an odd problem, and on four drives. Is it at all possible that you did something to the NAS since the previous reboot? I'm curious if you made any changes to the configuration that might in-directly have caused this. Four drives having this issue is so very odd which is why I ask the question. I don't see how it could be a hardware related issue.

Glad you fixed though.

 

[artlessknave]

nope, I haven't changed anything in weeks. that is the state it was in when I looked at it after the power blipped.

 

[rvassar]

Any chance your backplane has a SAS expander built into it, or has a "split-mode" where half the drives can be connected to another controller? All that would be required is to have the SAS addresses of the devices appear on multiple "SAS cores" inside the HBA at the same time. This could be caused by various bits and pieces asserting power-fail or power-good at slightly different times during your power blip. It shouldn't happen, but... A/C cycle testing in industry usually does cycles spanning minutes, not fractions of a second where various internal power rails may not assert power-fail.

 

[artlessknave]

hmm. afaik it's a single path sas expander backplane, and its definitely connected to a single HBA. this has been runnning this way for about a month now. (moved it over from a 2u+2u jbod setup that i hated). I would have to open it up to check the model number of the backplane, but I would think the "split mode" backplanes would be limited to multiple node chassis, which this is definitely not.

the multi paths were incorrectly joining distinct hard drives into multi path connections, which cleared out when i nuked the multipath config that should never have existed to begin with.

 

[rvassar]

A lot of BP's now days carry extra functionality that you may not see exposed in your configuration. Split-mode and multiple controllers gets used in performance configs with SAS SSD's, etc... The bits and pieces of the feature may be present even if your chassis doesn't support or need it. This permits the ODM to re-use the same parts for other models. You could even have a BP that has PCIe lanes for NVMe drives that never get used in your config because there's no PCIe cable installed, etc... But the power glitch may have caused the BP to "boot" and transition from one topology to another as it resumed operation. If the HBA was up and running it might have seen multiple paths to the SAS address and mapped a multi-path topology.

Suggestion: Fix the UPS.

 

[artlessknave]

the chassis had an x8 board in it. it's from before SSDs were really a thing....and DEFINITELY before nvme was a thing. or bifurcation. hell, i was surprised x8 even know what uefi is. afaik it's just a regular "el1" expander backplane, but I will be trying to find the model number the next I have it open. the chassis itself doesn't display a model number anywhere I can find. it was a netscout appliance.

Suggestion: Fix the UPS.

yes, yes, I know. i put it back in in a dual power config for now. the problem with the thing is it turns off and just stays off and i cant figure out why. I *think* the batteries might need replacing but they show no signs of that that I can tell; the damn thing just wont turn on when power is restored.