Missing HDD?

[enoch85]

Hi,

I have a kind of strange issue. I recently bought a HP DL380 G8 and an LSI 9300-8i (IT-mode original) to host my FreeNAS storage. I flashed the card to latest FW (P16) and everything went fine. I then moved my old HDDs to the new server and booted on the same USB boot drives as I had in my old server. I tested it and all the drives showed up when booting the new FreeNAS server, including the 6 new drives (6 TB) I added with the purpose of replacing the old ones (4 TB) and grow my pool.

With all the 12 HDDs in place, I booted the server and the LSI card showed me all 12 drives. The old ZFS pool got imported and I started to replace one drive at the time.

So far so good.

As I took for granted that all my HDDs where shown to FreeNAS it wasn't until I came to the 4:th drive I realized that there wasn't any drive available to replace the last two ones with. it was like one just vanished from FreeNAS.

I tried camcontrol devlist, nothing there. I tried camcontrol refresh, nothing changed. I rebooted the FreeNAS server and still no avail.

Am I missing something here? I read somewhere that FreeNAS may need a couple of reboots before it detects all drives. Could that be the case?

One thing I might add is that I started out with one boot USB, and then attached a new one. Maybe that new one got the "daXX" from the last drive?

Current situation

LSI Boot message:

FreeNAS drives:

The new drives are 5 X HGST 6 TB (da7 -10) + 1 X WDC 6 TB (da2), yet I can only see 4 X HGST 6 TB in FreeNAS, where is the last one?

If you are confused by the odd number of drives, I used the WDC 6 TB (da2) to replace one of the old 4 TB drives, did a reboot after that as well.

Any help is appreciated!

 

[dlavigne]

Is this a multipath setup?

 

[Chris Moore]

What model ? Specifically. Some of the new drives are made to turn themselves off if the get 3.3 volt power.

 

[enoch85]

What model ? Specifically. Some of the new drives are made to turn themselves off if the get 3.3 volt power.

The new drives are these: HGST Ultrastar 7K6 HUS726T6TALN6L4

Is this a multipath setup?

Multipath? Sorry, I don't follow.

 

[Chris Moore]

The new drives are these: HGST Ultrastar 7K6 HUS726T6TALN6L4

Legacy mode with regard to the 3.3 volt power question, so that isn't the answer.

 

[enoch85]

Legacy mode with regard to the 3.3 volt power question, so that isn't the answer.

Aah you meant the model of the LSI card?

Afaik it's a LSI 9300-8i bought new this week.

 

[Snow]

Does the pool show in a degraded state and give you a warring? Have you unplug and replug all your SAS cables back in. I would put all the hardware you had back the same way to make sure its not some Hardware/bug. see if it all comes back up.

Did the card come with new cables or did your server come with new cables? When did you replaces your hba? after you loaded your disks from your old pool and old usb or be for? It could be a bad SAS cable Swap them on the HBA but be for you do this mark all the serials down so you can compare. So if the other drive does shows up that cable is probable bad. I had this same thing happen to me well doing a upgrade of my LSI/HBA on my back up system. Also make sure you have your config backed up. What do you see when you type

Code:

zpool status 

do all your disks show?

 

[enoch85]

Does the pool show in a degraded state and give you a warring? Have you unplug and replug all your SAS cables back in. I would put all the hardware you had back the same way to make sure its not some Hardware/bug. see if it all comes back up.

Did the card come with new cables or did your server come with new cables? When did you replaces your hba? after you loaded your disks from your old pool and old usb or be for? It could be a bad SAS cable Swap them on the HBA but be for you do this mark all the serials down so you can compare. So if the other drive does shows up that cable is probable bad. I had this same thing happen to me well doing a upgrade of my LSI/HBA on my back up system. Also make sure you have your config backed up. What do you see when you type

Code:

zpool status 

do all your disks show?

Thanks for your pointers! Appriciated!

Yesterday I did a reboot of the server, noticed that one of the disks wasn't active when POSTing, so I swapped places for a drive known to be working with the faulty one, rebooted again and - success, now all the drives are presented to FreeNAS.

It was probably my fault all the time not checking if all the disks where active during the POST.

Currently I'm resilvering the last drive to be replaced. The order of the drives are all messed up so I'm going to reorder them nice and clean starting from da0 to da11. It shoudn't be an issue right? I mean with ZFS it dosen't matter which physical order the drives are in, correct?

 

[Chris Moore]

Currently I'm resilvering the last drive to be replaced. The order of the drives are all messed up so I'm going to reorder them nice and clean starting from da0 to da11. It shoudn't be an issue right? I mean with ZFS it dosen't matter which physical order the drives are in, correct?

ZFS does not care at all about the 'order' of the drives in fact FreeNAS doesn't even look at the da#. That information is only presented for your amusement. FreeNAS is looking at the gptid, which is why the zpool status looks like this:

Code:

  pool: Emily
state: ONLINE
  scan: scrub repaired 0 in 0 days 05:23:05 with 0 errors on Tue Nov 13 05:23:07 2018
config:

        NAME                                            STATE     READ WRITE CKSUM
        Emily                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/af7c42c6-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b07bc723-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b1893397-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b2bfc678-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b3c1849e-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b4d16ad2-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/bc1e50e5-c1fa-11e8-87f0-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/a03dd690-c1fb-11e8-87f0-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/a6ed2ed5-c240-11e8-87f0-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/b9de3232-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/baf4aba8-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/bbf26621-bf05-11e8-b5f3-0cc47a9cd5a4  ONLINE       0     0     0
        logs
          gptid/ae487c50-bec3-11e8-b1c8-0cc47a9cd5a4    ONLINE       0     0     0
        cache
          gptid/ae52d59d-bec3-11e8-b1c8-0cc47a9cd5a4    ONLINE       0     0     0

errors: No known data errors

Instead of having dev/da# listed in there.

Just so you know, you probably won't be able to get them to appear in what you consider 'proper' order.

 

[Snow]

Cool Happy its all is working I hate when you go to upgrade and it all goes south. Yeah I found that out back when I was so used to windows and HDD's have to be in the same order on the same bus or they would not load. Now I don't even think windows care's if you swap them around. I know ZFS does not.

 

[Chris Moore]

Yeah I found that out back when I was so used to windows and HDD's have to be in the same order on the same bus or they would not load. Now I don't even think windows care's if you swap them around.

Windows is funny. Sometimes it cares and other times it doesn't. I think the code has been modified so many times over the years, without a unified principle, that you can hit a condition and get different, unrelated results, depending on the exact circumstances and you might get a different result the next time if the conditions going in are just a little different. I have a room of 'kiosk' systems that I manage at work and they were all initially loaded from the same image, but on some of them you get different results than on others. I have a less than harmonious relationship with Windows.

 

[Snow]

Windows is a great user Gui but thats where it ends. Even there server ver just links to all the back end stuff they hide in normal windows .