Add three hard disk: pool gone

[DVitoD]

Well, this is a great way of discovering a lot of features :D

After discovering some other bugs yesterday (still need to post these, it concerns /dev/ada name switches, GUI hanging needing hard poweroff in the CLI, etc), I now face a most interesting feature :)p):

1. Install with one SSD for FreeNAS itself, and 1 WD Black 2TB drive. 1 pool/volume, called 'tankpool'.
   
2. Ran that for a couple of days, then added 1 Hitachi 1TB and 2 WD Black 2TB's to it for a total now of 4 HDD's and 1 SSD. Did not change the pool, left the disks there for testing only.
   
3. Now I know why these two WD's from .2. were lying isolated in the closet: they were defective. Smart emailed me nicely about this.
4. Today I moved out these two defective ones, and put in 1 WD RE4 500 GB and another WD Black 2TB. These disks were 100% fine, because I took them out of my Synologies where they were as hot spares.
5. Didn't change any cabling: the original disks from .1. stayed at the same cable.
6. Boot. Log in in GUI: alert (also via email):

   The volume tankpool (ZFS) state is UNKNOWN

7. So the two new disks are recognized, 1 of the already existing disks is recognized, but the one and only disk that is relevant, the volume 'tankpool' from step 1, is suddenly gone.
   
8. The GUI shows as is attached.
9. SSH: some CLI commands:
   
10. Code:

     FreeBSD 9.3-RELEASE-p5 (FREENAS.amd64) #2 r275790+f84e770: Tue Dec 23 23:35:33 PST 2014 
    [root@freenas] ~# zpool status
      pool: freenas-boot
    state: ONLINE
      scan: none requested
    config:
    
            NAME        STATE     READ WRITE CKSUM
            freenas-boot  ONLINE       0     0     0
              ada0p2    ONLINE       0     0     0
    
    errors: No known data errors
    [root@freenas] ~# zpool status tankpool
    cannot open 'tankpool': no such pool
    
    [root@freenas] ~# grep 'ada[0-9]' /var/run/dmesg.boot
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    ada0: <M4-CT128M4SSD2 040H> ATA-9 SATA 3.x device
    ada0: Serial Number
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C)
    ada0: quirks=0x1<4K>
    ada0: Previously was known as ad4
    ada1 at ahcich2 bus 0 scbus2 target 0 lun 0
    ada1: <WDC WD5002ABYS-02B1B0 02.03B03> ATA-8 SATA 2.x device
    ada1: Serial Number
    ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada1: Command Queueing enabled
    ada1: 476940MB (976773168 512 byte sectors: 16H 63S/T 16383C)
    ada1: Previously was known as ad8
    ada2 at ahcich3 bus 0 scbus3 target 0 lun 0
    ada2: <WDC WD2002FAEX-007BA0 05.01D05> ATA-8 SATA 3.x device
    ada2: Serial Number
    ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada2: Command Queueing enabled
    ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
    ada2: Previously was known as ad10
    ada3 at ahcich4 bus 0 scbus4 target 0 lun 0
    ada3: <Hitachi HDS721010CLA332 JP4OA3MA> ATA-8 SATA 2.x device
    ada3: Serial Number
    ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
    ada3: Command Queueing enabled
    ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
    ada3: Previously was known as ad12
    [root@freenas] ~# 

I also checked the (very nice) manual for a solution, but it doesn't contain that (yet).

Sysinfo:

1. FreeNAS-9.3-STABLE-201412240734
2. 7618MB RAM

I'm lost at what to do now. Clearly, I can not do status -v if the pool itself doesn't even show up, which makes sense because the drive suddenly isn't detected anymore.

Is there anyway to fix this or should I nuke the box and start all over again?

Thank you in advance for help :D:)

 

[DKarnov]

This will sound obvious, but even though you said you didn't change cabling, you had your hands in the box messing around, so verify data and power cables to the drive.

 

[DVitoD]

This will sound obvious, but even though you said you didn't change cabling, you had your hands in the box messing around, so verify data and power cables to the drive.

Thank you :)

Yes, it sounds obvious and it is obvious. Hence I did it three times: boot - nope didn't work - shutdown - check cables - boot -nope didn't work - shutdown - check cables - boot - nope didn't work - shutdown - check cables.

Then I told my wife: 'this is not possible, can you please check'. So all in all checked 4 times, but nope. Cables are perfectly fine, no pool to be seen :(

 

[Ericloewe]

What happens if you disconnect everything but the SSD and the disk containing the pool?

 

[DVitoD]

What happens if you disconnect everything but the SSD and the disk containing the pool?

Thanks Eric, I will try that and report back

 

[DVitoD]

Thank you for helping, Eric: smart move :D

(I should have thought of that myself, I'm over-tired, sorry :().

Remove three disks = no disks left (where there should have been the 1 disk, the 2TB WD Black, which holds the pool, 'tankpool').

Next: put that disk on another SATA-port. I will do that tomorrow, wife is threathening to feed the dinner to the dogs if I don't come eat it

Thank you again :)

 

[DVitoD]

I'm bad

I obediently sat down to eat dinner. Then wife's sister called. I know how that goes: she walks away to her office (in order not to bother me with the wife-sister-blabla, as she know's I love her extremely dearly --- but not her sister, who clearly is a whining 'what do you think the world will think of my new dress' piece of lalalala).

Next, with wife gone to the office talking to her her sister, I did the 'psss, boys, snack?' to my dogs, and dropped the food in these walking garbage cans, as I wanted to know what is wrong with this test rig (don't worry, I'll steal something to eat out of the frig when wife is sleeping :D).

SATA-port changed: no deal. Pool still doesn't show up. Yet: the drive IS spinning.

Next: I'll test this disk in a different server with a smartctl -t long /dev/

"I'll be back" :p

 

[joeschmuck]

Guess I'm not following completely so let em run over what you have done to date and I'll add some questions into the mix...

You had a system running a SSD as the boot device (ada0) and a 2TB drive (ada2) with a pool called 'tankpool'.
Next you physically installed two additional drives (ada1 and ada3).
Q: How did you add these into your system via the FreeNAS GUI? Be very detailed and specific as this is the part that truly matters and likely the part which was done incorrectly I suspect.

If you added the new drives to the current pool (what I think you said in your original posting step #2), you messed up (unfortunately not an uncommon thing) and that is what I'd like to see if that is what you did because if so, then we can stop troubleshooting the problem and just explain what was incorrect. If you added the two drives as a different pool name then we must continue to troubleshoot.

Cheers,
-Mark (a.k.a. Joe)

 

[DVitoD]

Guess I'm not following completely so let em run over what you have done to date and I'll add some questions into the mix...

You had a system running a SSD as the boot device (ada0) and a 2TB drive (ada2) with a pool called 'tankpool'.
Next you physically installed two additional drives (ada1 and ada3).
Q: How did you add these into your system via the FreeNAS GUI? Be very detailed and specific as this is the part that truly matters and likely the part which was done incorrectly I suspect.

If you added the new drives to the current pool (what I think you said in your original posting step #2), you messed up (unfortunately not an uncommon thing) and that is what I'd like to see if that is what you did because if so, then we can stop troubleshooting the problem and just explain what was incorrect. If you added the two drives as a different pool name then we must continue to troubleshoot.

Cheers,
-Mark (a.k.a. Joe)

Hi Mark :)

No, I didn't add them to the software (= FreeNAS system software): I only mounted them into the rack and connected the cables. What I was routineously wanting to do is: go to disks, and see that everybody was there, including the original 2TB disk that had tankpool on it. Everybody was happily smiling at me, except tankpool, as that disk dropped out. It turned out (next post): it miraculously died

 

[DVitoD]

*** When Murphy meets Darwin... ***
​

(I will go and buy me one of these 'Apple's': according to those who know, 'then everything just works, and no virusses, and cool, and stuff' (:)))

1. Put the disk in my old FreeBSD 6.x machine: 'no, I don't want to know youth, everything was better in the old days'.
2. Put the disk in my pfSense backup server: 'go away, can't see the certificate, won't accept it: blocked and banned'.
3. Put it in my Debian box: 'whut 'ya sayin', this' a BSD-disk? Don't want to talk to him' :)))
4. Put it in my Synology 1813 with 5 disks, looking to see disk 6: happily showing me 5 disks and nothing more.

It is my sad duty to share with you that I assume this disk is dead - even 'though it is spinning.

(To be honest: of my batch of 24 (!) WD Black disks, most of them (22) have died. Whereas all my RE3 and RE4 disks (the RE3's being almost 10 years old I guess) are still going strong. Lesson learned: stay with REx).

So sorry for bothering you with this, but thank you for thinking with me and kicking me sharp again. I will order a new disk and do a full new install. No serious data lost, it was a test box only.

Now: off to sneak some food out of the fridge while wife isn't watching, battle my dogs of as they want that food too, then fire up an episode of Hill Street Blues on my bed room XBMC, and then go to zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz :D

Thank you all :),

Bye,

 

[Ericloewe]

Odd that it died so suddenly...

 

[Z300M]

*** When Murphy meets Darwin... ***
​

(I will go and buy me one of these 'Apple's': according to those who know, 'then everything just works, and no virusses, and cool, and stuff' (:)))

1. Put the disk in my old FreeBSD 6.x machine: 'no, I don't want to know youth, everything was better in the old days'.
2. Put the disk in my pfSense backup server: 'go away, can't see the certificate, won't accept it: blocked and banned'.
3. Put it in my Debian box: 'whut 'ya sayin', this' a BSD-disk? Don't want to talk to him' :)))
4. Put it in my Synology 1813 with 5 disks, looking to see disk 6: happily showing me 5 disks and nothing more.

It is my sad duty to share with you that I assume this disk is dead - even 'though it is spinning.

(To be honest: of my batch of 24 (!) WD Black disks, most of them (22) have died. Whereas all my RE3 and RE4 disks (the RE3's being almost 10 years old I guess) are still going strong. Lesson learned: stay with REx).

Most of my dead drives are WD too, but not in a short space of time. How long have you had your 24 WD Blacks?

 

[joeschmuck]

I'm curious how old those drives were and how many hours were on them but it sounds like you were using very old materials from the start. It is odd that all your drives were dying so maybe the next step would be to install some new drives, even if it's one to test with.

Hill Street Blues, now that is something I haven't heard for decades.

 

[DVitoD]

I'm curious how old those drives were and how many hours were on them but it sounds like you were using very old materials from the start. It is odd that all your drives were dying so maybe the next step would be to install some new drives, even if it's one to test with.

Hill Street Blues, now that is something I haven't heard for decades.

WD Black 2TB aren't that old --- actually, they were brand new :D

And all died within months, with 25% on delivery DOA

The problem is x-fold:

1. Physical stores over here only keep the blue/green/red, because that is what the consumer buys. All the others 'need to be ordered for you so you have to come back'. Nope: not without a discount. You get no stock risk since it is delivery on order, so don't bullshit me with 'manufacturer recommend retail price'.
2. So I need to go to a website to order stuff. Which in 99% of the time don't keep their own inventory. So it comes from a warehouse. There's many different warehouses I noted: some of them don't even package the disks properly, they just throw in a bunch of disks in a box and send that out. Probably throw it into the van also.
   
3. Then you find out the delivery 'service' also likes throwing around stuff.
4. Then you have some cases that can't stand the vibration.
5. Then it might also be the were DOA right in the factory.

Anyway: I've got the replacement disk (properly packed) and currently running the tests. Back to the game of testing FreeNAS.

As to Hill Street Blues: I was extremely happy: it turned out you can buy the whole series retail on Amazon. I did :)

 

[Z300M]

WD Black 2TB aren't that old --- actually, they were brand new :D

And all died within months, with 25% on delivery DOA

The problem is x-fold:

1. Physical stores over here only keep the blue/green/red, because that is what the consumer buys. All the others 'need to be ordered for you so you have to come back'. Nope: not without a discount. You get no stock risk since it is delivery on order, so don't bullshit me with 'manufacturer recommend retail price'.
2. So I need to go to a website to order stuff. Which in 99% of the time don't keep their own inventory. So it comes from a warehouse. There's many different warehouses I noted: some of them don't even package the disks properly, they just throw in a bunch of disks in a box and send that out. Probably throw it into the van also.
   
3. Then you find out the delivery 'service' also likes throwing around stuff.
4. Then you have some cases that can't stand the vibration.
5. Then it might also be the were DOA right in the factory.

Anyway: I've got the replacement disk (properly packed) and currently running the tests. Back to the game of testing FreeNAS.

To my mind, the retailer's mark-up is the reward for keeping things in stock, displaying them and having them immediately available. When items are drop-shipped from the manufacturer or distributor, the "retailer" has done nothing to deserve a mark-up.

Many of the DOA complaints on NewEgg and Amazon a while back were associated with poorly packed hard disks. The last few drives I've purchased from NewEgg were packed well, each in its own "air-pack cocoon"; I'm now waiting for a drive from an Amazon Marketplace seller with 100% feedback rating: I hope it arrives with adequate packaging.

 

[DVitoD]

To my mind, the retailer's mark-up is the reward for keeping things in stock, displaying them and having them immediately available. When items are drop-shipped from the manufacturer or distributor, the "retailer" has done nothing to deserve a mark-up.

How nice of you to recognize this :)

It is actually classic economic theory (my line of work) what you describe. The logics of retailers versus wholesale.

Many of the DOA complaints on NewEgg and Amazon a while back were associated with poorly packed hard disks. The last few drives I've purchased from NewEgg were packed well, each in its own "air-pack cocoon"; I'm now waiting for a drive from an Amazon Marketplace seller with 100% feedback rating: I hope it arrives with adequate packaging.

I also got a load of disks from WD directly some years ago. They were packed perfectly: each disks in it's own case, properly sealed and covered with shock resistant stuff. So what still wonders me: if ALL disks come from WD in the first place, what on earth do they do with them in these warehouses? Deliberately unpack them and throw them, unprotected, in boxes? Is that how you 'save on logistics'?

(I have this assumption many of these brain dead 'managers', who are retarded and are too dumb to understand that they have been tricked into the big college scam to get a grossly inflated, extremely expensive, com-ple-te-ly worthless 'degree', actually think like this ).

 

[Z300M]

How nice of you to recognize this :)

It is actually classic economic theory (my line of work) what you describe. The logics of retailers versus wholesale.

I've never read that anywhere: it just seems logical to me.

I also got a load of disks from WD directly some years ago. They were packed perfectly: each disks in it's own case, properly sealed and covered with shock resistant stuff. So what still wonders me: if ALL disks come from WD in the first place, what on earth do they do with them in these warehouses? Deliberately unpack them and throw them, unprotected, in boxes? Is that how you 'save on logistics'?

(I have this assumption many of these brain dead 'managers', who are retarded and are too dumb to understand that they have been tricked into the big college scam to get a grossly inflated, extremely expensive, com-ple-te-ly worthless 'degree', actually think like this ).

Ten years or more ago when I was doing the rounds of "computer flea-markets" (typically on Saturdays and Sundays in rented space), I would often see cartons of perhaps twenty hard disks from various manufacturers -- each drive in its anti-static bag but with just a single layer of cardboard between each drive. These, I assume, are what are sold as "bare" drives, and it is up to the vendor to package them properly for shipping.

 

[DVitoD]

I've never read that anywhere: it just seems logical to me.

These, I assume, are what are sold as "bare" drives, and it is up to the vendor to package them properly for shipping.

A manufacturer ships 100.000 drives to a wholesaler. He packs them carefully (as WD did when they shipped a large lot to me) because he doesn't want them to be damaged because of RMA's from the wholesaler to the manufacturer. Somewhere, somehow, after that it seems to go wrong (?)

 

[Ericloewe]

The last batches I got on Amazon seemed packed straight from WD. Individual carboard boxes with plastic dampers.

Local computer parts store has them lying around in just their antistatic bags. It's a miracle they've all survived all these years.