best practices for storage switch-up

[cvejic]

I built and configured a FreeNAS box a few years ago. Wanting to future-proof the NAS a bit, I used a Xeon SoC board with 128GB ECC RAM and (4) 6TB WD Gold drives, all packed into a Silverstone NAS case. Time went by and space started running low so 4 more 6TB drives were added along with an hba card in the only PCIe slot for the last 4 drives. About 8 months ago I started experiencing my first issues. A drive from the ZFS RAID 1+0 container went down. Since they are supposed to have a million hour MTF I thought it was unusual but passed it off as bad luck. Then another drive died. And another. And another. Thus far I have managed to replace broken drives before experiencing any critical failures, but after a close call I purchased a bunch of cold storage drives and backed up my files as a "just in case" move. I've been looking for a solution to both my storage space and stability issues.

Basically, I think the drive problem is related to heat issues. Although the case does have 8 hot-swap bays, I don't think the small fans in the case are powerful enough to provide good air flow. Cleaning the filters regularly helps, but even doing that I get the occasional heat warning from (and only from) the drives. To resolve both issues I purchased an external JBOD enclosure that has 12 hot-swap bays with 4 large fans that provide good air flow. The enclosure uses 3 SFF-8088 mini-SAS (at 24Gb/s per connection) interfaces. Problem is, the only slot is hosting that internal only hba controller card. I found an LSI00276 SAS 9201-16e hba card that has 4 SFF-8088 connectors to replace my old hba card, but I'm not certain of what the recommended procedure would be:

1. Replace the old hba with the new one, consider the old ZFS container lost, move all the drives into the enclosure, create a new ZFS container, and restore my files from cold storage. This is my fallback plan, but I don't love it because of how long it could take and the potential for little issues to crop up.

2. Replace the current hba card with the new one and move all of the drives to the external enclosure, then connect the enclosure to the new card. Hope and pray that the old drives and ZFS container will be auto-magically recognized through the new adapter.

3. Something better than the first two options. Maybe some internal SATA to SFF-8088 conversion cable (if such a thing exists) and 2 new 12TB drives in the enclosure that I could create a new ZFS container on and copy the contents of the old container over to?

If anyone has a suggestion I would be grateful for the advice. Thanks!

 

[Chris Moore]

A drive from the ZFS RAID 1+0 container went down. Since they are supposed to have a million hour MTF I thought it was unusual but passed it off as bad luck. Then another drive died. And another. And another. Thus far I have managed to replace broken drives before experiencing any critical failures, but after a close call I purchased a bunch of cold storage drives and backed up my files as a "just in case" move. I've been looking for a solution to both my storage space and stability issues.

Inside code tags like this: [code] your text [/code] would you share the output of zpool status please?
It should look something like this:

Code:

  pool: Irene
 state: ONLINE
  scan: scrub repaired 0 in 0 days 03:20:25 with 0 errors on Wed Feb 20 03:20:26 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Irene                                           ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/8710385b-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/87e94156-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/88db19ad-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/89addd3b-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8a865453-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8b66b1ef-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/8c69bc72-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8d48655d-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8e2b6d1f-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8efea929-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/8fd4d25c-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0
            gptid/90c2759a-becf-11e8-b1c8-0cc47a9cd5a4  ONLINE       0     0     0

errors: No known data errors

Basically, I think the drive problem is related to heat issues.

Very likely, that case is known to have poor airflow and those Gold drives get quite hot.

The enclosure uses 3 SFF-8088 mini-SAS (at 24Gb/s per connection) interfaces. Problem is, the only slot is hosting that internal only hba controller card. I found an LSI00276 SAS 9201-16e hba card that has 4 SFF-8088 connectors to replace my old hba card, but I'm not certain of what the recommended procedure would be:

Shutdown the NAS, swap out the cards, cable the new card to the new enclosure, mount all the drives in the new enclosure, power on the new enclosure, power on the NAS. The NAS will see all the drives in their new locations and tell you there is a change (in the log) but the pool should import and work just as it did. I have actually done that myself both at home and at work. FreeNAS uses the gptid to locate the drives, not the da# so it doesn't care where the drive is connected as long as it is able to find the drive.

Replace the old hba with the new one, consider the old ZFS container lost, move all the drives into the enclosure, create a new ZFS container, and restore my files from cold storage. This is my fallback plan, but I don't love it because of how long it could take and the potential for little issues to crop up.

No, you should not have that problem at all.

Replace the current hba card with the new one and move all of the drives to the external enclosure, then connect the enclosure to the new card. Hope and pray that the old drives and ZFS container will be auto-magically recognized through the new adapter.

This isn't magical. This is how it should work. The SAS cabling is just a data path. Kind of like replacing a network switch should not change the network configuration of the computers that are connected by the switch.

If anyone has a suggestion I would be grateful for the advice. Thanks!

I would have suggested something totally different, but you already bought hardware. Or, did you?

 

[cvejic]

First, thank you foir the quick response! And sorry about my delay in responding. I periodically have to shutdown the NAS to cool off and I saw your response just after I had shutdown. Next, referring to my use of the term "magical" I should mention that I don't believe in "technology"; the earth is flat, gravity is caused by angels pulling us to the ground, and whenever software runs a wizard did it. I face many challenges over my personal belief system as I am in fact employed as a development lead and insist upon telling customers when their "incantation" is ready. But out of respect I'll accept for now this "by design" concept you seem to be bandying about.

I did purchase the hardware, but only Friday (the JBOD enclosure) and this morning (the new hba controller) so I *might* be able to cancel the order if the outcome is considerably desirable (there may be a restocking fee or similar for the enclosure as it was a 'special order'). In either case I would be very interested in hearing your thoughts.

Finally, here is the output of the 'zfs status' you requested:

Code:

pool: POOL-1
 
state: ONLINE

status: One or more devices has experienced an unrecoverable error.  An
    
        attempt was made to correct the error.  Applications are unaffected.

action: Determine if the device needs to be replaced, and clear the errors

    using 'zpool clear' or replace the device with 'zpool replace'.

   see: http://illumos.org/msg/ZFS-8000-9P

   scan: resilvered 153M in 0 days 00:00:07 with 0 errors on Sun Feb 24 18:47:25 2019

config:

    NAME                                            STATE     READ WRITE CKSUM

    POOL-1                                          ONLINE       0     0     0

      mirror-0                                      ONLINE       0     0     0

        gptid/0fb34c8c-8603-11e6-978a-0cc47acd8c28  ONLINE       0     0     0

        gptid/101e426e-8603-11e6-978a-0cc47acd8c28  ONLINE       0     0     0

      mirror-1                                      ONLINE       0     0     0

        gptid/108dd619-8603-11e6-978a-0cc47acd8c28  ONLINE       0     0     2

        gptid/10fc02bd-8603-11e6-978a-0cc47acd8c28  ONLINE       0     0     0

      mirror-2                                      ONLINE       0     0     0
        gptid/970383bd-9c99-11e6-b3d7-0cc47acd8c28  ONLINE       0     0     0

        gptid/e095e4b9-83e2-11e8-b1e8-0cc47acd8c28  ONLINE       0     0     0

      mirror-3                                      ONLINE       0     0     0

        gptid/578c3709-78ad-11e8-80a9-0cc47acd8c28  ONLINE       0     0     0

        gptid/c38b4977-fe65-11e7-b603-0cc47acd8c28  ONLINE       0     0     0

cache
      nvd0                                          ONLINE       0     0     0


errors: No known data errors


  pool: freenas-boot
 state: ONLINE

  scan: scrub repaired 0 in 0 days 00:00:41 with 0 errors on Thu Feb 21 03:45:41 2019

config:


    NAME        STATE     READ WRITE CKSUM

    freenas-boot  ONLINE       0     0     0

      ada0p2    ONLINE       0     0     0


errors: No known data errors

 

[Chris Moore]

and whenever software runs a wizard did it. I face many challenges over my personal belief system as I am in fact employed as a development lead and insist upon telling customers when their "incantation" is ready. But out of respect I'll accept for now this "by design" concept you seem to be bandying about.

LOL
If you were 15 years older, I would suspect you of being one of my coworkers. He has this poster on the wall of his office:

I did purchase the hardware, but only Friday (the JBOD enclosure) and this morning (the new hba controller) so I *might* be able to cancel the order if the outcome is considerably desirable (there may be a restocking fee or similar for the enclosure as it was a 'special order'). In either case I would be very interested in hearing your thoughts.

I am not sure what you paid for the parts you already bought, but this is what I did when I wanted more drive bays, for the simplicity of having it all in one chassis.
https://www.ebay.com/itm/Supermicro...-W-X8DAH-F-BPN-SAS2-846EL1-RAILS/382243380103
That is about the least expensive way to get a chassis that can accommodate a quantity of drives for the next time you want to expand storage.
You can cable the SAS expander backplane in that chassis to your existing SAS controller to run all 24 drive bays from the one SAS controller.
What I would do is take the existing system board out of that case to put mine in. It would be a pretty simple swap-out. I have done that a couple times myself. I have a chassis like that for my backup NAS and the main system is in my signature.

Finally, here is the output of the 'zfs status' you requested:

Good. There are no data errors for now.
Having a bunch of extra drive bays to work with makes changing things around much easier.

 

[cvejic]

Hah! Love that poster; I'm going to have to see about finding one.

Yah, I think I'm going to call tomorrow to see if I can cancel that order for the enclosure. It's about the same price and it has 12 fewer drive bays. Plus it sounds like you're saying it also has the option to maintain that logical simplicity of keeping the motherboard and drives in a single box. It's refurb vs new, but all you're using are the non-moving bits, some fans which are easily replaceable, and a psu which, since I keep everything on ups, should be fairly low risk.

Wow, awesome man. I appreciate the help! Let me know if you ever have a data analysis or machine learning question and I'd be happy to return the favor. :)

Thanks again!

 

[SweetAndLow]

Hah! Love that poster; I'm going to have to see about finding one.

Yah, I think I'm going to call tomorrow to see if I can cancel that order for the enclosure. It's about the same price and it has 12 fewer drive bays. Plus it sounds like you're saying it also has the option to maintain that logical simplicity of keeping the motherboard and drives in a single box. It's refurb vs new, but all you're using are the non-moving bits, some fans which are easily replaceable, and a psu which, since I keep everything on ups, should be fairly low risk.

Wow, awesome man. I appreciate the help! Let me know if you ever have a data analysis or machine learning question and I'd be happy to return the favor. :)

Thanks again!

Just made sure you are ok with the noise this box makes. Mine is in my basement and you can hear the fans spin up from every room. The PSU fans are the loudest because of the high pitch of them. The chassis fans are not as loud as long as the box isn't working as much.

 

[Chris Moore]

Just made sure you are ok with the noise this box makes. Mine is in my basement and you can hear the fans spin up from every room. The PSU fans are the loudest because of the high pitch of them. The chassis fans are not as loud as long as the box isn't working as much.

You just need to do some sound insulation like this:

ghetto acoustic by @jgreco
https://forums.freenas.org/index.ph...-mb-cpu-hds-recommendations.24453/post-255614

 

[cvejic]

Actually, my holiday gift to myself this December was to get some professionals to wire my home office to my basement (and a couple of other rooms), where I've placed a cheap half-rack. So I'm banishing all of my servers, switches, and my NAS down there. The scheduling took a bit of doing so it won't actually happen until March, but as of then I won't have to worry about the noise. I was more worried about fitting it into my rack, but I don't think it will be an issue. And absolute worst case, I'll just put a table down there.

 

[Chris Moore]

I wish I had a basement.

 

[cvejic]

It has its advantages, although it can be a bit of a reminder of all the things I need to take care of. There's the boiler that's 30 years old and needs to be replaced, next to the old heating oil drums that need to be dismantled and removed right past the water purifier that needs to be refilled and the cracks I need to putty. Still, I count my blessings; my previous home only had a crawl space that you literally had to belly climb through parts of to reach the pipes. I HATED going down there, but animals would eat off the insulation and the pipes would freeze if you didn't re-apply occasionally (Wisconsin winters are brutal). I moved two years ago; this house is still older, but thankfully in better condition. My favorite part of the basement is the 15 foot work bench the previous owner installed. It's perfect for wasting my time on diy projects, lol.