BUILD Building a FreeNAS box

[Ericloewe]

In the end, three of the original eight 5 TB drives died during the changeout.

Yikes

 

[jgreco]

In the end, three of the original eight 5 TB drives died during the changeout.

Holy crap. And people call me paranoid for using RAIDZ3.

 

[Mark Holtz]

Just to clarify... the three drives did not throw errors at the same time (thank goodness), but it did cause some worry points. To clarify:

• The first drive was ada2, and initially took itself offline on Wednesday, September 8th. I did a reboot which brought the drive back online, but then got reports of bad sectors. It was at this point that I elected to order five 8TB NAS drives from Amazon (order limit) at $199 each, with an expected receipt date of September 15th. ada2 later on threw some bad sectors.
• The Amazon drives arrived early on Friday, September 10th, and I began the replacement process, starting with the faulty ada2. As I replaced each drive, I also was wiping them utilizing Active@ Killdisk on my main desktop computer. Also, Newegg had a special on 8TB NAS drives also at $199, so I ordered three more.
• On Sunday, September 12th, as I replaced and resilvered the fifth 8TB at ada4, ada6 started to throw read/write errors. With no available 8 TB drives, I grabbed a good (at the time) 5 TB drive and used that as a temporary replacement.
• The 3 NAS drives from NewEgg arrived on Wednesday, September 15th, and I immediately started replacing ada5. And right after I started the resilvering process, the 5TB drive on ada6 threw errors. I was up late last night waiting for the resilvering process to complete on ada5 so that I could pull ada6 as quickly as possible.

The old drives were the HGST, while the new drives are Seagate NAS. And, I think the transfer rates of the Seagate NAS drives are much faster than the HGSTs. I am a little surprised, but was prepared, in how those old HGST drives died.

 

[Mark Holtz]

Oh, should I mention that I filed the following Jira's as a result of this resilvering process:

• NAS-112358 - Clicking on spinning "Resilvering" icon should also display time remaining
• NAS-112351 - Error: "Disk 10336936891386576613 is FAULTED" - What is 10336936891386576613?
• NAS-112268 - Storage -> Disks should include when drive was brought online in TrueNas
• NAS-112262 - During resilver process, WebUI shows green checkmark in Dashboard -> Pool status box
• NAS-112265 - During resilver, "Time Remaining" is blank in Storage / Pools / Pool Status which is probably related to NAS-112261 - WebUI crashed: "Connecting to TrueNAS ... Make sure the TrueNAS system is powered on and connected to the network"

 

[Etorix]

Holy crap. And people call me paranoid for using RAIDZ3.

“Just because you're paranoid doesn't mean they aren't after you.”

I suppose this was nice case of the strain of resilver killing further ageing drives.
@Mark Holtz Thanks for the write-up and the bug reports. Your adventure has made a good contribution.

 

[Mark Holtz]

Today is July 30th, 2018...

(Text Deleted)

Someday, when I get a better Internet connection, I will backup some critical folders to a cloud service.

Just as an amendment... six months after that posted, I actually did more to a better place which included internet that was about 960 times faster than the 1 Megabit upload speeds that I had with my ADSL connection. I read a story at the end of June about WD My Book Live users who lost all of their data. This is a product that got it's last firmware update six years ago. Although it shares little in common with my TrueNAS server, it got me to investigating the Cloud Sync Tasks and how it integrates with Backblaze. Although most of my stuff is covered by a backing up to local hard drive, I have identified several critical directories, and have set up an daily offsite/online backup to Backblaze, including downloading a copy of the files on my cloud drive services then reloading the backup to Backblaze. This allowed me to (temporarily) turn off periodic snapshot tasks for those directories due to disk space issues. Those have been re-enabled today.

 

[Constantin]

Although most of my stuff is covered by a backing up to local hard drive, I have identified several critical directories, and have set up an daily offsite/online backup to Backblaze, including downloading a copy of the files on my cloud drive services then reloading the backup to Backblaze. This allowed me to (temporarily) turn off periodic snapshot tasks for those directories due to disk space issues. Those have been re-enabled today.

Off-site backups are a critical issue that most folk ignore at their peril. The Z3 I run here will likely protect my data indefinitely as long as I keep maintenance up, etc. But if the house burns down, all that infrastructure is kaplooie. It's also something to think about in the context of password managers. How will you recover should your primary CPU and your phone get destroyed? Can you get into your backups? Etc. It's all well and good to follow best practices with a password manager, etc. but actually think through what the contingency plan entails. Paper copies of strong passwords in the right places could be part of the plan, ditto should those passwords be needed in case you unexpectedly get incapacitated.

 

[Mark Holtz]

Speaking of backups, I finally sat down and created the TAR command to archive my Plex data files as root from the jail... ... after completely losing them not once by TWICE.

Code:

tar
--exclude=/Plex\ Media\ Server/Cache
--exclude=/Plex\ Media\ Server/Backup
--exclude=/Plex\ Media\ Server/Vaultron
-zcvf /Plex\ Media\ Server/Backup/Plexbackup.tar.gz /Plex\ Media\ Server

This is, of course, all one line, but broken down into some human readable for this forum. Backup is linked to my backup folder, while Vaultron is linked to my Media library.

 

[Mark Holtz]

And, as I expected, the prices dropped to where I wanted them to be for 10TB and 12TB drives during the Black Friday holiday. My timing sucks.But, a data loss would have been worse....

 

[jgreco]

And, as I expected, the prices dropped to where I wanted them to be for 10TB and 12TB drives during the Black Friday holiday. My timing sucks.But, a data loss would have been worse....

I feel your pain. Hopefully Chia implodes and HDD prices go back to normal.