Hello everyone,
I just built my first TrueNAS system (specs below) with repurposed old gaming computer hardware and eight 16TB Ironwolf Pro drives in a single pool, ZFS-R2. This hardware was all running smoothly for over a year as a windows machine running some server software. The only thing I changed when I converted it to TrueNAS was adding the eight HDDs, moved the HBA controller from my linux machine onto this machine, added a 10 gbps NIC, new SATA power cables from PSU to the HDDs, and the chassis is new as well. All the HDDs are connected via the HBA.
I migrated over the data from my old HDDs via SMB shares (from a linux mint computer I'm using to run various apps like Plex, home assistant, etc) and when I checked it out I had errors below:
I ran a scrub and zpool status revealed the following:
This is of course quite concerning to me because the whole point in this project was to better protect my data. The idea is to have this TrueNAS system for all files (including big media files), and to back up the really important documents (tax stuff, photos, etc) to both a second local system and also the cloud (Storj or backblaze I think, but haven't done this yet). I was loving how fast and seemingly reliable the system was at first, but now with these errors I've lost all my confidence in it and I'm really disappointed.
Specific questions:
1) How serious are these errors? Can I simply delete the affected files and copy them over again and carry on?
2) Is this indicative of an underlying problem? Or is it expected that out of like 12 TB transferred some files may have errors?
3) Does this just affect the files or also the operating system? (the part about "Applications may be affected." scares me, is this a default phrase?)
4) Could additional files be corrupted as well or does the system pretty reliably pick out the bad ones and I can assume the rest are OK?
5) What exactly does CKSUM "185 too many errors" even mean (from the scrub log)?
6) Does any of this indicate that a specific drive is faulty? I have a hard time believing all 8 brand new drives are faulty. I assume it must be something upstream?
7) What should I do now? I'm basically done transferring all data onto the pool, and next step was to set up a cloud backup, but if the whole thing needs to be rebuilt then obviously that comes first, however painful...
Thank you for any advise. I have read through many forums posts which are similar but not quite answering my questions.
I just built my first TrueNAS system (specs below) with repurposed old gaming computer hardware and eight 16TB Ironwolf Pro drives in a single pool, ZFS-R2. This hardware was all running smoothly for over a year as a windows machine running some server software. The only thing I changed when I converted it to TrueNAS was adding the eight HDDs, moved the HBA controller from my linux machine onto this machine, added a 10 gbps NIC, new SATA power cables from PSU to the HDDs, and the chassis is new as well. All the HDDs are connected via the HBA.
Code:
OS: TrueNAS Core13.0-U5.3 Mobo: Gigabyte GA-Z270XP-SLI (rev 1.0) CPU: Intel i7-6700k RAM: Corsair 32 GB (8x4) (forget exact model) OS drive: Samsung 980 pro m2 Pool drives: Seagate Ironwolf Pro 16 TB x 8 HBA: Silverstone SST-ECS04 (LSISAS2308 RAID-on-chip controller). PSU: Corsair Rm850x NIC: TP-Link 10GB PCIe network card (TX401)
I migrated over the data from my old HDDs via SMB shares (from a linux mint computer I'm using to run various apps like Plex, home assistant, etc) and when I checked it out I had errors below:
Code:
WARNING truenas.local had an unscheduled system reboot. The operating system successfully came back online at Fri Sep 1 23:40:23 2023. 2023-09-01 23:40:23 (America/Chicago) CRITICAL Pool RaptorDrive state is DEGRADED: One or more devices has experienced an error resulting in data corruption. Applications may be affected. The following devices are not healthy: Disk ATA ST16000NT001-3LV ZR5BVCV0 is DEGRADED Disk ATA ST16000NT001-3LV ZR5DY5JJ is DEGRADED Disk ATA ST16000NT001-3LV ZR5BGTE4 is DEGRADED Disk ATA ST16000NT001-3LV ZR60M2ME is DEGRADED Disk ATA ST16000NT001-3LV ZR5BVNWB is DEGRADED Disk ATA ST16000NT001-3LV ZR5CAPSJ is DEGRADED Disk ATA ST16000NT001-3LV ZR5DPBJG is DEGRADED Disk ATA ST16000NT001-3LV ZR70SQV0 is DEGRADED 2023-09-02 01:25:02 (America/Chicago)
I ran a scrub and zpool status revealed the following:
Code:
Warning: the supported mechanisms for making configuration changes are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY RESULT IN SYSTEM FAILURE. root@truenas[~]# zpool status -v RaptorDrive pool: RaptorDrive state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 668K in 02:01:41 with 92 errors on Sat Sep 2 01:30:19 2023 config: NAME STATE READ WRITE CKSUM RaptorDrive DEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 gptid/fb577ec1-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 185 too many errors gptid/fc09d02d-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 187 too many errors gptid/fc2c57b7-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 190 too many errors gptid/fc383726-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 187 too many errors gptid/fc229ad8-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 188 too many errors gptid/fcc38e50-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 188 too many errors gptid/fa5e8a52-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 190 too many errors gptid/fbf72583-4862-11ee-8024-40ed006fd9a0 DEGRADED 0 0 190 too many errors errors: Permanent errors have been detected in the following files: /mnt/RaptorDrive/Matt Personal/Radiology/Study Materials/AIRP/Gastrointestinal/AIRP - GI 114a - Celiac Disease.mp4 /mnt/RaptorDrive/Matt Personal/My Videos/Safari Video.m2ts /mnt/RaptorDrive/Matt Personal/Radiology/Study Materials/AIRP/Pediatric/AIRP - PD 109 - Cystic Renal Disease.mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x20 - Allies.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x11 - The Return (2).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x13 - Irresponsible.mkv /mnt/RaptorDrive/Media/TV Shows/Star Wars - The Clone Wars/Season 3 - Secrets Revealed (2010-2011)/The Clone Wars S03E05 HDTV - Corruption.avi /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x15 - The Game.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x05 - Travelers.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x02 - The Seed.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 12 S12 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S12E12 - About Last Night... [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stranger Things (2016)/Season 3/Stranger.Things.S03E04.iNTERNAL.1080p.WEB.X264-AMRAP.mkv /mnt/RaptorDrive/Media/TV Shows/Stranger Things (2016)/Season 3/Stranger.Things.S03E08.iNTERNAL.1080p.WEB.X264-AMRAP.mkv /mnt/RaptorDrive/Media/TV Shows/Star Wars - The Clone Wars/Season 4 - Battle Lines (2011-2012)/The Clone Wars S04E06 HDTV - Nomad Droids.avi /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 2 S02 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S02E10 - Chickenpox [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/The Deuce (2017)/Season 1/The Deuce (2017) - S01E06 - Why Me (1080p BluRay x265 RCVR).mkv /mnt/RaptorDrive/Media/TV Shows/The Office US (2005)/Season 4 (WEB)/TheOffice (US) (2005) - S04E01-E02 - Fun Run (1080p AMZN WEB-DL x265 LION).mkv /mnt/RaptorDrive/Media/TV Shows/The Mandalorian (2019)/The.Mandalorian.S01E06.REPACK.1080p.WEBRiP.x264-PETRiFiED.mkv /mnt/RaptorDrive/Media/TV Shows/The Office US (2005)/Season 7 (Bluray)/The Office (US) (2005) - S07E06 - Costume Contest (1080p BluRay x265 LION).mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 7 S07 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S07E06 - Lil' Crime Stoppers [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 12 S12 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S12E04 - Canada on Strike [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 12 S12 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S12E06 - Over Logging [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 1/1x18 - The Gift.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x13 - Quarantine.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x03 - Broken Ties.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x05 - GhostIn The Machine.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x12 - Outsiders.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 16 S16 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S16E06 - I Should Have Never Gone Ziplining [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x06 - Tabula Rasa.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x11 - Be All My Sins Remember'd.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x14 - Harmony.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x16 - Trio.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x09 - Tracker.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 3 S03 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S03E08 - Two Guys Naked in a Hot Tub (2) [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 4 S04 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S04E17 - A Very Crappy Christmas [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 16 S16 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S16E13 - A Scause for Applause [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 18 S18 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S18E07 - Grounded Vindaloop [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 20 S20 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S20E08 - Members Only [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 4 S04 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S04E06 - Cherokee Hair Tampons [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x12 - Epiphany.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x05 - Condemned.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x15 - The Tower.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x06 - The Real World.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 6 S06 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S06E15 - The Biggest Douche in the Universe [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Scrubs (2001)/Season 9/Scrubs (2001) - S09E12 - Our Driving Issues (1080p WEB-DL x265 Panda).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x14 Tao of Rodney.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 1 S01 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S01E08 - Starvin' Marvin [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x19 - The Kindred (2).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x01 - Search And Rescue.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 9 S09 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S09E07 - Erection Day [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Star Wars - The Clone Wars/Season 4 - Battle Lines (2011-2012)/The Clone Wars S04E01-E02 HDTV - Water War - Gungan Attack.avi /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 11 S11 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S11E01 - With Apologies to Jesse Jackson [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x17 - Coup D etat.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x03 - Reunion.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 15 S15 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S15E13 - A History Channel Thanksgiving [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/The Office US (2005)/Season 2 (WEB)/TheOffice (US) (2005) - S02E06 - The Fight (1080p AMZN WEB-DL x265 LION).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 1/1x14 - Sanctuary.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x10 - The Return (1).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x04 - Doppelganger.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x07 - Missing.mkv /mnt/RaptorDrive/Media/TV Shows/The Boys (2019)/Season 1/The Boys (2019) - S01E05 - Good for the Soul (1080p BluRay x265 Silence).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 4/4x18 - The Kindred (1).mkv /mnt/RaptorDrive/Media/TV Shows/The Mandalorian (2019)/The.Mandalorian.S01E05.1080p.WEBRiP.x264-PETRiFiED.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x06 - The Shrine.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x07 - Instinct.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x09 - Aurora.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 5/5x13 - Inquisition.mkv /mnt/RaptorDrive/Media/TV Shows/South Park (1997)/South Park (1997) Season 5 S05 [1080p H265][MP3 5.1 Ch]/South Park (1997) - S05E06 - Cartmanland [1080p H265][MP3 5.1 Ch].mp4 /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 1/1x04 - 38 minutes.mkv /mnt/RaptorDrive/Media/TV Shows/Scrubs (2001)/Season 9/Scrubs (2001) - S09E09 - Our Stuff Gets Real (1080p WEB-DL x265 Panda).mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 2/2x08 - Conversion.mkv /mnt/RaptorDrive/Media/TV Shows/Stargate Atlantis/Season 3/3x01 - No Man's Land.mkv
This is of course quite concerning to me because the whole point in this project was to better protect my data. The idea is to have this TrueNAS system for all files (including big media files), and to back up the really important documents (tax stuff, photos, etc) to both a second local system and also the cloud (Storj or backblaze I think, but haven't done this yet). I was loving how fast and seemingly reliable the system was at first, but now with these errors I've lost all my confidence in it and I'm really disappointed.
Specific questions:
1) How serious are these errors? Can I simply delete the affected files and copy them over again and carry on?
2) Is this indicative of an underlying problem? Or is it expected that out of like 12 TB transferred some files may have errors?
3) Does this just affect the files or also the operating system? (the part about "Applications may be affected." scares me, is this a default phrase?)
4) Could additional files be corrupted as well or does the system pretty reliably pick out the bad ones and I can assume the rest are OK?
5) What exactly does CKSUM "185 too many errors" even mean (from the scrub log)?
6) Does any of this indicate that a specific drive is faulty? I have a hard time believing all 8 brand new drives are faulty. I assume it must be something upstream?
7) What should I do now? I'm basically done transferring all data onto the pool, and next step was to set up a cloud backup, but if the whole thing needs to be rebuilt then obviously that comes first, however painful...
Thank you for any advise. I have read through many forums posts which are similar but not quite answering my questions.