LordHelmet
Cadet
- Joined
- Jan 2, 2023
- Messages
- 3
Good day,
Here are the specs of my setup:
]
Version of Truenas is TrueNAS-SCALE-22.02.2.1
So I had to cold start everything after a power outage (had a clean shutdown) and I noticed some alerts on the system:
I also noticed amber LED on 2 drives located on vdev raidz2-2 (on the KTN-STL3 enclosure). I went and checked the status of the pool on the dashboard.
Pool status is online
Disks with error is 0.
However in the "storage" section it says Pool online, Disks with error: 9
I noticed that all 3 of my spare drives were in use and resilvering is in progress.
Went in CLI and got this:
Here are the data errors:
All the problematic drives are in the same enclosure and everything was fine before the shutdown. Weird coincidence.
I have access to all the data and have a copy of everything so I let it do it's things.
A few hours later:
Now somehow I have 2 spares in use and in spare-6 both the original drives and the spares are silvering ? I just don't get what is happening right now.
There is clearly a problem but isn't the array supposed to be degraded when a rebuild is in progress?
Why would a spare and the original drive be rebuilding at the same time?
Somehow d6590add-659e-454a-8b36-296b8fc3fa95 that was being replaced by 784f8ca3-5c4b-4fda-919b-71d5dd32a09a is now ok and 784f8ca3-5c4b-4fda-919b-71d5dd32a09a is back as a spare but I did not replace any drive in the enclosure yet.
I also see that it goes from spare 0-1 to spare 6. Is it playing what a mole with drives?
I'm thinking that the EMC enclosure is the issue here. Drive failure happens, all drives showing errors in the same enclosure is a lottery ticket level of failure if it's really the drives.
Sorry if I sound ignorant, I pretty new to truenas and ZFS. I come from unraid and Truenas appears to be the big boys league and I'm getting a little overwhelmed :D
I've added the system debug to the post since I probably forgot stuff. Thanks for reading my wall of text.
Here are the specs of my setup:
HP Proliant ML350P Gen8
2X Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
378GiB of ECC memory
LSI SAS9200-16E IT mode raid card
HP dual 10gig SFP network card
Netapp DS4246 containing vdev raidz2-0 and raidz2-1
24 HGST 3TB HDD configured as 2x 12drives in raidz2
EMC KTN-STL3 containing vdev raidz2-2
15 HGST 3TB HDD confitured as 1x 12 drives in raidz2 + 3 hot spares for the pool
All drives are HGST HUS72403CLAR3000 3TB 7200rpm SAS drives
2X Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
378GiB of ECC memory
LSI SAS9200-16E IT mode raid card
HP dual 10gig SFP network card
Netapp DS4246 containing vdev raidz2-0 and raidz2-1
24 HGST 3TB HDD configured as 2x 12drives in raidz2
EMC KTN-STL3 containing vdev raidz2-2
15 HGST 3TB HDD confitured as 1x 12 drives in raidz2 + 3 hot spares for the pool
All drives are HGST HUS72403CLAR3000 3TB 7200rpm SAS drives
Version of Truenas is TrueNAS-SCALE-22.02.2.1
So I had to cold start everything after a power outage (had a clean shutdown) and I noticed some alerts on the system:
Code:
WARNING Core files for the following executables were found: /usr/bin/rrdcached (Sat Dec 31 17:11:59 2022), /usr/bin/git (Sat Dec 31 17:15:32 2022), /usr/bin/git (Sat Dec 31 17:16:13 2022). Please create a ticket at https://jira.ixsystems.com/ and attach the relevant core files along with a system debug. Once the core files have been archived and attached to the ticket, they may be removed by running the following command in shell: 'rm /var/db/system/cores/*'. 2022-12-31 17:16:34 (America/Montreal)
I also noticed amber LED on 2 drives located on vdev raidz2-2 (on the KTN-STL3 enclosure). I went and checked the status of the pool on the dashboard.
Pool status is online
Disks with error is 0.
However in the "storage" section it says Pool online, Disks with error: 9
I noticed that all 3 of my spare drives were in use and resilvering is in progress.
Went in CLI and got this:
Code:
NAME STATE READ WRITE CKSUM
zfs ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
c3d1ecad-f9ad-4d09-839d-7552f69fdf29 ONLINE 0 0 0
a83f77b3-620a-4e55-b172-3af628a709a6 ONLINE 0 0 0
4d2d4f62-771d-4432-a618-41b2a04e82fa ONLINE 0 0 0
05015c9d-1ae2-410c-91df-d56e8b4f7f02 ONLINE 0 0 0
51a2531f-d505-4bb1-8a22-c05fd189d759 ONLINE 0 0 0
6b3046ef-3fed-4139-889c-e4033f3c231a ONLINE 0 0 0
fd046b86-2471-4862-a718-446444c10896 ONLINE 0 0 0
42ec02ea-1828-448d-bbd5-6d7c97db120c ONLINE 0 0 0
e04a8906-49cc-42af-817d-3ccd9bd97eb9 ONLINE 0 0 0
46d5c3b9-811c-4323-8b27-426e25615fb2 ONLINE 0 0 0
0473a7ea-8b64-4f96-a7ac-43c12763656f ONLINE 0 0 0
64fce098-512a-4a32-a4be-4c8359837ba0 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
e1fe4ba2-9edc-4b0c-8a42-a76e5872f2b9 ONLINE 0 0 0
69f9ad2c-87fc-4804-84d8-8ae7b201caee ONLINE 0 0 0
c5b0ac2f-d9de-4408-9a3a-ead396a56333 ONLINE 0 0 0
e4cbeed8-a9f9-4fce-be23-e9e8edc751d6 ONLINE 0 0 0
e4380b62-7c31-48ff-9178-92ab26d334fe ONLINE 0 0 0
1fcc25c4-1936-4d57-8352-8455f030278e ONLINE 0 0 0
6a164df6-ae4c-4f40-a35a-1829ffc387b2 ONLINE 0 0 0
77a75b57-d838-4261-9a2e-12ee58dff156 ONLINE 0 0 0
231685b9-59cb-430e-8d4c-87dee5f7e2f1 ONLINE 0 0 0
99d2c5cf-aa4a-4988-a7d6-d88d703f0da4 ONLINE 0 0 0
2dd54415-3ceb-4a46-bc80-75a8e3862f74 ONLINE 0 0 0
5f5f5fff-66a6-48b9-ba34-5070e0ebb4bf ONLINE 0 0 0
raidz2-2 ONLINE 4 55 0
spare-0 ONLINE 0 0 0
4be3720e-3939-4135-9c5d-e38a81c4c0c9 ONLINE 3 15 0
2f766d8f-9672-463a-9d32-d6ca63d40070 ONLINE 0 0 0 (resilvering)
spare-1 ONLINE 0 0 0
d6590add-659e-454a-8b36-296b8fc3fa95 ONLINE 3 17 0
784f8ca3-5c4b-4fda-919b-71d5dd32a09a ONLINE 0 0 0 (resilvering)
55be3617-9ac0-4070-a5c6-083eef3b4555 ONLINE 3 17 0
fabb6dd6-8c73-4b61-9b8a-7facad602d6d ONLINE 3 15 0
0174f120-fae0-402c-ad48-50b781a2a4ca ONLINE 3 15 0
d14a495c-54f9-4a99-b090-a23860fac56e ONLINE 3 12 0
spare-6 ONLINE 0 0 0
1cd9bfb7-0d90-4f04-b490-b749aed38b59 ONLINE 3 12 0
680ec236-02af-4d9a-be49-453f184a0950 ONLINE 0 0 0 (resilvering)
7f90121f-c7a2-442f-85a9-0075ef41846a ONLINE 3 14 0
bdd110a4-29cb-4682-bd2f-cd433189a193 ONLINE 3 14 0
3bce65e9-b37b-48f2-81e7-b04e2272b8bf ONLINE 4 16 0
193227b9-6f02-4fcb-a007-a8a0e1063bdf ONLINE 4 14 0
51dc7795-627b-4f69-a895-f6518af0a5ed ONLINE 4 17 0
spares
2f766d8f-9672-463a-9d32-d6ca63d40070 INUSE currently in use
680ec236-02af-4d9a-be49-453f184a0950 INUSE currently in use
784f8ca3-5c4b-4fda-919b-71d5dd32a09a INUSE currently in use
errors: 274 data errors, use '-v' for a list
Here are the data errors:
Code:
errors: Permanent errors have been detected in the following files:
zfs/ix-applications/catalogs:<0xb217>
zfs/ix-applications/catalogs:<0xb219>
zfs/ix-applications/catalogs:<0xb21e>
zfs/ix-applications/catalogs:<0xb226>
zfs/ix-applications/catalogs:<0xb22c>
zfs/ix-applications/catalogs:<0xb22e>
zfs/ix-applications/catalogs:<0xb22f>
zfs/ix-applications/catalogs:<0xb234>
zfs/ix-applications/catalogs:<0xb235>
zfs/ix-applications/catalogs:<0xb236>
zfs/ix-applications/catalogs:<0xb238>
zfs/ix-applications/catalogs:<0xb23b>
zfs/ix-applications/catalogs:<0xb23e>
zfs/ix-applications/catalogs:<0x840>
zfs/ix-applications/catalogs:<0xb240>
zfs/ix-applications/catalogs:<0xb246>
zfs/ix-applications/catalogs:<0xb249>
zfs/ix-applications/catalogs:<0xb250>
zfs/ix-applications/catalogs:<0xb252>
zfs/ix-applications/catalogs:<0xb253>
zfs/ix-applications/catalogs:<0x755>
zfs/ix-applications/catalogs:<0xb25a>
zfs/ix-applications/catalogs:<0xb260>
zfs/ix-applications/catalogs:<0xb26e>
zfs/ix-applications/catalogs:<0xb271>
zfs/ix-applications/catalogs:<0xb27a>
zfs/ix-applications/catalogs:<0xb27c>
zfs/ix-applications/catalogs:<0x25c7d>
zfs/ix-applications/catalogs:<0xb27d>
zfs/ix-applications/catalogs:<0x781>
zfs/ix-applications/catalogs:<0x783>
zfs/ix-applications/catalogs:<0x785>
zfs/ix-applications/catalogs:<0x78c>
zfs/ix-applications/catalogs:<0x78e>
zfs/ix-applications/catalogs:<0x792>
zfs/ix-applications/catalogs:<0x94>
zfs/ix-applications/catalogs:<0x794>
zfs/ix-applications/catalogs:<0x7a0>
zfs/ix-applications/catalogs:<0x7a4>
zfs/ix-applications/catalogs:<0x7a6>
zfs/ix-applications/catalogs:<0x7a7>
zfs/ix-applications/catalogs:<0x7a9>
zfs/ix-applications/catalogs:<0x7af>
zfs/ix-applications/catalogs:<0x7b0>
zfs/ix-applications/catalogs:<0x7b1>
zfs/ix-applications/catalogs:<0x7b2>
zfs/ix-applications/catalogs:<0x7b3>
zfs/ix-applications/catalogs:<0x7b8>
zfs/ix-applications/catalogs:<0x7b9>
zfs/ix-applications/catalogs:<0x7ba>
zfs/ix-applications/catalogs:<0x7be>
zfs/ix-applications/catalogs:<0x7c0>
zfs/ix-applications/catalogs:<0x7c1>
zfs/ix-applications/catalogs:<0x7c5>
zfs/ix-applications/catalogs:<0x7e1>
zfs/ix-applications/catalogs:<0x7e2>
zfs/ix-applications/catalogs:<0x7e4>
zfs/ix-applications/catalogs:<0x7ee>
zfs/ix-applications/catalogs:<0x7ef>
zfs/ix-applications/catalogs:<0x7f2>
zfs/ix-applications/catalogs:<0x5f4>
zfs/ix-applications/catalogs:<0x7ff>
/mnt/zfs/books/Manga/Initial D/Initial D Vol 23/chapter280/Initial D v23 c280 pg096.png
/mnt/zfs/books/Manga/Jiraishin/V10/Jiraishin v10c19Cp119 copy.jpg
/mnt/zfs/books/Manga/Jiraishin/V10/Jiraishin v10c19Cp120 copy.jpg
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c55/Over Rev v06 c058 p175.png
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c55/Over Rev v06 c058 p176.png
/mnt/zfs/books/Manga/Jiraishin/V10/Jiraishin v10c19Cp123 copy.jpg
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c56/Over Rev v06 c056 p189.png
/mnt/zfs/books/Manga/Jiraishin/V10/Jiraishin v10c19Cp125 copy.jpg
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c56/Over Rev v06 c056 p188.png
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c56/Over Rev v06 c056 p193.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 19/Chapter 223/Initial D v19 c223 pg150.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 19/Chapter 223/Initial D v19 c223 pg151.png
/mnt/zfs/books/Manga/Jiraishin/V14/Jiraishin_v14c24Cp137.jpg
/mnt/zfs/books/Manga/Jiraishin/V14/Jiraishin_v14c24Cp155.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 20/Initial_D-v20c239[Inane]/Initial D v20 c239 pg175.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 12/Chapter 126/Initial D - Vol.12 - Ch.126 - 44.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 12/Chapter 127/Initial D - Vol.12 - Ch.127 - 48.png
/mnt/zfs/books/Manga/Jiraishin/V10/Jiraishin v10c19Bp087 copy.jpg
/mnt/zfs/books/Manga/Overrev/Over Rev v05 c55/Over Rev v06 c058 p183.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 06/Chapter 58/Initial D v06 c058 070-071.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 05/Chapter 45/Initial D v05 c045 034-035.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 06/Chapter 58/Initial D v06 c058 081.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 05/Chapter 46/Initial D v05 c046 055.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 23/chapter278/Initial D v23 c278 pg069.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 23/chapter279/Initial D v23 c279 pg083.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 23/chapter279/Initial D v23 c279 pg088.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 23/chapter279/Initial D v23 c279 pg089.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 05/Chapter 48/Initial D v05 c048 105.jpg
/mnt/zfs/books/Manga/Initial D/Initial D Vol 12/Chapter 125/Initial D - Vol.12 - Ch.125 - 33.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 12/Chapter 125/Initial D - Vol.12 - Ch.125 - 35.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 19/Chapter 223/Initial D v19 c223 pg143.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 12/Chapter 126/Initial D - Vol.12 - Ch.126 - 42.png
/mnt/zfs/books/Manga/Initial D/Initial D Vol 20/Initial_D-v20c235[Inane]/Initial D v20 c235 pg121.png
zfs/media:<0x0>
/var/db/system/syslog-d968e280827d4008b5065e3ef738a9cd/log/samba4/auth_audit.log
/var/db/system/rrd-d968e280827d4008b5065e3ef738a9cd/localhost/disk-sdae/disk_merged.rrd
zfs/ix-applications/docker/61e734d1c9a1c2bfc69e62d2721fe3b20d50c25315b366ea2adcd46df2f9dd18@63846876:<0x0>
zfs/nzb:<0x0>
zfs/games:<0x0>
/mnt/zfs/games/Sony PS Vita/app/PCSE00518/sce_sys/livearea/contents/bg0.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00328/Art/Maya/Shaders/GXM
/mnt/zfs/games/Sony PS Vita/app/PCSE00702/sce_sys/manual/02/006.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00702/sce_sys/manual/03/004.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00702/sce_sys/trophy/NPWR09047_00/TROPHY.TRP
/mnt/zfs/games/Sony PS Vita/app/PCSE00103/sce_module/libc.suprx
/mnt/zfs/games/Sony PS Vita/app/PCSA00086/assets/devel/built/tuids/e75/e75c80d7fea31f8d/texel.dat
/mnt/zfs/games/Sony PS Vita/app/PCSE00708/Media/StreamingAssets/Lua/Chapter_EX_04
/mnt/zfs/games/Sony PS Vita/app/PCSE01103/sce_sys/manual/02/015.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00113/data_psp2/objects/boss_laser/rocket_window.pxt
/mnt/zfs/games/Sony PS Vita/app/PCSE00273/DATA/41088/00000/41146.LIB
/mnt/zfs/games/Sony PS Vita/app/PCSE00273/DATA/42624/00000/42678.LIB
/mnt/zfs/games/Sony PS Vita/app/PCSE00761/Data_x32/Characters/SubSwarmer.GXM.phyre
/mnt/zfs/games/Sony PS Vita/addcont/PCSE00800/DW8EMPEDITEQUIP4/USRDIR/DLC/DATA/34344.bin
/mnt/zfs/games/Sony PS Vita/addcont/PCSE00800/DW8EMPEDITEQUIP4/USRDIR/DLC/DATA/32240.bin
/mnt/zfs/games/Sony PS Vita/app/PCSE00812/sce_sys/manual/02/007.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00529/Music/VOICEOVER/COMMENTATOR/brazilianportuguese/intro_inp_rc_moto2_nocareer_01.at9
/mnt/zfs/games/Sony PS Vita/app/PCSE00036/Lvl_BASE_01/objets/doorswitchdestruct/compiled
/mnt/zfs/games/Sony PS Vita/app/PCSE00749/eboot.bin
/mnt/zfs/games/Sony PS Vita/app/PCSE00113/data_psp2/objects/chi/Smoke04.pxt
/mnt/zfs/games/Sony PS Vita/app/PCSA00110/sce_module/libc.suprx
/mnt/zfs/games/Sony PS Vita/app/PCSE00735/Runner/Models/Tiles/Level2/Tile019/Textures/Top.aetex
/mnt/zfs/games/Sony PS Vita/app/PCSE00735/Runner/Models/Tiles/Level2/Tile020/Textures/Side.aetex
/mnt/zfs/games/Sony PS Vita/app/PCSA00106/data/spirit/cut_in/s202_00_005_c.gxt
/mnt/zfs/games/Sony PS Vita/app/PCSE00735/Runner/Models/Tiles/Level2/Tile021/Tile021.aescn
/mnt/zfs/games/Sony PS Vita/app/PCSE00273/DATA/41472/00000/41482.LIB
/mnt/zfs/games/Sony PS Vita/app/PCSE00792/jp/images/title_bg.tx2
/mnt/zfs/games/Sony PS Vita/app/PCSE00001/sce_sys/manual/02/023.png
/mnt/zfs/games/Sony PS Vita/app/PCSA00011/sce_sys/manual/03/043.png
/mnt/zfs/games/Sony PS Vita/app/PCSA00110/sce_sys/manual/008.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00273/DATA/41472/00000/41494.LIB
/mnt/zfs/games/Sony PS Vita/app/PCSA00110/sce_sys/manual/014.png
/mnt/zfs/games/Sony PS Vita/addcont/PCSE00800/DW8EMPEDITEQUIP4/USRDIR/DLC/DATA/33279.bin
/mnt/zfs/games/Sony PS Vita/app/PCSE00749/sce_sys/trophy/NPWR09279_00/TROPHY.TRP
/mnt/zfs/games/Sony PS Vita/app/PCSE00812/sce_sys/manual/02/024.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00529/Music/VOICEOVER/COMMENTATOR/brazilianportuguese/wc_day_rc_lrain_03.at9
/mnt/zfs/games/Sony PS Vita/app/PCSA00106/data/spirit/cut_in/s405_10_007_c.gxt
/mnt/zfs/games/Nintendo SNES games/ROMS/Exhaust Heat (E) [!].smc
/mnt/zfs/games/Sony PS Vita/app/PCSE01111/pack/media/stand/c11/base0.dds
/mnt/zfs/games/Sony PS Vita/app/PCSE00016/kabuto/geometry_data/MATES_ITEM/KD5_OUJI07_ITEM_HEAD_FLOWER/KD5_OUJI07_ITEM_HEAD_FLOWER.nut
/mnt/zfs/games/Nintendo SNES games/ROMS/F1 World Championship Edition (E).smc
/mnt/zfs/games/Sony PS Vita/app/PCSA00011/sce_sys/manual/17/019.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00016/kabuto/geometry_data/MATES_ITEM/KD5_OUJI07_ITEM_HEAD_HEADPHONE/KD5_OUJI07_ITEM_HEAD_HEADPHONE.nut
/mnt/zfs/games/Sony PS Vita/app/PCSA00011/sce_sys/manual/17/024.png
/mnt/zfs/games/Sony PS Vita/app/PCSE00016/kabuto/geometry_data/MATES_ITEM/KD5_OUJI07_ITEM_HEAD_OUKAN/KD5_OUJI07_ITEM_HEAD_OUKAN.nut
/mnt/zfs/games/Sony PS Vita/app/PCSE01280/sce_module/libsmart.suprx
/mnt/zfs/games/Sony PS Vita/app/PCSE00001/sce_sys/manual/01/020.png
/mnt/zfs/games/Sony PS Vita/app/PCSA00008/eboot.bin
/mnt/zfs/personnel/My Documents/My Pictures/Photos de lan/LPBA9/CIMG0835.JPG
/mnt/zfs/personnel/My Documents/My Pictures/Photos de lan/LPBA8/CIMG0251.JPG
/mnt/zfs/personnel/My Documents/My Pictures/Mes photos/2015-06-24 - St-Amable toasté/GIN_6891_00085.jpg
All the problematic drives are in the same enclosure and everything was fine before the shutdown. Weird coincidence.
I have access to all the data and have a copy of everything so I let it do it's things.
A few hours later:
Code:
raidz2-2 ONLINE 4 55 0
spare-0 ONLINE 0 0 0
4be3720e-3939-4135-9c5d-e38a81c4c0c9 ONLINE 3 15 0
2f766d8f-9672-463a-9d32-d6ca63d40070 ONLINE 0 0 0 (resilvering)
d6590add-659e-454a-8b36-296b8fc3fa95 ONLINE 3 17 0
55be3617-9ac0-4070-a5c6-083eef3b4555 ONLINE 3 17 0
fabb6dd6-8c73-4b61-9b8a-7facad602d6d ONLINE 3 15 0
0174f120-fae0-402c-ad48-50b781a2a4ca ONLINE 3 15 0
d14a495c-54f9-4a99-b090-a23860fac56e ONLINE 3 12 0
spare-6 ONLINE 0 0 75.9K
1cd9bfb7-0d90-4f04-b490-b749aed38b59 ONLINE 3 12 0 (resilvering)
680ec236-02af-4d9a-be49-453f184a0950 ONLINE 0 0 0 (resilvering)
7f90121f-c7a2-442f-85a9-0075ef41846a ONLINE 3 14 0
bdd110a4-29cb-4682-bd2f-cd433189a193 ONLINE 3 14 0
3bce65e9-b37b-48f2-81e7-b04e2272b8bf ONLINE 4 16 0
193227b9-6f02-4fcb-a007-a8a0e1063bdf ONLINE 4 14 0
51dc7795-627b-4f69-a895-f6518af0a5ed ONLINE 4 17 0
spares
2f766d8f-9672-463a-9d32-d6ca63d40070 INUSE currently in use
680ec236-02af-4d9a-be49-453f184a0950 INUSE currently in use
784f8ca3-5c4b-4fda-919b-71d5dd32a09a AVAILNow somehow I have 2 spares in use and in spare-6 both the original drives and the spares are silvering ? I just don't get what is happening right now.
There is clearly a problem but isn't the array supposed to be degraded when a rebuild is in progress?
Why would a spare and the original drive be rebuilding at the same time?
Somehow d6590add-659e-454a-8b36-296b8fc3fa95 that was being replaced by 784f8ca3-5c4b-4fda-919b-71d5dd32a09a is now ok and 784f8ca3-5c4b-4fda-919b-71d5dd32a09a is back as a spare but I did not replace any drive in the enclosure yet.
I also see that it goes from spare 0-1 to spare 6. Is it playing what a mole with drives?
I'm thinking that the EMC enclosure is the issue here. Drive failure happens, all drives showing errors in the same enclosure is a lottery ticket level of failure if it's really the drives.
Sorry if I sound ignorant, I pretty new to truenas and ZFS. I come from unraid and Truenas appears to be the big boys league and I'm getting a little overwhelmed :D
I've added the system debug to the post since I probably forgot stuff. Thanks for reading my wall of text.
