USB bootdrive errors, how to stop resilvering?

[HeadCase]

Hi,

I'm running my freenas with mirrored boot (2pcs kingston 32gb datatravellers). Few months ago it got into degraded state but I managed to resilver it and it has been running ok after that. System is saying that it has no alerts, boot volume state is healthy and both usb stick are online. I still have email reports that I posted underneath, isthat usb stick still somehow bad? I can see write errors but still system says :
Boot Volume Condition: HEALTHY and staus of both USB sticks is ONLINE.


NassikkaXL.local kernel log messages:
> ugen0.6: <Kingston> at usbus0 (disconnected)
> umass1: at uhub0, port 10, addr 8 (disconnected)
> (da4:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 88 75 e4 00 00 2a 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 88 75 e4 00 00 2a 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 88 75 e4 00 00 2a 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 88 75 e4 00 00 2a 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): WRITE(10). CDB: 2a 00 00 88 75 e4 00 00 2a 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Error 5, Retries exhausted
> (da4:umass-sim1:1:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Retrying command
> (da4:umass-sim1:1:0:0): READ(10). CDB: 28 00 00 00 06 38 00 00 10 00
> (da4:umass-sim1:1:0:0): CAM status: CCB request completed with an error
> (da4:umass-sim1:1:0:0): Error 5, Retries exhausted
> da4 at umass-sim1 bus 1 scbus9 target 0 lun 0
> da4: <Kingston DataTraveler 2.0 PMAP> s/n 50E54951351BBFB0392FF468 detached
> (da4:umass-sim1:1:0:0): Periph destroyed
> ugen0.6: <Kingston> at usbus0
> umass1: <Kingston DataTraveler 2.0, class 0/0, rev 2.00/1.10, addr 9> on usbus0
> umass1: SCSI over Bulk-Only; quirks = 0xc000
> umass1:9:1:-1: Attached to scbus9
> da4 at umass-sim1 bus 1 scbus9 target 0 lun 0
> da4: <Kingston DataTraveler 2.0 PMAP> Removable Direct Access SPC-4 SCSI device
> da4: Serial Number 50E54951351BBFB0392FF468
> da4: 40.000MB/s transfers
> da4: 29532MB (60481536 512 byte sectors)
> da4: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>

-- End of security output --

 

[Robert Trevellyan]

That USB stick looks like a dud to me.

 

[HeadCase]

I bought a new USB drive. Adding it to be third drive to boot mirror. Trying to remove bad drive after that. Too bad that when viewing status of boot drives you can't see serial numbers like hard drives in Storage -> View disks.

Edit : Just started to wonder is it even possible to remove a member disk from mirrored pool...

 

[Kevin Horton]

Too bad that when viewing status of boot drives you can't see serial numbers like hard drives in Storage -> View disks.

I got around that by having my two USB sticks be from different manufacturers. "camcontrol devlist" shows the manufacturer of the stick at each /dev/daX

 

[HeadCase]

Resilvering has been running for hours... Strange that time when resilvering is started changes and has been showing different start time everytime I look it using command zpool status.

pool: freenas-boot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Jul 9 16:00:46 2016
389M scanned out of 1.81G at 40.5K/s, 10h19m to go
688M resilvered, 20.94% done
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3p2 ONLINE 0 0 0
da4p2 ONLINE 0 0 0 (resilvering)
da5p2 ONLINE 0 0 0 (resilvering)

errors: No known data errors

 

[HeadCase]

Resilvering just keep on running. Once I saw almost 70% progress... but it always starts over before hitting 100%. I know that da4p2 is the bad one and da5p2 is new usb stick. I made a mistake when inserting new usb and attached it to pool and not to use replace command to da4p2.. is there something that I can try at this point? I cannot remove da4p2 because it's identical drive to da3p2 and I cannot identify which one is which.

pool: freenas-boot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Jul 11 09:54:29 2016
330M scanned out of 1.81G at 38.2K/s, 11h22m to go
633M resilvered, 17.74% done
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
da3p2 ONLINE 0 0 0
da4p2 ONLINE 0 0 0 (resilvering)
da5p2 ONLINE 0 0 0 (resilvering)

errors: No known data errors

 

[Robert Trevellyan]

Resilvering just keep on running. Once I saw almost 70% progress... but it always starts over before hitting 100%

That suggests the boot pool is corrupted.

is there something that I can try at this point?

1. Save config.
2. Clean install.
3. Restore config.

 

[Kevin Horton]

Any chance you replaced the wrong USB stick? Maybe the one you pulled out is the good one.

 

[HeadCase]

Any chance you replaced the wrong USB stick? Maybe the one you pulled out is the good one.

I haven't removed any USB sticks. There was two and I just added third one.

 

[HeadCase]

That suggests the boot pool is corrupted.

1. Save config.
2. Clean install.
3. Restore config.

Just did this except imported volumes between steps 2 and 3. Everything running fine now. Replaced all USB memorysticks.

 

[diskdiddler]

I am in the middle of the same problem, I added a 3'rd USB key then "replaced" the bad stick since that's what the menu seems to indicate to do (bad UI, again.....)
Then an INCREDIBLY LONG (20kb/s) resliver started.
I've cancelled that, pulled out the stick and got the system back to working with 2 sticks in there. But I'm stuck, if I insert the 3'rd stick in again it will start to resliver.

See here: https://forums.freenas.org/index.ph...ng-usb-key-for-boot-volume.44810/#post-301738

NAME STATE READ WRITE CKSUM
freenas-boot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
gptid/93096e78-b686-11e4-b873-28924a2d5aca ONLINE 0 0 1
gptid/ad8cafcc-b687-11e4-ba56-28924a2d5aca ONLINE 0 0 0
16174118798259370471 REMOVED 0 0 0 was /dev/da2p2

Anyone know how I can make FreeNAS FORGET the 3'rd stick, allow me to insert it and then just mirror, rather than resliver?

 

[Robert Trevellyan]

allow me to insert it and then just mirror, rather than resliver?

Not sure what you mean - adding a disk to a mirror requires that it be resilvered. You could split the mirror using the CLI but that has pitfalls, since the FreeNAS middleware wouldn't know about it.

The simplest thing is probably to save your config, then do a clean install to whatever combination of boot devices you're trying to achieve, then restore your config.

 

[diskdiddler]

Not sure what you mean - adding a disk to a mirror requires that it be resilvered. You could split the mirror using the CLI but that has pitfalls, since the FreeNAS middleware wouldn't know about it.

The simplest thing is probably to save your config, then do a clean install to whatever combination of boot devices you're trying to achieve, then restore your config.

https://forums.freenas.org/index.ph...ng-usb-key-for-boot-volume.44810/#post-301729
Eric seems to indicate I can just "re-mirror" the boot array, so it does a 1:1 copy (Faster?) and makes the key bootable? (I damn well hope so)

The resliver process was looking like 70+ hours at 34kb/s something is totally wrong with that.

 

[Robert Trevellyan]

Eric seems to indicate I can just "re-mirror" the boot array, so it does a 1:1 copy (Faster?)

It's still a resilver.

The resliver process was looking like 70+ hours at 34kb/s something is totally wrong with that.

This may indicate a problem with one of more of your boot devices.

 

[Stux]

You can detach the offlined drive, which will remove the device from the pool.

I *think* you do:

zpool detach freenas-boot /dev/da2p2

http://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6qvl/index.html

You can detach a drive from a 3-way mirror, but not from a two way mirror. To do that you need to split the mirror.

 

[diskdiddler]

You can detach the offlined drive, which will remove the device from the pool.

I *think* you do:

zpool detach freenas-boot /dev/da2p2

http://docs.oracle.com/cd/E19253-01/819-5461/6n7ht6qvl/index.html

You can detach a drive from a 3-way mirror, but not from a two way mirror. To do that you need to split the mirror.

Thank you, this seemed to work fine, it's now at least not auto-reslivering.
I have removed the faulty key, re-inserted the replacement.

 

[diskdiddler]

It's still a resilver.

This may indicate a problem with one of more of your boot devices.

Thanks Robert,

I've done a replace using the in built menu, it's 'reslivering' again at 35kb/s
I thought this might be due to me initially doing it with 3 keys in at a time, rather than 2 - straining the USB controller?


Something is vastly broken with this process, when I first did it, it was fine. I'm convinced the superior method is to just add a new mirror disk and break off the original. I don't know how to do this.

Can someone confirm my suspicion? Because that's what several others have told me at this point. I can only assume it's straining / damaging the USB keys in some capacity.

Regardless, I'm just going to let it finish.

 

[diskdiddler]

This is progressively getting slower, it is now down to 22.6, 22.5, 22.4KB/s
These are brand name USB3 sticks which have worked well for a long time.
The USB controller is unlikely to be the problem, there's only 2 sticks in there now.

When I do a scrub on the sticks it works around 4 or 10MB/s if I recall - once they are in sync and online.
Very convinced resliver process is not ideally coded for USB media at this point?

 

[Robert Trevellyan]

resliver process is not ideally coded for USB media at this point?

It's possible that write amplification on low cost NAND is hurting performance in this situation. Scrubs only read, unless corruption is detected.

 

[diskdiddler]

I should've updated the the thread, it eventually picked up pace to around 600KB/s - so it's finished successfully.
Machine is behaving fine now, I took the "bad" USB stick and put it in my PC and ran a surface test on it and a h2tesw, no errors. So I'm going to keep using it as a regular, read only installer USB for Windows 7 for example.