Upgrade bug?

[dtom10]

Hello forum,

I'm having a problem which I currently blame on the upgrade component of freenas.

A little back story on this. I did not made a config backup and that doesn't hurt much but I've hit a problem.

Nas system went down due to power outage. It's a HP Microserver Gen8 with 16Gb of ram and 4 X 3Tb WD Red in a RAID-Z1 configuration.

Did a couple of upgrades and now it won't boot either of the upgrade boot options. It failed the wizard boot option dropping me in kdb on the first attempt. Second attempt worked.

Boot device is mirrored. 1 internal USB 16Gb SanDisk stick + 1 PNY microsd 16Gb card.

Boot volume is shown as healthy in WebUI. Mirror shows up as online. Able to boot default and wizard options in GRUB. Config looks stock in either successful start-ups. Installed several plugins and the jails are gone in the UI (assuming the config is gone, so the info on the installed jails as well).

Need instructions on how to debug further. This is a home system so I can run whatever steps are needed to collect data on the problem, if there is one.

Screenshots of what I've considered relevant are attached. Error on compression algorithm inherit not supported is shown for either upgrade boot options.

 

[dlavigne]

Sounds like the boot device is damaged. Do a fresh install of the latest STABLE on a new stick. You'll have to recreate your config if you don't have a backup of it.

 

[dtom10]

These are new devices, the stick and microsd card worked without problems, they still do. I was hoping to get some debug out of this. But yeah, nevermind. Thanks for the answer.

 

[cyberjock]

These are new devices, the stick and microsd card worked without problems, they still do. I was hoping to get some debug out of this. But yeah, nevermind. Thanks for the answer.

Did you read our manual? It says not to use SD cards and adapters. Why? Because they cause all sorts of problems.

 

[dtom10]

Yeah, no, it doesn't state that. It states you CAN use usb sticks, cf cards and ssds for boot devices. Surely, i'm not the only one using the embedded microsd card reader of this machine. And no, I didn't do any troubleshooting myself as I don't know where to start with FreeBSD. Jumping to the conclusion microsd is bad without any actual investigation is just... jumping.

System is up and running and boots without problems on the "default" boot option.

 

[cyberjock]

Ok, so you are right. It used to say not to use microsd cards and such as they are horribly unreliable for boot devices and especially ZFS. Someone took the message out of the manual. It was definitely in there before because *I* put it there. Kind of getting tired of all of the warnings we put in the manual being constantly removed... /sigh. I'm almost convinced I should just give up trying to help with the manual.

Anyway, regardless of what the manual says (or doesn't say), sd cards are a poor choice. The technology is not designed to handle random read or random write workloads (which is what the boot device and ZFS do). The decision was probably made to remove the warning in the manual and just not mention sd/microsd cards. So instead it says:

The FreeNAS® operating system is installed to at least one device that is separate from the storage disks. The device can be a USB stick, compact flash, or SSD. Technically, it can also be installed onto a hard drive, but this is discouraged as that drive will then become unavailable for data storage.

Technically, it doesn't say microsd card. In fact, people are having more and more problems with USB devices in general, so they are somewhat not recommended anymore either.

Anyway, microsd cards are a poor choice for a boot device. There is no fixing it because the problems are because of how microsd cards work. I can tell you that when I wrote that warning, I did it because my upgrades were failing consistently. Yet every test I could think of to run on the microsd card ran fine. Later we found out that the microsd card didn't have a good enough random-write speed to prevent the writes from timing out. Once writes are being discarded it just makes a big mess of everything.

 

[dtom10]

How the mirror is supposed to work then? The boot device is mirrored right now, and when the latest upgrade was applied as well. I've booted at least once with the upgrade since it's part of the upgrade process if I got that right. Not saying is a normal scenario. The server got shutdown due to a power failure, but the mirror's purpose is to help with this exact situation. Otherwise it's a bug or a worthless feature that doesn't work properly. Wouldn't that render some kind of investigation or is it something that I must pay for? For starters, I wouldn't pay for this kind of treatment. MicroSD is not reliable, it got taken out of the manual, some UFO landed. I don't really care what the reasons are. I asked help not for RCA, not for repairing my installation. I asked help for others with the same config to find while googling for an answer.

 

[Ericloewe]

Due to the insane amounts of legacy cruft present on a modern x64 system, you can only bootstrap from a single device. If that device fails before GRUB can load itself and the ZFS driver so that it can access both members of the mirror, the boot process will fail. Remove the faulty device and/or change them around in the boot order.

 

[dtom10]

Dmesg confirms the boot is done from the mirror as far as I know from what is shown. And this works right now on my system. The errors I've captured in the screenshots are after grub loads and a selection is made to boot. I'm stuck at kernel load from what i'm seeing. The kernel image fails to unpack and grub errors out.

Code:

Timecounter "TSC-low" frequency 1247194376 Hz quality 1000
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: usbus2 usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
ugen1.2: <HP> at usbus1
ukbd0: <Virtual Keyboard > on usbus1
kbd2 at ukbd0
ums0: <Virtual Mouse> on usbus1
ugen0.2: <vendor 0x8087> at usbus0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen2.2: <vendor 0x8087> at usbus2
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
Root mount waiting for: usbus2 usbus0
uhub3: 6 ports with 6 removable, self powered
uhub4: 6 ports with 6 removable, self powered
ugen2.3: <vendor 0x0424> at usbus2
uhub5: <vendor 0x0424 product 0x2660, class 9/0, rev 2.00/8.01, addr 3> on usbus2
Root mount waiting for: usbus2 usbus0
uhub5: 2 ports with 1 removable, self powered
ugen0.3: <SanDisk> at usbus0
umass0: <SanDisk Cruzer Force, class 0/0, rev 2.00/1.27, addr 3> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:7:0:-1: Attached to scbus7
da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
da0: <SanDisk Cruzer Force 1.27> Removable Direct Access SCSI-6 device
da0: Serial Number 4C530201200119121492
da0: 40.000MB/s transfers
da0: 14907MB (30529536 512 byte sectors: 255H 63S/T 1900C)
da0: quirks=0x2<NO_6_BYTE>
Root mount waiting for: usbus2
Root mount waiting for: usbus2
ugen2.4: <Generic> at usbus2
umass1: <Generic Ultra Fast Media Reader, class 0/0, rev 2.00/2.09, addr 4> on usbus2
umass1:  SCSI over Bulk-Only; quirks = 0x4100
umass1:8:1:-1: Attached to scbus8
Trying to mount root from zfs:freenas-boot/ROOT/Wizard-2015-03-21_15:17:44 []...
da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
da1: <HP iLO Internal SD-CARD 2.09> Fixed Direct Access SCSI-0 device
da1: Serial Number 000002660A01
da1: 40.000MB/s transfers
da1: 15240MB (31211520 512 byte sectors: 255H 63S/T 1942C)
da1: quirks=0x2<NO_6_BYTE>
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)

 

[Ericloewe]

Scrub the boot pool and see if errors pop up.

 

[dtom10]

No errors reported.

 

[rogerh]

Might be worth removing one of the boot devices and trying to boot from the other. Don't know if FreeNAS actually installs Grub on both (see post 8).

 

[Ericloewe]

Might be worth removing one of the boot devices and trying to boot from the other. Don't know if FreeNAS actually installs Grub on both (see post 8).

It should. And it doesn't seem to be the problem, which makes it weirder.

Is that latest environment (2015-03-30) not working correctly or is the problem intermittent and on all environments?

 

[dtom10]

The upgrade environments both failed two attempts each, wizard first booted in kdb and a second attempt booted fine (running), default environment worked each time. I will test by booting one single device and see if it produces a different symptom.

 

[rogerh]

I would propose doing an update without the SD card plugged in. If this works normally then Cyberjock's phenomenon of file operations timing out because of the slow SD card could be to blame.

 

[cyberjock]

Do a file verification on the device once its booted. If it passes that (and the zpool scrub already passed), then you've pretty much validated that things are timing out randomly and your solution seems pretty obvious.

I don't really care what the reasons are. I asked help not for RCA, not for repairing my installation. I asked help for others with the same config to find while googling for an answer.

If that's the attitude you want to take, fine. The manual doesn't say to use a microsd card. So you are doing your own thing, which we've never claimed to support. End of discussion. Would you like me to lock the thread now since we've clearly given you the answer to your problem (clearly not the answer you wanted to hear) and not some RCA?

 

[dtom10]

Do what you need to do. Sorry for wasting people's time with this thread.