mistermanko
Guru
- Joined
- Jan 27, 2020
- Messages
- 577
It has been a long, rough ride but I think I found the culprit, more at the the end of the post.
Edit 19th of March:
After switching the mainboard which turned out not to be the solution, I switched to another cpu which in result was the final solution of all checksum errors. In the end, I had to buy a whole new set of hardware to find the error in the CPU. I have no idea if the CPU was having a hardware fault or if just FreeBSD had some compatibility issues with the 1230v3, I doubt it though. Because all hardware tests, be it memtest or benchmarking tools did not show any issues with the CPU it was impossible to confirm, but switching to a 1245v3 brought me peace. I could activate all cores and ran installs and updates which went all fine. Maybe I was really unfortunate, but it was expensive af to be honest to buy two sets of mobo, memory and mainboard just to find out the cpu had issues.
I decided to re-write this Post, as it did not provide enough information for experts to get a look at.
Further down might be a hint to a bug in FreeNAS 11.3, stay tuned!
What I intended to do?
I build a FreeNAS system with specs as follows:
PSU: be quiet! Pure Power 11 500W 80+ Gold
MOBO: Supermicro X10SLL-F
RAM: 16 GB ECC RAM Samsung (2x M391B5273DH0; 2x M391B5173QH0)
CPU: Intel Xeon E3-1230v3
BOOT-POOL:2x WD Green SSD SATA 240GB big no-no as I've learned
STORAGE-POOL: 6x 4TB WD RED (RAIDZ2)
HBA: LSISAS2008 IT-mode (x-flashed Dell H310)
I tried to install FreeNAS 11.3 via a USB 3.0 thumb drive and failed miserably.
Where did I fail?
When booting the installer it stops at ChecksumFailException: ...11.3-RELEASE-....tgz has invalid checksum, see here:
What did I try to fix this?
here's why I think it is a bug:
So where do we go from here? File a bug report? Re-sell my hardware and go on with my life?
######
EDIT on 28th of February: After almost an entire month and two tickets in the bugtracker, I could confirm the failure is in my hardware.
It's either the CPU or the Motherboard or a combination of them both.
I could reproduce the issue successfully and I have now a reliable solution to prevent the issue.
Vladimir over at jira was helping me out to find that my CPU had trouble with cryptographic functions, which caused a whole bunch of weird issues on FreeNAS.
In the history of events I eventually could get a finished install of 11.3 up and running but was soon overwhelmed with frequent error messages which were all over the place.
The email-alert service was spamming me with nonsensical alerts, the GUI felt janky and here and there was stuff not functioning as expected.
So I started to turn-off CPU features in the BIOS and run the installer over and over again to check if it interferes with the checksum validation of the base-os.tgz.
First I suspected AES-IN to misbehave but that wasn't the case. I went on with VT-x, then HT, then power saving and so on. I ended up disabling all CPU features. Still, the installer wouldn't perform a complete install of FN reliably. As mentioned further up, it sometimes finished, sometimes it did not. So all whats left was disabling CPU-cores.
...well and what can I say...it hit me pretty hard.
Suddenly the installer finished not once, not twice but five times in a row installing FreeNAS without any interruptions. I rebooted, gparted the disk and installed again and every time it worked.
I mean wtf?
So I double checked, enabled all cores again and here we are again fellow checksum error.
I disabled all but one core - installer worked like a charm, I rebooted and confirmed with a second installation run.
I went on with two cores - installer worked just perfect, again confirmed it a second time.
I enabled all but one core - it just worked, multiple times confirmed
I enabled all cores - boom, checksum error
Seriously, wtf?
Ok, now I enabled all the features again but just went single core. It worked
all features + duo core. check
all features + three cores. check
all features + all cores. NOPEEE! checksum errors again
...I'm tripping...
I went on with trying different FreeNAS isos. When I started this thread I had tested multiple iso of different releases from the download archive. I had trouble installing 11.3, 11.2 U7, U6, U2, only 9.10 would work - god knows why. Now all of the releases worked like a charme. No checksum errors, no aborts, nothing, as long as I deactivate one core in the BIOS.
I even tried updating from 11.3 to 11.3 U1 via the webGUI and it worked out well.
I'm pretty baffled, this was all not anticipated when I started to build my first FreeNAS system.
Thanks to all who have helped or tried to help!
all the best!
#mm
Edit 19th of March:
After switching the mainboard which turned out not to be the solution, I switched to another cpu which in result was the final solution of all checksum errors. In the end, I had to buy a whole new set of hardware to find the error in the CPU. I have no idea if the CPU was having a hardware fault or if just FreeBSD had some compatibility issues with the 1230v3, I doubt it though. Because all hardware tests, be it memtest or benchmarking tools did not show any issues with the CPU it was impossible to confirm, but switching to a 1245v3 brought me peace. I could activate all cores and ran installs and updates which went all fine. Maybe I was really unfortunate, but it was expensive af to be honest to buy two sets of mobo, memory and mainboard just to find out the cpu had issues.
Further down might be a hint to a bug in FreeNAS 11.3, stay tuned!
What I intended to do?
I build a FreeNAS system with specs as follows:
PSU: be quiet! Pure Power 11 500W 80+ Gold
MOBO: Supermicro X10SLL-F
RAM: 16 GB ECC RAM Samsung (2x M391B5273DH0; 2x M391B5173QH0)
CPU: Intel Xeon E3-1230v3
BOOT-POOL:
STORAGE-POOL: 6x 4TB WD RED (RAIDZ2)
HBA: LSISAS2008 IT-mode (x-flashed Dell H310)
I tried to install FreeNAS 11.3 via a USB 3.0 thumb drive and failed miserably.
Where did I fail?
When booting the installer it stops at ChecksumFailException: ...11.3-RELEASE-....tgz has invalid checksum, see here:
What did I try to fix this?
- tried different USB-Sticks (2x 3.0, 1x 2.0)
- tried installing via Legacy and via UEFI
- tried installing on different boot devices (2 x WD Green 240G drives, newly bought; the USB sticks from above; a 320G 2.5 spinning rust)
- tried installing via IPMI virtual storage
- of course did I download and checksum verify the .iso several times over the last 3 days
- set the BIOS to defaults, re-arranged the boot device list, checked the manual for UEFI/Legacy/boot settings, verified that I had the latest FW/BIOS installed (Supermicro FW rev 01.35; BIOS ver 3.2)
- studied the gpart man page and re-formatted my thumb drives and disks many many times
- Every possible boot device mentioned above, got installed successfully with a second system (MSI b450 gaming).
- Version 9.10.2-U6.iso installed successfully via one of the USB drives onto the 320G HDD on the Supermicro system.
- /u/cassato on reddit had the same problem and fixed it by de-attaching the USB 3.0 header on his board
- @cyberfly got the same error while updating from 11.2
- @JohnnyAlpha got the same error and @cfs decided to use CD-ROM out of frustation
######
EDIT on 28th of February: After almost an entire month and two tickets in the bugtracker, I could confirm the failure is in my hardware.
It's either the CPU or the Motherboard or a combination of them both.
I could reproduce the issue successfully and I have now a reliable solution to prevent the issue.
Vladimir over at jira was helping me out to find that my CPU had trouble with cryptographic functions, which caused a whole bunch of weird issues on FreeNAS.
In the history of events I eventually could get a finished install of 11.3 up and running but was soon overwhelmed with frequent error messages which were all over the place.
The email-alert service was spamming me with nonsensical alerts, the GUI felt janky and here and there was stuff not functioning as expected.
So I started to turn-off CPU features in the BIOS and run the installer over and over again to check if it interferes with the checksum validation of the base-os.tgz.
First I suspected AES-IN to misbehave but that wasn't the case. I went on with VT-x, then HT, then power saving and so on. I ended up disabling all CPU features. Still, the installer wouldn't perform a complete install of FN reliably. As mentioned further up, it sometimes finished, sometimes it did not. So all whats left was disabling CPU-cores.
...well and what can I say...it hit me pretty hard.
Suddenly the installer finished not once, not twice but five times in a row installing FreeNAS without any interruptions. I rebooted, gparted the disk and installed again and every time it worked.
I mean wtf?
So I double checked, enabled all cores again and here we are again fellow checksum error.
I disabled all but one core - installer worked like a charm, I rebooted and confirmed with a second installation run.
I went on with two cores - installer worked just perfect, again confirmed it a second time.
I enabled all but one core - it just worked, multiple times confirmed
I enabled all cores - boom, checksum error
Seriously, wtf?
Ok, now I enabled all the features again but just went single core. It worked
all features + duo core. check
all features + three cores. check
all features + all cores. NOPEEE! checksum errors again
...I'm tripping...
I went on with trying different FreeNAS isos. When I started this thread I had tested multiple iso of different releases from the download archive. I had trouble installing 11.3, 11.2 U7, U6, U2, only 9.10 would work - god knows why. Now all of the releases worked like a charme. No checksum errors, no aborts, nothing, as long as I deactivate one core in the BIOS.
I even tried updating from 11.3 to 11.3 U1 via the webGUI and it worked out well.
I'm pretty baffled, this was all not anticipated when I started to build my first FreeNAS system.
Thanks to all who have helped or tried to help!
all the best!
#mm
Last edited: