Jeff Chen
Dabbler
- Joined
- May 28, 2018
- Messages
- 11
My FreeNAS complains about boot pool errors every time it scrubs itself. The checksum of one of the USB thumb drives would always become 1 and I'd have to go to the CLI to clean it up manually every week.
So with WD Green 120GB SSD becoming cheap enough (50 Canadian Rupees each tax incl.), I decided why not, and snatched two of them, hoping to make a mirrored SSD boot.
My plan was to detach one of the USB drives, attach one SSD, let it resilver, then replace the other USB drive with the other SSD. The USB drives are only 16GB, so I will have quite some space left, and I was gonna stripe them as swap. Beautiful, isn't it?
[Instant Failure]
And it crashed and burned.
The first SSD was in, it resilvered fairly quickly and I thought it's time to replace the 2nd USB drive, only to find checksum numbers growing on the SSD. I wasn't quite clear with what I was doing and switched out the USB drive anyway. I even went ahead and created the slices for the future swap partitions. And after the 2nd SSD was in, my boot pool was crippled beyond rescue.
Reinstall time it is then. I was "smart enough" to have made a latest config backup before switching out the first USB drive. So I downloaded the latest 11.1-U5 ISO, dded it into my other USB drive and started installing. It went on fine initially, but after rebooting from the SSD, all kinds of hell were let loose. The log was filled with OSError and things about client.py not happy and etc. When the WebUI was finally up, it asked me for a License Update. WHAT???
[Deep In the Mud]
So here comes the repeating process. I thought I made a botched USB install drive, so I ran sha256 to make sure the ISO was ok, and then used etcher to make sure things were verified after being transferred to USB drive. Installed again, a little bit of improvement and I was at the web UI, but errors were still being spewed out like crazy in the console. I imported the shiny new config, rebooted, only to find my system crippled. No plugin, no jail, no iSCSI target. Everything seems to require a re-saving to start working. In panic mode, I even rolled my jail dataset back to last night (no real loss), but nothing improved.
After a series of failures, I decided to try the old USB drives - at least they didn't give me hell back then - only annoyances. And they proved to be trust worthy. I was able to get everything back up and running in a few moments.
[Not Giving Up - Yet]
But I would not stop there. I thought maybe it was the motherboard SATA ports, or the SATA cables, so I switched them all, but to no avail. I thought it was the SATA power cable they were sharing with my fan hub, so I rearranged and isolated it from the drives - nothing.
Then, I decided to pull the USB drives and dd them into the SSDs. I was considering that maybe the installer was doing something wrong with the formatting process.
I was curious about my shiny new SSDs and even did some speed test before dding the data over, and they were doing 450MB/s read and 260MB/s write in a USB enclosure, connected to a USB hub - without any issue. Not bad!
The dd process was a breeze, and I got two booting SSDs within 10 minutes. Back into the FreeNAS box, they booted up fast and good, everything seemed to be running. I was like "Bingo!... Maybe?"
[Short Sweetness]
After playing with the Web UI a while, everything was so much more responsive, and nothing seemed to be going wrong. In the beginning, I scrubbed the boot volumes consecutively several times in a row, and no checksum error ever popped up. So I decided to proceed and reclaim the remaining space on those SSDs.
gpart wasn't happy with the partition table inconsistency and spat out a scary "corrupted" status, but it was easily fixed by the recover command. However, just was I was going to create the bigger partition, I noticed the infamous checksum numbers starting to grow symmetrically on the two SSDs. In a few moments, it grew into over a hundred and with the text "too many errors" next to them.
I rebooted the system and nothing seemed to be wrong in the console. I verified the installation and no error was found. However "zpool status -v" showed me some permanent errors in files and metadata. And the list kept growing slowly, around the infamous python 3.6 directory.
[A Man Should Know When To Give Up]
I simply could not tolerate my system running in this state. Who knows if the data corruption is real or not, or when it is going to make my system go haywire? So I pulled the SSDs out, and popped back in the USB drives. A false alarm per week is way better than a painful slow death at least.
My final thought is that probably these SSDs are having problem with trim, and deleting data it shouldn't. The consistency of data corruption surrounding specific files/directories could mean it's a bug in FreeNAS (FreeBSD?) or the WD SSD firmware (both very possible). Is it possible that trim and freebsd-boot partition type don't mix?
And for now, these SSDs will be serving as striped L2ARC for my main storage. I do run some iSCSI targets and NFS shares so it should help me with the tiny reads at least. They have been running for hours and no checksum errors at all.
<<<<<<<<<[TL;DR]>>>>>>>>>
So here's the summary of the fact I know:
As for why I wanted a SSD-based swap - my system hasn't been maxed out with RAM yet (because they are so damn expensive at the moment), and when it starts to use swap, the whole system sort of freezes up. I checked and it seemed that one of my raidz2 drives is always busier than the others, and maxing out to 100% whenever it gets heavy. But it's another topic and I'm still trying to decide what could be going wrong.
So with WD Green 120GB SSD becoming cheap enough (50 Canadian Rupees each tax incl.), I decided why not, and snatched two of them, hoping to make a mirrored SSD boot.
My plan was to detach one of the USB drives, attach one SSD, let it resilver, then replace the other USB drive with the other SSD. The USB drives are only 16GB, so I will have quite some space left, and I was gonna stripe them as swap. Beautiful, isn't it?
[Instant Failure]
And it crashed and burned.
The first SSD was in, it resilvered fairly quickly and I thought it's time to replace the 2nd USB drive, only to find checksum numbers growing on the SSD. I wasn't quite clear with what I was doing and switched out the USB drive anyway. I even went ahead and created the slices for the future swap partitions. And after the 2nd SSD was in, my boot pool was crippled beyond rescue.
Reinstall time it is then. I was "smart enough" to have made a latest config backup before switching out the first USB drive. So I downloaded the latest 11.1-U5 ISO, dded it into my other USB drive and started installing. It went on fine initially, but after rebooting from the SSD, all kinds of hell were let loose. The log was filled with OSError and things about client.py not happy and etc. When the WebUI was finally up, it asked me for a License Update. WHAT???
[Deep In the Mud]
So here comes the repeating process. I thought I made a botched USB install drive, so I ran sha256 to make sure the ISO was ok, and then used etcher to make sure things were verified after being transferred to USB drive. Installed again, a little bit of improvement and I was at the web UI, but errors were still being spewed out like crazy in the console. I imported the shiny new config, rebooted, only to find my system crippled. No plugin, no jail, no iSCSI target. Everything seems to require a re-saving to start working. In panic mode, I even rolled my jail dataset back to last night (no real loss), but nothing improved.
After a series of failures, I decided to try the old USB drives - at least they didn't give me hell back then - only annoyances. And they proved to be trust worthy. I was able to get everything back up and running in a few moments.
[Not Giving Up - Yet]
But I would not stop there. I thought maybe it was the motherboard SATA ports, or the SATA cables, so I switched them all, but to no avail. I thought it was the SATA power cable they were sharing with my fan hub, so I rearranged and isolated it from the drives - nothing.
Then, I decided to pull the USB drives and dd them into the SSDs. I was considering that maybe the installer was doing something wrong with the formatting process.
I was curious about my shiny new SSDs and even did some speed test before dding the data over, and they were doing 450MB/s read and 260MB/s write in a USB enclosure, connected to a USB hub - without any issue. Not bad!
The dd process was a breeze, and I got two booting SSDs within 10 minutes. Back into the FreeNAS box, they booted up fast and good, everything seemed to be running. I was like "Bingo!... Maybe?"
[Short Sweetness]
After playing with the Web UI a while, everything was so much more responsive, and nothing seemed to be going wrong. In the beginning, I scrubbed the boot volumes consecutively several times in a row, and no checksum error ever popped up. So I decided to proceed and reclaim the remaining space on those SSDs.
gpart wasn't happy with the partition table inconsistency and spat out a scary "corrupted" status, but it was easily fixed by the recover command. However, just was I was going to create the bigger partition, I noticed the infamous checksum numbers starting to grow symmetrically on the two SSDs. In a few moments, it grew into over a hundred and with the text "too many errors" next to them.
I rebooted the system and nothing seemed to be wrong in the console. I verified the installation and no error was found. However "zpool status -v" showed me some permanent errors in files and metadata. And the list kept growing slowly, around the infamous python 3.6 directory.
[A Man Should Know When To Give Up]
I simply could not tolerate my system running in this state. Who knows if the data corruption is real or not, or when it is going to make my system go haywire? So I pulled the SSDs out, and popped back in the USB drives. A false alarm per week is way better than a painful slow death at least.
My final thought is that probably these SSDs are having problem with trim, and deleting data it shouldn't. The consistency of data corruption surrounding specific files/directories could mean it's a bug in FreeNAS (FreeBSD?) or the WD SSD firmware (both very possible). Is it possible that trim and freebsd-boot partition type don't mix?
And for now, these SSDs will be serving as striped L2ARC for my main storage. I do run some iSCSI targets and NFS shares so it should help me with the tiny reads at least. They have been running for hours and no checksum errors at all.
<<<<<<<<<[TL;DR]>>>>>>>>>
So here's the summary of the fact I know:
- SSDs and USB boot drives are fine
- Motherboard port, cable or power not cause of problem
- USB installer verified to be correct
- Installer formatting disk probably not the cause of problem
- Data on boot partition of SSD corrupt spontaneously over short span of time in the exact same directory
- When SSDs are used as L2ARC, nothing goes wrong for hours
As for why I wanted a SSD-based swap - my system hasn't been maxed out with RAM yet (because they are so damn expensive at the moment), and when it starts to use swap, the whole system sort of freezes up. I checked and it seemed that one of my raidz2 drives is always busier than the others, and maxing out to 100% whenever it gets heavy. But it's another topic and I'm still trying to decide what could be going wrong.
Attachments
Last edited:
