Hey all, to preface I'm a definite FreeBSD newb, but I've recently run into a very perplexing issue.
My current hardware:
Things were working fantastic until I added the two WD2002FFSX drive mirror pool to the onboard SATA headers. This pool exists purely for a friend who wanted to try, for free, what an offsite backup solution could look like. When initially installed, drives sat for ~2 weeks as he sorted out what he wanted to do (maybe a jail, blah blah). I finally made the executive decision to just put a vm on it and thats when I started running in to issues.
Whenever I would go to install Ubuntu server, my entire server would eventually lock up and become entirely unresponsive while the logs were just chock full of:
I have tried the following:
In a final attempt of "wtf", I un-plugged two of the hot spares connected to the LSI HBA so I could connect this pool. Lo and behold, everything worked. No errors, Ubuntu server installed and is entirely happy.
I hadn't considered the onboard SATA may be at fault before as the boot pool is connected to it as well and I've had zero issues out of the boot pool. The only configuration difference between the two sets is that the two SSD's are demarcated as "Solid State Drives" in the BIOS, whereas the WD's are marked as "Hard Drives". I am hesitant to change this as I can't really find concrete data on what this value actually does, other than its related to the Intel PCH spec.
Anyway...I would really prefer to not have to buy another HBA. According to this https://www.supermicro.com/support/resources/OS/C236.cfm, this board is no longer "checkmarked" as of 11.1. Being very new to FreeBSD as a whole, I tried to google for any bc issues with 11.1+ that might cause this, but found nothing relevant.
Any insight or help would be appreciated.
My current hardware:
Code:
- Xeon E3-1225
- SuperMicro X11SAE-O
- 64GB Kingston KVR24E17D8/16
- LSI 9305-16i
- Total of 21 drives in the following topology:
- LSI:
- 12x WD2002FFSX (two pools)
- 3x WD2005FBYZ (hot spares for pools
- 1x SA400S37/120G (secondary pool l2arc)
- Onboard SATA:
- 2x SA400S37/120G (boot pool mirror)
- 2x WD2002FFSX (mirror pool)
- Onboard M.2:
- 1x WDS250G3X0C (primary pool l2arc)
Things were working fantastic until I added the two WD2002FFSX drive mirror pool to the onboard SATA headers. This pool exists purely for a friend who wanted to try, for free, what an offsite backup solution could look like. When initially installed, drives sat for ~2 weeks as he sorted out what he wanted to do (maybe a jail, blah blah). I finally made the executive decision to just put a vm on it and thats when I started running in to issues.
Whenever I would go to install Ubuntu server, my entire server would eventually lock up and become entirely unresponsive while the logs were just chock full of:
Code:
Feb 11 22:06:23 leviathan (ada3:ahcich6:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 c0 d5 51 40 1a 00 00 00 00 00 Feb 11 22:06:23 leviathan (ada3:ahcich6:0:0:0): CAM status: ATA Status Error Feb 11 22:06:23 leviathan (ada3:ahcich6:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF ) Feb 11 22:06:23 leviathan (ada3:ahcich6:0:0:0): RES: 41 10 c0 d5 51 00 1a 00 00 00 00 Feb 11 22:06:23 leviathan (ada3:ahcich6:0:0:0): Retrying command Feb 11 22:06:33 leviathan (ada2:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 e8 98 3e 41 40 18 00 00 00 00 00 Feb 11 22:06:33 leviathan (ada2:ahcich5:0:0:0): CAM status: ATA Status Error Feb 11 22:06:33 leviathan (ada2:ahcich5:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF ) Feb 11 22:06:33 leviathan (ada2:ahcich5:0:0:0): RES: 41 10 98 3e 41 00 18 00 00 00 00 Feb 11 22:06:33 leviathan (ada2:ahcich5:0:0:0): Retrying command Feb 11 22:06:46 leviathan (ada3:ahcich6:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 f0 10 6f 41 40 18 00 00 00 00 00 Feb 11 22:06:46 leviathan (ada3:ahcich6:0:0:0): CAM status: ATA Status Error Feb 11 22:06:46 leviathan (ada3:ahcich6:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF ) Feb 11 22:06:46 leviathan (ada3:ahcich6:0:0:0): RES: 41 10 10 6f 41 00 18 00 00 00 00 Feb 11 22:06:46 leviathan (ada3:ahcich6:0:0:0): Retrying command Feb 11 22:07:11 leviathan (ada2:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 20 28 5a 40 40 38 00 00 00 00 00 Feb 11 22:07:11 leviathan (ada2:ahcich5:0:0:0): CAM status: ATA Status Error Feb 11 22:07:11 leviathan (ada2:ahcich5:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF ) Feb 11 22:07:11 leviathan (ada2:ahcich5:0:0:0): RES: 41 10 28 5a 40 00 38 00 00 00 00 Feb 11 22:07:11 leviathan (ada2:ahcich5:0:0:0): Retrying command Feb 11 22:09:56 leviathan (ada2:ahcich5:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 10 20 c3 42 40 0e 00 00 00 00 00 Feb 11 22:09:56 leviathan (ada2:ahcich5:0:0:0): CAM status: ATA Status Error Feb 11 22:09:56 leviathan (ada2:ahcich5:0:0:0): ATA status: 41 (DRDY ERR), error: 10 (IDNF ) Feb 11 22:09:56 leviathan (ada2:ahcich5:0:0:0): RES: 41 10 20 c3 42 00 0e 00 00 00 00 Feb 11 22:09:56 leviathan (ada2:ahcich5:0:0:0): Retrying command
I have tried the following:
- Different cables
- Different headers on the board
- Each drive individually
- Reset MB to optimized defaults
- Updated MB BIOS
- I originally set it in AHCI mode, out of desperation tried RAID
In a final attempt of "wtf", I un-plugged two of the hot spares connected to the LSI HBA so I could connect this pool. Lo and behold, everything worked. No errors, Ubuntu server installed and is entirely happy.
I hadn't considered the onboard SATA may be at fault before as the boot pool is connected to it as well and I've had zero issues out of the boot pool. The only configuration difference between the two sets is that the two SSD's are demarcated as "Solid State Drives" in the BIOS, whereas the WD's are marked as "Hard Drives". I am hesitant to change this as I can't really find concrete data on what this value actually does, other than its related to the Intel PCH spec.
Anyway...I would really prefer to not have to buy another HBA. According to this https://www.supermicro.com/support/resources/OS/C236.cfm, this board is no longer "checkmarked" as of 11.1. Being very new to FreeBSD as a whole, I tried to google for any bc issues with 11.1+ that might cause this, but found nothing relevant.
Any insight or help would be appreciated.
