In short: the closest PCIe slot to the CPU is special and may not work with some -or any!- SAS/SATA Controllers for reasons not to do with the controllers.
The tale of woe and misery:
I was recently forced by circumstance to build my fourth unique FreeNAS server to suit some special conditions. Having some history, I decided to do this one all spit-and-polish correctly. I had been convinced by repeated admonitions that although my previous three servers on ASUS motherboards with Athlon II processors have worked perfectly in heavy use, that there was some unreliability lurking in there and only a Supermicro motherboard would keep my data safe. So off go the orders for:
X9SCL-F-O
Xeon E3
16GB Kingston unbuffered ECC
7 - 3TB disks of various brands to spread around the sudden death syndromes
Fractal Node 804 case
All pieces dribbled in and were duly buckled into the case. The motherboard and processor were longer in shipping, so I did a test install to be sure the disks were actually connected, recognized, working, and began their burn in with an earlier incarnation of my first FreeNAS, an ASUS with an Athlon II. That particular motherboard had only four SATA ports, so I used an IOCrest/Syba four-port based on the SiI313x series, which had been in steady use until it was replaced in an upgrade to a working system with the M1015 I was convinced was the only salvation for my data, too. I figured it would be safe to exercise the disk tests, as it had worked flawlessly for a year before being retired. All disks worked properly in a 7-disk setup.
When the Supermicro and Xeon arrived, I mounted the CPU and cooler, replaced the motherboard in the case, and populated up the RAM. I also used the Syba controller, as the new, and formidable, Supermicro only had six SATA ports on the board, and I didn't want to part with another $100 for one more SATA port.
All disks recognized in the bios, all reporting correct, but when the FreeNAS boot was done, only six disks showed up. Check power and data cables. Check again. No, really, really check them out again. Swap SATA cables. Only six. Finally in checking, I found that the disk attached to the Syba controller was not showing up. AHA! It's that pesky, dirt-for-brains Syba controller that's not working. Hmmm. But it works flawlessly in the other running system. OK, swap it anyway for the two-port ASM1061 controller. That one doesn't work either. Neither did an old SiI3132 card.
I've spent about 20 hours now scouring for info on FreeNAS, SATA controllers, FreeBSD compatibility, Supermicro motherboard compatibility, cursing at the thought that only Supermicro motherboards work with FreeNAS and only M1015/SAS2008 SATA controllers can be used... and then I came on this in Amazon:
==================
This review is from: Syba PCI Express SATA II 4 x Ports RAID Controller Card SY-PEX40008 (Personal Computers)
Before you buy this card you should understand the limitations of the design.
It uses a SiI3124 which supports four SATA 2.0 ports and ...
If you are only going to connect one mechanical hard drive (roughly 150 MB/s)
then there won't be a problem...
I got this card working with FreeNAS 9.1.1 but not without some struggle. I am
using a SuperMicro X9SCM-F motherboard which has four PCIe slots and I first
tried it in slot 7 (the slot closest to the CPU). I could see the option ROM
running at boot (and enumerating the drives) but I couldn't get the OS to
recognize the card. I booted from a FreeDOS image and tried updating the BIOS
but UPDFLASH.EXE reported that no controller was present. After moving the card
to slot 4 I was able to update the BIOS and the controller and drives were
recognized in FreeNAS. I'm not sure what the problem was with slot 7 but it may
have something to do with the fact that slot 4 is routed through the platform
controller hub (C204) while slot 7 connects directly to the CPU.
=================
Uh, wait. I put my SATA controller on the PCIe nearest the CPU, too. There are only four, but I suspect that the nearest one is that "slot 7" logically. I swapped the SATA controller to a different port, and now, all is well. Not only is all well, but every SATA controller I have of any vintage works as long as it's not in that slot.
It might be as simple as that PCIe slot is haunted. I know it's cursed. I cursed it. :)
The moral is that there are differences in the PCIe bus slots. They're not all the same. And apparently they're especially different in the Intel server chip set and not so much in the AMD consumer boards, and possibly not even in the Intel consumer boards. Even a very little bit of this would make for internet legends about what controller boards work and what don't. I suspect that it could even be that some controllers work better or worse in favored/non-favored slots depending on the minutae of what is actually causing this discrepancy. That could well be a faster slot for graphics card use. I haven't dug all the way to the bare metal to find out why.
But I'm happy, not having to pay the $100 for the morally-correct M1015. :D
The tale of woe and misery:
I was recently forced by circumstance to build my fourth unique FreeNAS server to suit some special conditions. Having some history, I decided to do this one all spit-and-polish correctly. I had been convinced by repeated admonitions that although my previous three servers on ASUS motherboards with Athlon II processors have worked perfectly in heavy use, that there was some unreliability lurking in there and only a Supermicro motherboard would keep my data safe. So off go the orders for:
X9SCL-F-O
Xeon E3
16GB Kingston unbuffered ECC
7 - 3TB disks of various brands to spread around the sudden death syndromes
Fractal Node 804 case
All pieces dribbled in and were duly buckled into the case. The motherboard and processor were longer in shipping, so I did a test install to be sure the disks were actually connected, recognized, working, and began their burn in with an earlier incarnation of my first FreeNAS, an ASUS with an Athlon II. That particular motherboard had only four SATA ports, so I used an IOCrest/Syba four-port based on the SiI313x series, which had been in steady use until it was replaced in an upgrade to a working system with the M1015 I was convinced was the only salvation for my data, too. I figured it would be safe to exercise the disk tests, as it had worked flawlessly for a year before being retired. All disks worked properly in a 7-disk setup.
When the Supermicro and Xeon arrived, I mounted the CPU and cooler, replaced the motherboard in the case, and populated up the RAM. I also used the Syba controller, as the new, and formidable, Supermicro only had six SATA ports on the board, and I didn't want to part with another $100 for one more SATA port.
All disks recognized in the bios, all reporting correct, but when the FreeNAS boot was done, only six disks showed up. Check power and data cables. Check again. No, really, really check them out again. Swap SATA cables. Only six. Finally in checking, I found that the disk attached to the Syba controller was not showing up. AHA! It's that pesky, dirt-for-brains Syba controller that's not working. Hmmm. But it works flawlessly in the other running system. OK, swap it anyway for the two-port ASM1061 controller. That one doesn't work either. Neither did an old SiI3132 card.
I've spent about 20 hours now scouring for info on FreeNAS, SATA controllers, FreeBSD compatibility, Supermicro motherboard compatibility, cursing at the thought that only Supermicro motherboards work with FreeNAS and only M1015/SAS2008 SATA controllers can be used... and then I came on this in Amazon:
==================
This review is from: Syba PCI Express SATA II 4 x Ports RAID Controller Card SY-PEX40008 (Personal Computers)
Before you buy this card you should understand the limitations of the design.
It uses a SiI3124 which supports four SATA 2.0 ports and ...
If you are only going to connect one mechanical hard drive (roughly 150 MB/s)
then there won't be a problem...
I got this card working with FreeNAS 9.1.1 but not without some struggle. I am
using a SuperMicro X9SCM-F motherboard which has four PCIe slots and I first
tried it in slot 7 (the slot closest to the CPU). I could see the option ROM
running at boot (and enumerating the drives) but I couldn't get the OS to
recognize the card. I booted from a FreeDOS image and tried updating the BIOS
but UPDFLASH.EXE reported that no controller was present. After moving the card
to slot 4 I was able to update the BIOS and the controller and drives were
recognized in FreeNAS. I'm not sure what the problem was with slot 7 but it may
have something to do with the fact that slot 4 is routed through the platform
controller hub (C204) while slot 7 connects directly to the CPU.
=================
Uh, wait. I put my SATA controller on the PCIe nearest the CPU, too. There are only four, but I suspect that the nearest one is that "slot 7" logically. I swapped the SATA controller to a different port, and now, all is well. Not only is all well, but every SATA controller I have of any vintage works as long as it's not in that slot.
It might be as simple as that PCIe slot is haunted. I know it's cursed. I cursed it. :)
The moral is that there are differences in the PCIe bus slots. They're not all the same. And apparently they're especially different in the Intel server chip set and not so much in the AMD consumer boards, and possibly not even in the Intel consumer boards. Even a very little bit of this would make for internet legends about what controller boards work and what don't. I suspect that it could even be that some controllers work better or worse in favored/non-favored slots depending on the minutae of what is actually causing this discrepancy. That could well be a faster slot for graphics card use. I haven't dug all the way to the bare metal to find out why.
But I'm happy, not having to pay the $100 for the morally-correct M1015. :D
