With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS.
Work in progress, as usual.
History
By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop topologies and this kind of sucked. SATA and Serial Attached SCSI (SAS) evolved from those, using a serial bus and hub-and-spoke design.
Early SATA/150 and SATA/300 were a bit rough and had some issues, as did SAS 3Gbps. You probably want to avoid older controllers, cabling, expanders, etc. that do not support 6Gbps because some of it has "gotchas" in it. In particular a lot of it has 2TB size limitations. Most SAS 3Gbps hard drives are fine though.
Similarities, Differences, Interoperability
SAS and SATA operate at the same link speeds and use similar cabling. SAS normally operates at a higher differential voltage than SATA and can run over longer cabling.
SAS and SATA use different connectors on the drive. The SATA drive connector has a gap between the signal and power sections, which allows separate power and data cables to be easily connected. The SAS drive connector does not have a gap, and instead has a second set of pins on top. This second set of pins is the second (redundant) SAS port. There are pictures of the top and the bottom of the drive connector.
SATA drives can be attached to a SAS port. Electrically, the SAS port is designed to allow attachment of a SATA drive, and will automatically run at SATA-appropriate voltages. Physically, the SAS backplane connector has an area that will allow either the gapless SAS or the gapped SATA connector to fit. See picture of SAS backplane socket.
SAS drives are incompatible with SATA ports, however, and a SATA connector will not attach to an SAS drive. Don't try. The gap is there to block a SAS drive from being connected to typical SATA cabling, or to a SATA backplane socket.
When a SATA drive is attached to a SAS port, it is operated in a special mode using the Serial ATA Tunneling Protocol (STP).
SATA drives are inherently single-ported, meaning that they can only be attached to one thing at a time. SAS devices, however, are usually dual-ported. This means that, electrically, there are two ports on the single SAS connector. One is the primary and one is the secondary. The secondary port may be supported by a backplane or enclosure to allow the attachment of a second host, or to allow multiple paths back to a host for a high-availability configuration.
Some people use a special device called an interposer to take an inexpensive SATA drive and make it look like a nearline SAS drive (usually to get multipathing). Don't do this. They're crummy, just another thing to break.
The primary takeaway: You can connect SATA drives to an SAS port and it is expected to work. You cannot connect SAS drives to a SATA port. That absolutely won't work.
Cabling
As already noted, single SAS internal cables are virtually identical to SATA. The difference is that they can be longer. SATA is limited to 1 meter. It is therefore best to use cables less than 1 meter long if at all possible.
However, most SAS deployments involve larger numbers of disks, and SAS has some special connectors used to reduce wiring and aggregate lanes together.
For SAS 6Gbps, this is often the SFF8087 (internal, "Mini SAS") or SFF8088 (external). Four lanes gives you a total capacity of 24Gbps over a single SFF8087 connector. Some newer boards are using SFF8643 ("Mini SAS HD") for SAS 6Gbps.
For SAS 12Gbps, this is the SFF8643 (internal, "Mini SAS HD") and SFF8644 (external) connector. Again, four lanes gives you 48Gbps over a single SFF8643 connector. Now and then you will also see the SFF8643 used for 6Gbps, especially on dense small form factor mainboards, and it is also used for NVMe. Be warned.
A multilane connector may be broken into its four individual lanes using a breakout cable. For example, if you get an SAS HBA, it probably comes with one or two SFF8087's on it, but you may want to directly attach hard drives. A breakout cable allows this. This is a SFF8087-to-single-SAS breakout cable; this is a SFF8643-to-single-SAS breakout cable.
Also, in some scenarios, a mainboard may offer discrete SAS ports which you desire to aggregate into a multilane cable, and so reverse-breakout cables are available as well.
Internal connectors can be transformed into external connectors using an adapter plate. This allows you to create servers using storage in more than one chassis. This is "not for beginners" but the concepts aren't hard.
It is possible to mix 6Gbps and 12Gbps SAS. Just as with SATA, significant effort has been put into backwards compatibility.
SAS Ports
Some mainboards have SAS ports. These may be single-device ones that look like (and will work with) SATA, or they may have a multilane connector (SFF8087 or SFF8643). These usually work fine with FreeNAS if they are hooked up to something like an Intel PCH Storage Control Unit. Most of the rest of the time, you need to add an SAS Host Bus Adapter ("HBA"), which will typically give you eight lanes on two multilane connectors. Be aware that you should use an LSI HBA crossflashed to IT mode, which is discussed in this linked article in greater depth. Please do not try to use a RAID controller.
SAS Expanders
A SAS expander essentially takes a SAS multilane connection and allows the attachment of additional SAS devices. These devices all share in the available bandwidth of the SAS multilane connection. SAS expanders can be cascaded as well. In the following picture:
we see three SAS expanders. The first one only distributes to the second and third. The second and third each attach to hard disks. Modern expanders typically have enough channels that you wouldn't need to cascade them for just this small number of disks. A typical modern expander might have 36 lanes, allowing 24 disks, two upstream four lane host connections, and a downstream four lane connection to another expander.
There are advantages and disadvantages to expanders. A primary advantage is cabling simplicity: if you have a 24 drive chassis with a backplane that uses an expander, you need only a single SFF8087 to attach from the backplane to the HBA. The two main downsides are that those 24 drives then share the 24Gbps that's available on a SFF8087, and that in some cases some specific SATA disks have been known to not play nicely and have caused problems for other attached devices on a SAS expander.
As a matter of throughput, a typical modern hard drive can push 125-150MBytes/sec (that's about 1-1.25Gbps) so if you load up 24 disks * 1.25Gbps, you do exceed the 24Gbps that the multilane is capable of. This, however, assumes that you are doing sequential access to all drives simultaneously. That is unlikely at best.
The picture changes for SSD, and expanders may not be a good idea for use with large numbers of SSD's if you are expecting high throughput.
SAS expanders can come pre-installed on a backplane, or can be purchased as separate devices. The separate devices often come on what appears to be a PCIe card, but this is only to take advantage of mainboard power. An expander such as the Intel RES2SV240 may be attached anywhere convenient inside a chassis and powered via a Molex power plug. If you have a free PCIe slot, of course, that is a great place to put it too.
Supermicro backplanes (TQ, A, BE16, BE26)
Supermicro offers backplanes for many of their chassis in a variety of configurations.
The TQ option brings each individual bay out to an individual SAS connector. This is straightforward and nonthreatening to those who are unfamiliar with multilane. However, it is a bad idea to have twenty four individual cables to have to dig through if you suspect a bad cable, etc.
The A option is the best generalized option. It is the same as the TQ except that it brings groups of four bays out to a single SFF8087. The SFF8087 is a latching connector and is therefore substantially safer than the individual cables in the TQ. For a 24 drive chassis, then, there will be six SFF8087 connectors on the backplane. You must connect all of them to something, or the corresponding bays will be dead. You can attach them to three eight-port HBA's (such as three IBM ServeRAID M1015's) and this is a high performance configuration that allows full 6Gbps on all slots. You could also attach them to an SAS expander, but if so, why not just buy a backplane with an expander?
The BE16 (or 12Gbps BE1C) option brings out the attached bays as a single SFF8087. For a 12-drive SATA array, this is an ideal choice because there is no contention on the 24Gbps link and the cabling is stupid-simple. Very attractive option. For a 24-drive SATA array, I still think this is probably just fine because you're not likely to actually hit contention issues.
The BE26 (or 12Gbps BE2C) option adds a secondary expander onto the attached bays, making the SAS secondary ports available. This is useless on a SATA array, but if you're deploying SAS drives and you want the multipath capabilities, this is your beast.
External Shelves
External shelves fall into two general categories, ones with controllers and ones with expanders. Do not try to use one with a RAID controller built in. They'll just be problematic under ZFS. An external drive shelf that has an SAS expander in it, however, is very straightforward and may be attached in a manner similar to any other SAS expander.
Note that external shelves introduce a significant risk in the form of power catastrophes. If your shelf powers off but your server doesn't, this can be destructive to the pool.
Sidebands
SAS multilane cables may also include support for sideband signalling. This is a way for the backplane and the RAID controller, or mainboard, to indicate status, such as failed drive indication. This isn't generally useful in FreeNAS, which lacks software support for this murky and often arcane area of hardware design. For example, a RAID controller with sideband support and a compatible backplane can support features like "Identify Drive" or "Drive Fail" to identify a specific drive. In a reverse breakout cable scenario, four single SAS lanes from a mainboard plus an SGPIO header might connect to a single SFF8087. Discussed somewhat further at ftp://ftp.seagate.com/pub/sff/SFF-8448.PDF
Work in progress, as usual.
History
By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop topologies and this kind of sucked. SATA and Serial Attached SCSI (SAS) evolved from those, using a serial bus and hub-and-spoke design.
Early SATA/150 and SATA/300 were a bit rough and had some issues, as did SAS 3Gbps. You probably want to avoid older controllers, cabling, expanders, etc. that do not support 6Gbps because some of it has "gotchas" in it. In particular a lot of it has 2TB size limitations. Most SAS 3Gbps hard drives are fine though.
Similarities, Differences, Interoperability
SAS and SATA operate at the same link speeds and use similar cabling. SAS normally operates at a higher differential voltage than SATA and can run over longer cabling.
SAS and SATA use different connectors on the drive. The SATA drive connector has a gap between the signal and power sections, which allows separate power and data cables to be easily connected. The SAS drive connector does not have a gap, and instead has a second set of pins on top. This second set of pins is the second (redundant) SAS port. There are pictures of the top and the bottom of the drive connector.
SATA drives can be attached to a SAS port. Electrically, the SAS port is designed to allow attachment of a SATA drive, and will automatically run at SATA-appropriate voltages. Physically, the SAS backplane connector has an area that will allow either the gapless SAS or the gapped SATA connector to fit. See picture of SAS backplane socket.
SAS drives are incompatible with SATA ports, however, and a SATA connector will not attach to an SAS drive. Don't try. The gap is there to block a SAS drive from being connected to typical SATA cabling, or to a SATA backplane socket.
When a SATA drive is attached to a SAS port, it is operated in a special mode using the Serial ATA Tunneling Protocol (STP).
SATA drives are inherently single-ported, meaning that they can only be attached to one thing at a time. SAS devices, however, are usually dual-ported. This means that, electrically, there are two ports on the single SAS connector. One is the primary and one is the secondary. The secondary port may be supported by a backplane or enclosure to allow the attachment of a second host, or to allow multiple paths back to a host for a high-availability configuration.
Some people use a special device called an interposer to take an inexpensive SATA drive and make it look like a nearline SAS drive (usually to get multipathing). Don't do this. They're crummy, just another thing to break.
The primary takeaway: You can connect SATA drives to an SAS port and it is expected to work. You cannot connect SAS drives to a SATA port. That absolutely won't work.
Cabling
As already noted, single SAS internal cables are virtually identical to SATA. The difference is that they can be longer. SATA is limited to 1 meter. It is therefore best to use cables less than 1 meter long if at all possible.
However, most SAS deployments involve larger numbers of disks, and SAS has some special connectors used to reduce wiring and aggregate lanes together.
For SAS 6Gbps, this is often the SFF8087 (internal, "Mini SAS") or SFF8088 (external). Four lanes gives you a total capacity of 24Gbps over a single SFF8087 connector. Some newer boards are using SFF8643 ("Mini SAS HD") for SAS 6Gbps.
For SAS 12Gbps, this is the SFF8643 (internal, "Mini SAS HD") and SFF8644 (external) connector. Again, four lanes gives you 48Gbps over a single SFF8643 connector. Now and then you will also see the SFF8643 used for 6Gbps, especially on dense small form factor mainboards, and it is also used for NVMe. Be warned.
A multilane connector may be broken into its four individual lanes using a breakout cable. For example, if you get an SAS HBA, it probably comes with one or two SFF8087's on it, but you may want to directly attach hard drives. A breakout cable allows this. This is a SFF8087-to-single-SAS breakout cable; this is a SFF8643-to-single-SAS breakout cable.
Also, in some scenarios, a mainboard may offer discrete SAS ports which you desire to aggregate into a multilane cable, and so reverse-breakout cables are available as well.
Internal connectors can be transformed into external connectors using an adapter plate. This allows you to create servers using storage in more than one chassis. This is "not for beginners" but the concepts aren't hard.
It is possible to mix 6Gbps and 12Gbps SAS. Just as with SATA, significant effort has been put into backwards compatibility.
SAS Ports
Some mainboards have SAS ports. These may be single-device ones that look like (and will work with) SATA, or they may have a multilane connector (SFF8087 or SFF8643). These usually work fine with FreeNAS if they are hooked up to something like an Intel PCH Storage Control Unit. Most of the rest of the time, you need to add an SAS Host Bus Adapter ("HBA"), which will typically give you eight lanes on two multilane connectors. Be aware that you should use an LSI HBA crossflashed to IT mode, which is discussed in this linked article in greater depth. Please do not try to use a RAID controller.
SAS Expanders
A SAS expander essentially takes a SAS multilane connection and allows the attachment of additional SAS devices. These devices all share in the available bandwidth of the SAS multilane connection. SAS expanders can be cascaded as well. In the following picture:
we see three SAS expanders. The first one only distributes to the second and third. The second and third each attach to hard disks. Modern expanders typically have enough channels that you wouldn't need to cascade them for just this small number of disks. A typical modern expander might have 36 lanes, allowing 24 disks, two upstream four lane host connections, and a downstream four lane connection to another expander.
There are advantages and disadvantages to expanders. A primary advantage is cabling simplicity: if you have a 24 drive chassis with a backplane that uses an expander, you need only a single SFF8087 to attach from the backplane to the HBA. The two main downsides are that those 24 drives then share the 24Gbps that's available on a SFF8087, and that in some cases some specific SATA disks have been known to not play nicely and have caused problems for other attached devices on a SAS expander.
As a matter of throughput, a typical modern hard drive can push 125-150MBytes/sec (that's about 1-1.25Gbps) so if you load up 24 disks * 1.25Gbps, you do exceed the 24Gbps that the multilane is capable of. This, however, assumes that you are doing sequential access to all drives simultaneously. That is unlikely at best.
The picture changes for SSD, and expanders may not be a good idea for use with large numbers of SSD's if you are expecting high throughput.
SAS expanders can come pre-installed on a backplane, or can be purchased as separate devices. The separate devices often come on what appears to be a PCIe card, but this is only to take advantage of mainboard power. An expander such as the Intel RES2SV240 may be attached anywhere convenient inside a chassis and powered via a Molex power plug. If you have a free PCIe slot, of course, that is a great place to put it too.
Supermicro backplanes (TQ, A, BE16, BE26)
Supermicro offers backplanes for many of their chassis in a variety of configurations.
The TQ option brings each individual bay out to an individual SAS connector. This is straightforward and nonthreatening to those who are unfamiliar with multilane. However, it is a bad idea to have twenty four individual cables to have to dig through if you suspect a bad cable, etc.
The A option is the best generalized option. It is the same as the TQ except that it brings groups of four bays out to a single SFF8087. The SFF8087 is a latching connector and is therefore substantially safer than the individual cables in the TQ. For a 24 drive chassis, then, there will be six SFF8087 connectors on the backplane. You must connect all of them to something, or the corresponding bays will be dead. You can attach them to three eight-port HBA's (such as three IBM ServeRAID M1015's) and this is a high performance configuration that allows full 6Gbps on all slots. You could also attach them to an SAS expander, but if so, why not just buy a backplane with an expander?
The BE16 (or 12Gbps BE1C) option brings out the attached bays as a single SFF8087. For a 12-drive SATA array, this is an ideal choice because there is no contention on the 24Gbps link and the cabling is stupid-simple. Very attractive option. For a 24-drive SATA array, I still think this is probably just fine because you're not likely to actually hit contention issues.
The BE26 (or 12Gbps BE2C) option adds a secondary expander onto the attached bays, making the SAS secondary ports available. This is useless on a SATA array, but if you're deploying SAS drives and you want the multipath capabilities, this is your beast.
External Shelves
External shelves fall into two general categories, ones with controllers and ones with expanders. Do not try to use one with a RAID controller built in. They'll just be problematic under ZFS. An external drive shelf that has an SAS expander in it, however, is very straightforward and may be attached in a manner similar to any other SAS expander.
Note that external shelves introduce a significant risk in the form of power catastrophes. If your shelf powers off but your server doesn't, this can be destructive to the pool.
Sidebands
SAS multilane cables may also include support for sideband signalling. This is a way for the backplane and the RAID controller, or mainboard, to indicate status, such as failed drive indication. This isn't generally useful in FreeNAS, which lacks software support for this murky and often arcane area of hardware design. For example, a RAID controller with sideband support and a compatible backplane can support features like "Identify Drive" or "Drive Fail" to identify a specific drive. In a reverse breakout cable scenario, four single SAS lanes from a mainboard plus an SGPIO header might connect to a single SFF8087. Discussed somewhat further at ftp://ftp.seagate.com/pub/sff/SFF-8448.PDF
