NVMe hot-swap

Status
Not open for further replies.

jixam

Dabbler
Joined
May 1, 2015
Messages
47
We want to build an NVMe-based MySQL server. That one will most likely end up running ZFS on FreeBSD. However, we prefer hardware parts that can be reused for a FreeNAS build, so I take the liberty of asking in here.

The hope is to get hot-swap NVMe. Is that feature actually supported in FreeNAS?

I see that SuperMicro has a number of NVMe hot-swap capable servers. Apparently they all include 8+ additional drives with a SAS3 backplane and I believe that SAS3 is a poor fit for FreeNAS.

So, do we even have any good options for such a build?
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
http://www.supermicro.nl/products/nfo/Ultra.cfm

Those would offer 4x NVMe/SAS hotswap bays in 2U. You can use 6Gbps HBAs since those Backplanes are direct-attached, you'd only need adapter cables from SFF-8087 HBA-side to SFF-8643 for 4 HDDs each. But by the time that one drops out of production the 12Gbps support should be stable as well.
 

jixam

Dabbler
Joined
May 1, 2015
Messages
47
Cool, so there is no issue (FreeNAS or hardware related) in mixing a SAS2 HBA with a SAS3 backplane, as long as I stay clear of the expander option?

It seems that the SYS-6028UX-TR4 will fit my needs. I then install two AOC-S2308L-L8E cards to support 12 SAS drives and an external connector for a future JBOD (our FreeNAS build), or I install an AOC-2UR8N4-I2XT to get the NVMe hot-swap support (our database build).

I hope that's right :smile:
 

zambanini

Patron
Joined
Sep 11, 2013
Messages
479
the sas backplane is an expander...the sm boards use the lsi expander chips
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
Most Supermicro chassis can be ordered with direct attached backplane as well as the expander versions, the Ultra lineup however does only and exclusively use direct-attached backplanes.
 

jixam

Dabbler
Joined
May 1, 2015
Messages
47
Most Supermicro chassis can be ordered with direct attached backplane as well as the expander versions, the Ultra lineup however does only and exclusively use direct-attached backplanes.

Huh? Several of the descriptions in your Ultra link say "SAS3 via Expander".
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
now that you're saying it, a whopping two out of 12 systems come with an Expander now. Right next to them are the systems without expander, otherwise unchanged.
 

jixam

Dabbler
Joined
May 1, 2015
Messages
47
Well, if one will make and the other will break a FreeNAS system, it does seem important to mention the exceptions.

Anyway, I now noticed the SYS-6028R-TDWNR server. It is not an Ultra but should save me about 600 Euros since it has NVMe built-in (and still has room for 2x HBA for the SAS drives).

I am left with my original question, though: does the NVMe support even work with FreeNAS (or just FreeBSD). The compatibility matrix implies "no", but I have a feeling that this is just because it has not been tested. I guess that quite few people are using this combination yet.

If I ask a Supermicro reseller, will he tell me whether it should plausibly work, or will he just quote the matrix?
 

marbus90

Guru
Joined
Aug 2, 2014
Messages
818
P3700 with FreeNAS is no issue.

Do you already have NICs? The Ultras with NVMe have several flavours of 10/40Gbps interfaces, your linked system only dual gigabit.
 

jixam

Dabbler
Joined
May 1, 2015
Messages
47
We are still on gigabit, so I figured we could retrofit a 10 Gbps interface, should we manage to upgrade the network before this server is retired.

If an Ultra is better for our needs, I don't mind paying a bit more. I just don't really see how it is, but I could easily be overlooking something.
 

jixam

Dabbler
Joined
May 1, 2015
Messages
47
The hope is to get hot-swap NVMe. Is that feature actually supported in FreeNAS?

Just to answer my own question, in case anyone finds it: FreeBSD does not currently support PCIe hot-plug, so while the 2.5-inch form factor makes NVMe replacement easier, it does not enable hot-swap at this time.
 
Status
Not open for further replies.
Top