Boot FreeNAS from NVMe PCIe SSD?

[snicke]

Is it supported to boot FreeNAS from an NVMe PCIe SSD drive?

Supermicro states that UEFI mode is necessary for NVMe boot in general: http://www.supermicro.com/support/faqs/faq.cfm?faq=19283

Is UEFI boot supported by FreeNAS?

 

[jgreco]

Not that I'm aware of. Why would you waste an NVMe SSD in this way?

 

[snicke]

Why would you waste an NVMe SSD in this way?

Well, that's another (and of course valid) question.

What's the problem of booting from an NVMe SSD? Is it lack of support for UEFI boot mode in FreeNAS or something else?

 

[jgreco]

FreeBSD supports UEFI booting after version 10.1.

https://wiki.freebsd.org/UEFI

Hard to know for sure what will happen with FreeNAS, but basically the legacy boot mechanism works for all normal use cases, so I wouldn't guess anyone's put significant effort into making an odd edge case work, unless it was super-easy to make it work.

 

[Ericloewe]

If you must use a PCI-e SSD, an AHCI SSD should probably work.

 

[snicke]

If you must use a PCI-e SSD, an AHCI SSD should probably work.

Like this one then with the M.2 to PCIe adapter to fit it in an ordinary PCIe slot? http://www.kingston.com/us/company/press/article/7547

Yeah I know, 240 GB boot drive. WTF? Well, I simply cannot find any smaller (and cheaper) PCIe SSDs with ordinary (not M.2) PCIe interfaces (and AHCI support).

Use case? I plan an Intel Skylake build with Supermicro X11SSM-F board. I want to use all 8 SATA ports on the board for an 8 HDD pool. Hence, how to boot if currently no driver support in FreeNAS for the USB ports on the board? Well either I go with:

1. The X11SSH-F board and use an M.2 interfaced PCIe SSD (but probably need AHCI support right?), but the M.2 PCIe 3.0 port on the X11SSH-F board only has support for 2 PCIe lanes as I understand it (X2 not X4) so the X4 SSDs on the market will run at reduced speed (how reduced?) and inte won't be cheaper than the Kingston PCIe disk. Also that M.2 port feels less "future proof" since only 2 PCIe lanes are supported so is it a better choice to go with an X11SSM-F board which gives me another PCIe X4 slot compared to the X11SSH-F board? Better to have four PCIe 3.0 slots (two X4 and two X8) in the X11SSM-F compared to 3 PCIe 3.0 slots (one X4 and two X8) and one M.2 PCIe X2 slot (slower and not future proof?) on the X11SSH-F board even if M.2 interfaces seems kind of hot these days?
2. (Preferred option as of now) The X11SSM-M board and use an PCIe SSD with ACHI support like the one suggested above (http://www.kingston.com/us/company/press/article/7547). Even if it will be an expensive boot drive compared to an 32 GB SATA SSD.
3. Some adapter solution from an 32 GB SATA SSD to PCIe slot like: http://www.newegg.com/Product/Product.aspx?Item=9SIA1JM1194254 but I cannot find it locally, don't know how reliable it is and it adds cost too (adapter + SSD have to be bought).
4. Some cheap and reliable PCIe interfaced SATA/SAS controller card (e.g. IBM ServeRAID M1015). But it probably consumes more power constantly (up to 10 W?) and the price for the controller card + small SSD might be similar to the Kingston PCIe SSD mentioned above.
5. Do something else?

(And if I go with option 2 above maybe, maybe I could use the excessive 240 GB space to jails etc in the future (combination of boot and jails drive)? https://forums.freenas.org/index.ph...to-boot-drive-system-jails.26986/#post-256682)

 

[Ericloewe]

The X11SSH-F board and use an M.2 interfaced PCIe SSD (but probably need AHCI support right?)

PCI-e SSDs come in AHCI (the SATA protocol) and NVMe (new protocol, better for flash performance) flavors. AHCI should be a bit more compatible.

but the M.2 PCIe 3.0 port on the X11SSH-F board only has support for 2 PCIe lanes as I understand it (X2 not X4) so the X4 SSDs on the market will run at reduced speed (how reduced?)

2x 1GB/s (big B) is still much faster than SATA 6Gb/s. Not that you'd need it in a FreeNAS boot device.

that M.2 port feels less "future proof" since only 2 PCIe lanes are supported

I wouldn't say so.

so is it a better choice to go with an X11SSM-F board which gives me another PCIe X4 slot compared to the X11SSH-F board? Better to have four PCIe 3.0 slots (two X4 and two X8) in the X11SSM-F compared to 3 PCIe 3.0 slots (one X4 and two X8) and one M.2 PCIe X2 slot (slower and not future proof?) on the X11SSH-F board even if M.2 interfaces seems kind of hot these days?

It depends on what you plan to do with the M.2 slot.

Cute little boot device? Still cheaper than a SATA DOM, probably. Much faster and maybe more reliable, too.
High-end SLOG device? The extra bandwidth is always nice.

Some adapter solution from an 32 GB SATA SSD to PCIe slot like: http://www.newegg.com/Product/Product.aspx?Item=9SIA1JM1194254 but I cannot find it locally, don't know how reliable it is and it adds cost too (adapter + SSD have to be bought).

That's just a crummy SATA controller. Very few of those are worth touching with a ten foot pole.

Some cheap and reliable PCIe interfaced SATA/SAS controller card. But it probably consumes more power constantly (up to 10 W?) and the price for the controller card + small SSD might be similar to the Kingston PCIe SSD mentioned above.

Well, that is significantly more flexible. Honestly, if your only alternative is the Kingston SSD, this seems like a better option.

 

[Mirfster]

Not that I own a SuperMicro board, but in other boards you can set the USB Port Emulation. Is it not a viable option to set it as USB 2.0, use USB stick(s) and call it a day?

 

[snicke]

Not that I own a SuperMicro board, but in other boards you can set the USB Port Emulation. Is it not a viable option to set it as USB 2.0, use USB stick(s) and call it a day?

That's the problem. Supermicro X11 (Intel Skylake) boards only support XHCI for USB and that is currently not supported by FreeBSD 9. Hence no boot from USB. Look here and search for XHCI for more info: https://forums.freenas.org/index.php?threads/supermicro-x10-x11-motherboard-faq.23332/

 

[jde]

I would vote for your option No. 4 with an IBM M1015 and an inexpensive SATA SSD or SATA DOM. Its known to work and gives you a better upgrade path if you need to add more drives in the future. Honestly, I wouldn't worry about the extra 10 watts. The PCIe NVMe solution you are leaning towards is unproven and likely more expensive than the M1015/SATA SSD/DOM solution.

 

[jde]

What kind of 8 drive pool are you planning? If you are planning on a raidz2 or 3 setup with 2 or 3 TB drives, it may make more economic sense to knock it down to a 7 drive pool and bump up the drive capacites up by a TB. This would free up an on-board SATA port for a boot device. If you are planning a stripe of 4 mirrored vdevs, this suggestion obviously won't help.

 

[snicke]

I would vote for your option No. 4 with an IBM M1015 and an inexpensive SATA SSD or SATA DOM.

Yeah, I'm actually considering that exact option right now, i.e. with the IBM M1015 card. Though I'm quite sure option 2 will work perfectly fine. Note that the Kingston disk supports AHCI and should show up as an ordinary SATA disk.

Thanks for your input.

 

[snicke]

What kind of 8 drive pool are you planning? If you are planning on a raidz2 or 3 setup with 2 or 3 TB drives, it may make more economic sense to knock it down to a 7 drive pool and bump up the drive capacites up by a TB. This would free up an on-board SATA port for a boot device. If you are planning a stripe of 4 mirrored vdevs, this suggestion obviously won't help.

Sure you're not a mind reader? ;) Just what I was thinking when I saw your signature with 7 disks in RAIDZ3.

I'm considering a RAIDZ3 of 8x3TB disks since I like the extra redundancy. If I go down to 7 drives the parity overhead feels a little too high. 3TB disk are well priced now and if/when I need more storage it might be time to retire the 3TB disks and then it seems good to just double all drives in the RAIDZ3 to 6TB disks, which should have decreased considerably in price by then. That's how my thoughts go right now...

I will look more into the IBM M1015 card, e.g. power consumption. If I'm going that way I can easily add another SSD for jails and maybe system dataset (to be able to spin down the disks in the pool when not used).

 

[jde]

If there any possibility you might hit the 80% recommended fill limit within the next 2-3 years I would even more strongly recommend going the M1015 route. As you are aware, with all 8 on-board SATA ports filled with a raidz3 setup, your only option to increase pool size (short of later buying an M1015 or similar) is to replace all 8 drives. If you buy the M1015 now and you start running short of pool space, its trivial to add an inexpensive new pool (i.e. 2 drive mirror) for data that doesn't need raidz3 redundancy (i.e. backups of other machines on your network).

 

[jgreco]

You might be better off spending a few extra bucks now on 4TB drives; a 7 drive RAIDZ3 pool of 3TB drives gets you only 12TB, whereas the same thing out of 4TB drives gets you 16TB.

 

[snicke]

@jgreco True when considering 7 drives. Better situation with 8 drives however.

Still into 8 drives in RaidZ3 and the extra IBM controller card for boot drive and maybe additional SSD for jails. But 4 TB drives could be a good trade off anyway. Will do some price calculations on that. Maybe double to 8 TB drives (instead of 6 TB) if/when more storage is needed. Thanks for the input.

Skickat från min SM-G925F via Tapatalk

 

[jgreco]

Yes, be sure to consider that there's a certain amount of value being lost when you actually replace drives. Especially if you look at the total cost of the NAS when you determine the cost per TB, which most people seem to forget to do, the value derived point is rarely 3TB disks.

 

[joeschmuck]

I would definitely go with a IBM ServeRAID M1015 option since you are purchasing such a new motherboard and might run into difficulties. As for drive size and future expansion, well it depends on what uses you have planned for this device, both now and down the road, including the day dreams (I like to dream about if I had no money limits and I could build up my ideal NAS).

 

[gpsguy]

Dreaming how you'd spend the money if you won the Powerball lottery?

I like to dream about if I had no money limits and I could build up my ideal NAS.

 

[joeschmuck]

Dreaming how you'd spend the money if you won the Powerball lottery?

Trust me, I did a lot of that already, but I think we all do. I might even upgrade my system from that AMD CPU to a nice Xeon CPU and Supermicro MB.