Which SSD as boot device?

[fmiz]

To avoid issues you may want to take a new SSD with a controller which is not SilliconMotion over a used Intel.

Consumer SSDs are NOT suitable for SLOG. A dedicated SLOG is only useful if you do sync=always, but performance is always worse than sync=never (and living with the risk). If you do invest in a SLOG, go for an Optane drive or a Radian RMS-200.

From what I've found, it uses a Phison ps3111-s11 controller (the manufacturer does not specify anything). I fully understand it is not suitable, it is a cheap dramless ssd, the worst of the worst, I will never use sync=always, jgreco (or maybe someone else) already explained many times here how to use an SLOG correctly. But then, what does sync=standard do with bhyve, knowing that it is slower than sync=never, and which should I use? Sync=always without an SLOG with my setup does not make any sense, and sync=always with an SLOG without power loss protection still makes no sense. What I think I need to find out is what does sync=standard mean when zvols are used in bhyve.

 

[rvassar]

Instead of buying a 10€ usb flash drive and expect a failure in a few years, I will try to follow the "newer" recommendation and switch to a cheap SSD boot drive (with less than 30€). But I have no sata ports left in my system (x10slh-f with 6 4TB wd red in raidz2), so I've bought a cheap usb 3 to sata adapter I can find on amazon (made by sabrent, apparently).

I've run something like this. In fact, when my USB thumb drives started failing in odd ways, this was how I migrated. I booted off the remaining functioning USB thumb drive, added the SSD in a USB enclosure, and created a mirror pair. I then added a second SSD on a MB sata port and replaced the remaining thumb drive. Eventually I removed the USB adapter and moved the other drive to another sata port.

My first thought has been to use the SSD also as SLOG for bhyve vdevs. I tested with fio as in [3] and as expected, changing sync to disabled instead of standard has performance benefits (to no one's surprise?), yet it is unclear to me what sync=standard does in a VM context (how does zfs decide to do a sync or async write?). After reading [2] and the fio result, I expect that a SLOG device can make a difference, but I have not tried testing with a ramdisk. Another thing that I could do is just store vdevs for bhyve on the ssd.

There are some significant differences between a device capable of booting TrueNAS, and a device qualified for SLOG duty. Which isn't to say I'm advocating using inferior devices as boot devices. I'm just pointing out that the way TrueNAS is built, you can recover from boot device loss. You really want to review the resources tab above and understand what you are getting yourself into before implementing a SSD SLOG. Making poor hardware choices here can result in data loss. Loosing a boot device is more of an annoyance.

On the topic of how does ZFS decide to do sync or async writes... It's not ZFS that really chooses. The application performing the write initiates the process and may in fact demand it. When it opens a file handle it sets certain options and flags, is it creating a file, appending a file, etc... A couple of these flags (O_SYNC & O_DSYNC) mandate that the write() operation be performed as if a fsync() call was included before the write statement returns to the calling program. Essentially flushing the change to the hardware before the calling program can continue to execute. This is used by a variety of programs to ensure data consistency in transactions where there are separate failure domains and consistency is required. For example, the NFS file sharing protocol requires in the RFC specifications, that all writes be performed "O_SYNC". This is because the calling system is remote, and the server may crash or network access may be disrupted, or any number of other failures that might lead the calling system to assume the data was successfully saved and discard the transaction as complete. NFS is not the only example, SMTP, certain database transactions, some iSCSI operations, etc..., will set the O_SYNC flag and benefit from a fast SLOG. In your bhyve situation, the hypervisor will pass this requirement thru from the virtualized OS & application.

What I care mostly is avoiding the kind of failure I had with my current setup using a usb flash drive, everything else is just nice (and having an ssd with all that space unused looks like a waste to me).
Thanks for enduring to this point in this convoluted question.

Unused space in a flash drive can have benefits to lifespan and wear levelling. But... Don't allow yourself to form a blind spot to best practice on account of the lack of SATA ports on your system. USB is not a reliable storage bus. You most certainly do not want to attempt to implement a SLOG using any kind of USB solution, and I would argue against using your USB boot pool for bhyve storage as well. The neat thing about ZFS is it tracks pool devices by a kind of UUID. You can literally power down, and randomly swap the SATA connections on your existing pool right now. When TrueNAS imports the pool at boot, it will simply stitch the pool back together and run as if nothing happened. Find a way to add some more SATA ports using a supported PCIe card. You can then simply unplug your 4Tb RAIDz2 devices and plug them in to the additional ports. Adding some additional ports for your pool drives will free up your MB SATA connectors for boot SSD's, etc...

 

[fmiz]

[...]
There are some significant differences between a device capable of booting TrueNAS, and a device qualified for SLOG duty. Which isn't to say I'm advocating using inferior devices as boot devices. I'm just pointing out that the way TrueNAS is built, you can recover from boot device loss. You really want to review the resources tab above and understand what you are getting yourself into before implementing a SSD SLOG. Making poor hardware choices here can result in data loss. Loosing a boot device is more of an annoyance.

This is something I missed and that I really should have thought myself, thanks.

[...]
In your bhyve situation, the hypervisor will pass this requirement thru from the virtualized OS & application.

Is this possible thanks to virtio? This means that sync=standard has added value even in my current setup (without an unsafe SLOG).

Unused space in a flash drive can have benefits to lifespan and wear levelling. But... Don't allow yourself to form a blind spot to best practice on account of the lack of SATA ports on your system.
[...]

I have thought about this... I would like to add a PCIe card, but I have no space left for more 3.5" HDDs in my box, and I still have plenty of free space left. I tried looking up on ebay but from what I've found, it looks like I would need at least 70€ for an old SAS adapter, then start documenting a little about SAS to SATA cables. Maybe it would be more useful for future proofing my setup to buy a card with an external port to plug into an external sas enclosure. I have to say, I do not know many SAS adapter models (only the glorious M1015), maybe this is why I failed to find anything cheaper. I'm not trying to spend as little as possible, the issue is that I just need to replace a boot drive and everything else I'm adding on top is ...wasted, fortunately my usecase is very simple. But now at least I'm sure that buying an old intel 320 does not give me any advantage.

 

[rvassar]

I have thought about this... I would like to add a PCIe card, but I have no space left for more 3.5" HDDs in my box, and I still have plenty of free space left. I tried looking up on ebay but from what I've found, it looks like I would need at least 70€ for an old SAS adapter, then start documenting a little about SAS to SATA cables. Maybe it would be more useful for future proofing my setup to buy a card with an external port to plug into an external sas enclosure.

SAS & SATA are electrically interchangeable, they simply implement different protocols. Virtually all SAS controllers can talk to a SATA device, and a SATA controller will never talk to a SAS device. Physically, the SAS drives implement a plastic block between the power & data plugs to prevent a SAS drive from being inserted on a SATA only controller. (Enterprise NVMe devices then add pins at this location as well for further NVMe feature enablement.) I run a IT flashed Dell H310, which is a LSI/Broadcom variant. I bought SFF-8087 to SATA breakout cables for my devices. Example on Amazon

My 3.5 inch bays are all occupied, as are the 2.5" SSD slots. I added a couple 5-1/4" hotswap trays to external CD slots so I can do sneakernet backups, etc... I have considered the external SAS controller, but I figure I'll upgrade my drive capacity for the bulk storage before needing more spindles, and I'm more likely to move to 2.5" SSD's for VM's, which can substitute 6 in 1 in the 5-1/4 slots using a dock of some sort... Don't know if that would be of interest in your use case, but having a couple free ports always lends options like this.

I will caution that it is possible to find all manner of plain PCIe SATA boards. These are often implemented with Marvell chips, and have sketchy or non-existant drivers in TrueNAS. I'm not suggesting it's not possible, but you need to do your homework very carefully if you attempt that route.

 

[ChrisRJ]

To the best of my knowledge it should not be a problem to use an ordinary SATA controller or HBA only for boot drives (as long as it comes with a BIOS that support booting off of it). Even a traditional hardware RAID controller might be ok for the boot drives (but only for these).

 

[Etorix]

If one uses a PCIe slot just for booting, the easiest (and cheapest) way is to put a small NVMe M.2 drive in an adaptor.

 

[fmiz]

If one uses a PCIe slot just for booting, the easiest (and cheapest) way is to put a small NVMe M.2 drive in an adaptor.

Eh... if only the bios for X10SLH-F supported it. I would have to patch the bios myself (there are guides available). This feature is available for newer hardware (from what I've found, X10, and X11 supermicro boards do not support it). I understand that to many here my hw is ancient at this point

I will caution that it is possible to find all manner of plain PCIe SATA boards. These are often implemented with Marvell chips, and have sketchy or non-existant drivers in TrueNAS. I'm not suggesting it's not possible, but you need to do your homework very carefully if you attempt that route.

No, this is where I draw the line, if I had to build modules for FreeBSD then import them into TrueNAS, at that point I would switch everything to zfs on linux. I'm still running TrueNAS because it requires minimal maintenance, I'm already thinkering too much with other things.
Is Your Dell H310 a standard PCIe card? Doing a quick search I found many "mini" versions that seems to use a custom connector for PCIe. I also had read that older HBAs tend to run hot and need a decent amount of airflow, in my setup most of the airflow is blocked by the HDDs... it is funny how a small thing like a faulty boot drive leads to quite an upgrade: SAS HBA, SAS to sata breakout cable, SSD, and finally another fan mounted somewhere .
I will use the usb adapter as a semi-temporary solution and start to keep an eye for good enough SAS HBAs to buy when I finally decide to add more storage. At that point, some thinkering will be in order.
Thanks for your advice.

 

[rvassar]

No, this is where I draw the line, if I had to build modules for FreeBSD then import them into TrueNAS, at that point I would switch everything to zfs on linux. I'm still running TrueNAS because it requires minimal maintenance, I'm already thinkering too much with other things.

TrueNAS uses FreeBSD as a base. I wasn't suggesting importing any drivers, just that you can use it as a foundation to investigate if a device driver exists, works, etc... If it works well in FreeBSD, it probably plays on TrueNAS as well. I'm mostly trying to keep you from spending money on a cheap SATA board and being disappointed.

Is Your Dell H310 a standard PCIe card? Doing a quick search I found many "mini" versions that seems to use a custom connector for PCIe.

Yes. The mini version you'll probably find heavily discounted because they only work in the (mostly 1U) systems with the custom "sandwich" connection.

I also had read that older HBAs tend to run hot and need a decent amount of airflow, in my setup most of the airflow is blocked by the HDDs... it is funny how a small thing like a faulty boot drive leads to quite an upgrade: SAS HBA, SAS to sata breakout cable, SSD, and finally another fan mounted somewhere .

I haven't had too much trouble, but I run an oddball gamer case with a lot of airflow. IT mode should in theory reduce the thermal dissipation on account of the card not performing the parity calculations, etc... But I have no data to back up that assertion.

I will use the usb adapter as a semi-temporary solution and start to keep an eye for good enough SAS HBAs to buy when I finally decide to add more storage. At that point, some thinkering will be in order.
Thanks for your advice.

I ran that exact solution for almost a year. It was actually pretty stable, and it will give you time to research your path. Good luck!

 

[ChrisRJ]

I understand that to many here my hw is ancient at this point

Well, I run a couple of SuperMicro X9SRi-F boards here ;-)

 

[Constantin]

LSI chips are famous for running hot. My motherboard came with one baked in and it’s the hottest component on the board despite the small dedicated fan that i retrofit.

Naturally, SuperMicro used a custom-sized heat sink, making a swap to Cu or taller HX impossible. Location under path of any potential PCIe cards also problematic.

 

[andy0087]

Now I am engaged in the choice of a drive and system for installation at home.
I thought Kingston wasn’t very good, I guess I’m not very good at specs. You need to create a thread.

 

[fmiz]

Now I am engaged in the choice of a drive and system for installation at home.
I thought Kingston wasn’t very good, I guess I’m not very good at specs. You need to create a thread.

I was tempted to create a new thread, but after searching for more info on this forum I decided to ask here in this thread. There are already many other threads about this, started at different times by different people, maybe there should be a list of good "enough" boot SSDs to use. Someone here at somepoint wrote something along the lines of "any SSD is better than a USB flash drive" of at least an order of magnitude (talking about reliability). Now, an order of magnitude is clearly intended as a guideline and not really quantitatively, in my case I had my USB flash drive last for 6 years (looks like I've been very lucky), no one has seen an SSD last for 60 years.
But making a list is not that easy, it is possible that a manufacturer changes the controller in an SSD at a certain point in time, without changing the model number. Maybe it is even reasonable, if the controller was not specified in the beginning...
What is clear is that the choice of a boot device is not as critical as the media you use to store your data, it can be easily replaced (maybe with a configuration backup), cheap SSDs are fine. Unfortunately they require a SATA connection, instead of an USB one that will probably be never used in a NAS. Or, as Etorix said, an M.2 drive could be used, if you have an M.2 slot to spare.

 

[Patrick M. Hausen]

There are M.2 SATA to USB cases, if you are really short on internal connectors. In "stick" form factor without a cable, even. Still preferrable over a USB thumbdrive.

 

[HoneyBadger]

There are M.2 SATA to USB cases, if you are really short on internal connectors. In "stick" form factor without a cable, even. Still preferrable over a USB thumbdrive.

Exactly. For boot devices that don't see a lot of I/O at once, the USB protocol holds up well enough, and you can put any decently-respected SSD (or even an HDD) behind it - the point of concern then becomes the quality of the USB-to-SATA or USB-to-NVMe conversion chip in the adaptor.

 

[zubo100]

Hello everyone! I am new here, and I am currently in the proces of upgrading my previous NAS (DIY Xpenology) to TrueNAS Scale. My question is also about the boot drives. What exactly are they used for? How much does their speed influence the performance of the entire server?

I have a Asrock Z390m-itx/ac which has 6 Sata ports and 1 M.2 NVME. If the NVME is in use, only 5 SATA ports can be used.
I have 5x 4TB WD Red+ HDD, and one Samsung 256 NVME drive.
I plan to use the server as a NAS, and for a few aplications, (Plex, Pihole, Docker...) and for 1 virtual machine. The NVME drive was supposed to be dedicated to the virtual machine which would mean, that I do not have a free SATA port for boot. (And I am pretty much out of budget :D )

However I have a PCIe slot, which I could use to expand my storage, and I have 2 older SSDs Kingston SV300S37A/60G. The question is, which configuration makes more sense?

Config 1:
Sata 1 -> 4TB WD Red+
Sata 2 -> 4TB WD Red+
Sata 3 -> 4TB WD Red+
Sata 4 -> 4TB WD Red+
Sata 5 -> 4TB WD Red+
Sata 6 -> disabled
M.2 NVME -> Samsung 256GB - used for the virtual machine
PCIe -> Axagon Adapter from PCIe to 2x Sata (Kingston SV300S37A/60G) - used as boot

Config 1:
Sata 1 -> 4TB WD Red+
Sata 2 -> 4TB WD Red+
Sata 3 -> 4TB WD Red+
Sata 4 -> 4TB WD Red+
Sata 5 -> 4TB WD Red+
Sata 6 -> Kingston SV300S37A/60G - used as boot
M.2 NVME -> disabled
PCIe -> Axagon Adapter from PCIe to MSata and NVME (Samsung 256GB) - used for the virtual machine

On the Xpenology I used the NVME drive as a boot drive, and also the drive where APPS are installed.
After reading through a few documentations, I believe that trueNas does not support using the boot drive for anything else. As such, it would be a waste of space to use the Samsung NVME as a boot drive only.

Can I use the NVME drive as boot drive, and for the VirtualMachine?
Where are the TrueNas apps installed?
How much does the speed of the boot drive influence the performance of the server?
Would using the Kingston SV300S37A/60G as a boot drive make sense?

Thank you for any kind of help!

 

[Patrick M. Hausen]

I believe that trueNas does not support using the boot drive for anything else.

Yes.

Can I use the NVME drive as boot drive, and for the VirtualMachine?

No.

Where are the TrueNas apps installed?

In the pool that you configure for the apps.

How much does the speed of the boot drive influence the performance of the server?

Almost not at all.

Would using the Kingston SV300S37A/60G as a boot drive make sense?

Not knowing that particular product I still assume yes.

 

[Davvo]

The NVME drive was supposed to be dedicated to the virtual machine [...]

You probably want a mirrored vdev for that VM.

 

[Patrick M. Hausen]

You probably want a mirrored vdev for that VM.

Or regular snapshots and replication to a redundant storage pool.

 

[Etorix]

I have a Asrock Z390m-itx/ac which has 6 Sata ports and 1 M.2 NVME. If the NVME is in use, only 5 SATA ports can be used.

Typically, M.2 slots share a SATA lane with one motherboard port but NOT their PCIe lanes (or then with a PCIe slot). Which means that using a M.2 SATA drive disables one SATA port, but using a M.2 NVMe drive leaves ALL SATA ports enabled.
This appears to be the case here:

So you can boot from one 60 GB SSD, still have 5 SATA ports for the HDD pool and use the NVMe drive for VM. And NOT use the Axagon card, which is unlikely to be ZFS-qualified HBA.

Other options would involve a (proper) HBA and/or investigating how the PCIe slot can be bifurcated to run several NVMe drives.

The board has two Intel NICs… but these are i211 and i219, not the better i210.

 

[zubo100]

In the pool that you configure for the apps.

Can I select the boot pool for this?

Typically, M.2 slots share a SATA lane with one motherboard port but NOT their PCIe lanes (or then with a PCIe slot). Which means that using a M.2 SATA drive disables one SATA port, but using a M.2 NVMe drive leaves ALL SATA ports enabled.
This appears to be the case here:
View attachment 57951

So you can boot from one 60 GB SSD, still have 5 SATA ports for the HDD pool and use the NVMe drive for VM. And NOT use the Axagon card, which is unlikely to be ZFS-qualified HBA.

Thank you for this!!! I have absolutely not realised the simple fact that my Samsung is an NVME and not SATA. What you recommended is exactly what I will do. I originaly wanted to use 2 SSDs for a boot mirror, but I can live with 1. In case the boot drive fails, I can simply replace it with another.

The board has two Intel NICs… but these are i211 and i219, not the better i210.

The board also has WiFi which I planned to use as a dedicated port for the VM (since my NAS is not even 2m away from the router), but unfortunately TrueNas does not support Wireless Interfaces (or at least I think so... I am completely new to TrueNas, so I might be mistaken ), and I am not yet sure ir there is a possibility to do so. If not, I gues 2 NICs with link agregataion will have to be enough for all the trafic.

Thank you all for the replies!