Looking To Add M.2 NVMe

William Miceli

Dabbler
Joined
Jan 24, 2018
Messages
18
Hello!

I currently have:
  • Dell PowerEdge T320
  • FreeNAS 11.2 - Release (U1)
  • Xeon E5-2440 (6C/12T)
  • 48 GiB ECC DDR3
  • (4) 4TB Seagate NAS HDDs in RAIDZ2
  • Corsair 60 GB SSD (for boot)
I just yesterday went through upgrading from FN 11.1-U6.3... finding out that both of my USB drives (mirrored) that I was using for boot, are probably dying, because I was running into a bunch of random issues, similar to others on the forum who have found they also had dying USB drives. I was previously using my Corsair 60 GB SSD as a ZIL, but had to put that in as the boot drive just to get things running again.

I also use this system for running multiple VMs on, and without any ZIL, I definitely notice a difference...

Was looking at these two items to buy to use as a new ZIL:
The main questions I have:
  • Do NVMe drives work with FreeNAS, and is there some dependence on the hardware that I can check for (or if anyone has a similar Dell system that is using NVMe drives)?
  • Any other issues anyone can think of?
  • Any better options (can be anything)?
  • Would upgrading from using my Corsair SSD for boot to a M.2 SATA drive provide much advantage?
Thank you!
 
Last edited:

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Do NVMe drives work with FreeNAS
Yes, for some time now.

I was previously using my Corsair 60 GB SSD as a ZIL
Yeah, that's a very bad idea. It doesn't have power loss protection, so it's useless as an SLOG device. You'd be better off turning off sync writes and getting better performance for the same data protection.

Would upgrading from using my Corsair SSD for boot to a M.2 SATA drive provide much advantage?
SATA is SATA, regardless of the connector. Not that an NVMe device is going to provide a meaningful benefit over SATA for a boot device, anyway.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,829
I was previously using my Corsair 60 GB SSD as a ZIL, but had to put that in as the boot drive just to get things running again.
The SSD is a better boot drive than a USB stick. If you have an encrypted pool, be sure to have a recovery key, etc. downloaded in advance in case the single boot drive goes.

As far as the SLOG goes, I'd look at a couple of resources:
  1. The Local thread re: SLOG performance benchmarking
  2. The ServeTheHome recommendations for FreeNAS systems page
  3. The ServeTheHome page on best SLOGs
  4. The FreeNAS hardware recommendations resource
As @Ericloewe pointed out, you ideally want a device with power-loss protection. Some SSDs offer that, so do very expensive Optane options like the 4800X series from Intel. Or, you can gamble and go for a m.2 based Optane offering that doesn't have any DRAM chips and hope that your number never comes up (it's quite unlikely). In that case, consider the Optane 905P series. If I were to build today, I'd likely choose the slightly older but cheaper 800 series of Optane instead.

...but ONLY if the m.2 slot is actually a PCIe 3.0x2 or higher. Otherwise, you may spend a lot of money for a high-bandwidth Optane product only to get hobbled by a slow bus interface. For the time being, I am very happy with my dual S3710 SSD setup, which doesn't break speed records but is mirrored and fairly robust. My L2ARC drive is a leftover Samsung 840 drive.
 
Last edited:

William Miceli

Dabbler
Joined
Jan 24, 2018
Messages
18
Yeah, that's a very bad idea. It doesn't have power loss protection, so it's useless as an SLOG device. You'd be better off turning off sync writes and getting better performance for the same data protection.
I tried to find some high performance/high endurance drive that explicitly stated it had power loss protection, which I did find some, however they were a bit out of my budget being usually $300+.

I do have two questions though...
Regarding disabling turning off sync writes, I was under the impression, from articles warning against turning it off saying that it was very dangerous with respect to data integrity, that in the case of an application writing "data" to disk and then reading that same "data" before it was flushed to the actual storage disks that the old "data" would be returned instead, potentially causing data corruption.

I just recently found one article that said ZFS will store data not yet committed to disk into system memory, and act as a kind of "overlay" to the storage pool, where in a scenario such as the one above, requests to read the data not yet committed would be read from the "overlay" instead of the storage pool, which would prevent data corruption due to reading old data.

I would assume the second one would be more "true", is the correct?

Also, for the power loss protection issue...
I went through @Constantin's awesome links (Thank you by the way!) and was kind of wondering about both of your opinions on whether it might be a more cost-effective route by simply buying an UPS? They can be a little pricey, but just kind of wondering that if I get an UPS and have it to simply shutdown my server after a minute or two of power loss, FreeNAS would then have time to flush all uncommitted writes to disk and power off, versus buying the more expensive higher-tier drives that include power loss protection, would the UPS generally be more cost-effective in the long-run? My server also has redundant power-supplies in case one fails as well. But that way I can buy the <$80 NVMe drives a little more safely? XD


As @Ericloewe pointed out, you ideally want a device with power-loss protection. Some SSDs offer that, so do very expensive Optane options like the 4800X series from Intel. Or, you can gamble and go for a m.2 based Optane offering that doesn't have any DRAM chips and hope that your number never comes up (it's quite unlikely). In that case, consider the Optane 905P series. If I were to build today, I'd likely choose the slightly older but cheaper 800 series of Optane instead.

I definitely appreciate the recommendations, but holy guacamole, the 905P is more expensive than what I paid for my entire server, haha! Not to say that it isn't worth it, but unfortunately I have to budget a bit since I'm still in college and don't have a "real" job yet :P

I'm sure you'd probably be hesitant to recommend anything that would go against best practices, which I definitely understand. But I guess would you say that for the sub-$80 price of the Samsung 970 EVO, would it be worth the investment?

I don't have mission-critical stuff anywhere on my FreeNAS, and have daily backups to Backblaze B2 storage, so worst-case scenario seems like it would just take me a while to fix whatever broke, and if my data is lost locally, I pay the expense of downloading from Backblaze.

...but ONLY if the m.2 slot is actually a PCIe 3.0x2 or higher. Otherwise, you may spend a lot of money for a high-bandwidth Optane product only to get hobbled by a slow bus interface. For the time being, I am very happy with my dual S3710 SSD setup, which doesn't break speed records but is mirrored and fairly robust. My L2ARC drive is a leftover Samsung 840 drive.

True, I did see a few M.2 drive variations that were a bit cheaper, and found out that they only used two PCIe lanes instead of the usual 4, haha. And to this, I'm curious what you think of the UPS proposition as well.

Thank you @Ericloewe and @Constantin!
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Regarding disabling turning off sync writes, I was under the impression, from articles warning against turning it off saying that it was very dangerous with respect to data integrity, that in the case of an application writing "data" to disk and then reading that same "data" before it was flushed to the actual storage disks that the old "data" would be returned instead, potentially causing data corruption.
That's interesting, I don't think I'd ever heard that concern before. I see where it's coming from and I can see it being a problem at some layers with some systems. With ZFS, I'm going to say that it's most likely not an issue - the only major difference to a sync write is that it gets written to the ZIL to satisfy the requirement that the data be committed to non-volatile storage.
I'm pretty sure that I/O is serialized in the sense that a read will reflect all previous writes. Of course, that leaves threading concerns to address, but that's a problem for the application.


might be a more cost-effective route by simply buying an UPS?
You should have a UPS, but it won't make async magically as safe as sync. It does eliminate or reduce power failures as a point of failure, but the box can still panic and lose a few seconds of data.
 

William Miceli

Dabbler
Joined
Jan 24, 2018
Messages
18
With ZFS, I'm going to say that it's most likely not an issue - the only major difference to a sync write is that it gets written to the ZIL to satisfy the requirement that the data be committed to non-volatile storage.
So in the case of keeping sync set to "standard", having a SLOG wouldn't see any performance increase for async writes since they go directly to system memory first, but sync writes which would be directed to the ZIL would be the only performance increase, as it would change the the ZIL from being on the storage pool hard disks to the SLOG.
But also that adding a SLOG (in my low price range) would actually decrease my data protection, as the SLOG drives that I seem to like have DRAM modules.
Hopefully that's correct?

You should have a UPS, but it won't make async magically as safe as sync. It does eliminate or reduce power failures as a point of failure, but the box can still panic and lose a few seconds of data.
Ah, alrighty, that makes sense.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
So in the case of keeping sync set to "standard", having a SLOG wouldn't see any performance increase for async writes since they go directly to system memory first, but sync writes which would be directed to the ZIL would be the only performance increase, as it would change the the ZIL from being on the storage pool hard disks to the SLOG.
But also that adding a SLOG (in my low price range) would actually decrease my data protection, as the SLOG drives that I seem to like have DRAM modules.
Hopefully that's correct?
Yup. Note that the ZIL is never read from unless something goes wrong, because the writes are also kept in RAM.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,829
I'd definitely go with a UPS regardless. Then go through the process of connecting the UPS to the server - i.e. make sure the server can read UPS status messages (usually via USB). Then test the rig and confirm that you can yank the power cord for the UPS out of the wall, followed by the server noticing said failure, followed by the server gracefully shutting down. It's an easy test and best done when there is no data in the pool yet.

As for the SLOG, I see nothing wrong with conducting a simple test like buying a 32GB Optane module and seeing if it makes a significant difference in your workflow. The cost is pretty modest. As I understand it, the SLOG will only hold a very small amount of data. While the consumer-grade 32GB sticks are not nearly as fast as the 905P series, it's a way to see if a difference is made. If you experience a epiphany, it might be worthwhile looking for a faster SLOG solution... or wait a year or two until Intel brings out a x4 version of the less expensive Optane adapters.

As for me, I use a mirrored set of S3710's, which basically is supposed to prevent bad writes. They're not the fastest in the world, but they can be had for a modest sum on eBay and they fulfill the requirements of a SLOG device re: power protection.
 

William Miceli

Dabbler
Joined
Jan 24, 2018
Messages
18
Yup. Note that the ZIL is never read from unless something goes wrong, because the writes are also kept in RAM.
Awesome, and that's a really interesting point. I never really considered it might be like that, but it definitely makes a lot of sense for it to function in that way.

Thank you!! :D
 

William Miceli

Dabbler
Joined
Jan 24, 2018
Messages
18
I'd definitely go with a UPS regardless. Then go through the process of connecting the UPS to the server - i.e. make sure the server can read UPS status messages (usually via USB). Then test the rig and confirm that you can yank the power cord for the UPS out of the wall, followed by the server noticing said failure, followed by the server gracefully shutting down. It's an easy test and best done when there is no data in the pool yet.

I definitely will be getting an UPS eventually, and will probably be the next "upgrade" I do for my server. I probably need to get an UPS with Active PFC, but even at business I was working for recently, their server had an UPS with Active PFC, the server was the exact same model that I have, and when the power went out, the server still shut off, so I feel like I have to try and dig a little more about UPSs to see if maybe they just had a defective UPS, or if there was something I'm missing. (I my position at the (small) business wasn't technical, but being the most technical person there, I was often fixing their stuff, haha)

I already have more data on my pool that I can store elsewhere unfortunately, but I should probably just be able to just physically disconnect my drives to test it though when I get an UPS, right? Or maybe an easier way would be to just "offline" the ZFS pool?

As for the SLOG, I see nothing wrong with conducting a simple test like buying a 32GB Optane module and seeing if it makes a significant difference in your workflow. The cost is pretty modest. As I understand it, the SLOG will only hold a very small amount of data. While the consumer-grade 32GB sticks are not nearly as fast as the 905P series, it's a way to see if a difference is made. If you experience a epiphany, it might be worthwhile looking for a faster SLOG solution... or wait a year or two until Intel brings out a x4 version of the less expensive Optane adapters.

I found even a 16 GB Optane M.2 on Amazon which is at a very reasonable price. I notice that they mention often about SATA, but nothing about NVMe, and reviews mentioning SATA SSD performance, which definitely isn't bad in itself, since my only experience with a SLOG was with my low-end SATA SSD anyway, so I'm sure it would definitely help to get me back to at least the performance I was seeing previously! Do you know if there would be anything special needed to do for setting this up as a SLOG? I ask just because it seems like this one doesn't function as a traditional drive. Or does FreeNAS just access drives at such a low level that it's just a regular drive to it?

I kept looking for NVMe drives online, and actually found one from Crucial with Power Loss Protection (Datasheet Link) (within my budget too! XD). For some reason, the datasheet does not list that it is included, but on their SSD comparison page, it shows it does. I called Crucial, who did confirm with me that the P1 series does have the power loss protection on it. The price currently being $89 on Amazon is definitely something I can afford. The seq. writes are rated at 950 MB/s, which isn't the fastest, but probably twice what I was getting with my SATA SSD anyway, and I can't complain for the price anyway I feel like. I feel like 500 GB is significantly more than I'll ever need for a SLOG, but I assume it will probably not be a bad thing anyway, as it will just be able to last longer. Does this seem like a reasonable option?
 
Top