Drive suggestion

[Fastline]

Hello guys,

Planning to build a flash storage NAS. I want to know what could be the best drive in SATA 2.5 and NVMe (Gen3/Gen4). U.2/U3 or M.2.

Secondly, for the storage, except for the SLOG, do i need features such as PLP or high endurance or a normal consumer grade SSD will work here?

Thanks

 

[garm]

Workload Tuning — OpenZFS documentation

openzfs.github.io

 

[joeschmuck]

Planning to build a flash storage NAS.

I built one a few months ago. Very nice by the way.

Here is the link to it.

My New All NVMe Virtualized TrueNAS Server - Story and a Guide

PART 1: WHY A ESXi SERVER? I have a personal preference for virtualization and it started a long time ago while using VMWare Workstation, it was easy to operate on my Windoze computer and I'm accustom to it. While this thread is mainly how to select the right components and put together a...

www.truenas.com

I want to know what could be the best drive in SATA 2.5 and NVMe (Gen3/Gen4). U.2/U3 or M.2.

That depends on what the goal is.

Fast data throughput, M.2 Gen 5 today, but if you wait a few years Gen 7 will be much faster. You can search the internet for data on these things. I'm not sure if Gen 6 is even available, I have not looked into it.

With speed comes heat, be very aware of that. If you want a cooler system a Gen 3 M.2 is good as you should not need a heatsink for those, but jump up to Gen 4 or Gen 5, a heatsink is required.

If you want cost effective then look into SATA 2.5" drives, they are easy to use and make connectivity easy.

Either way you will need to determine what the purpose of the drives are for and capacity, then figure it out through a lot of research, that is what I needed to do. My 4TB M.2 Gen 4 modules cost me $200 each. BUT I just located 8TB Gen 4 M.2 for $58.25 (USD) from AliExpress, Samsung 980 Pro. That definitely sounds wrong. I want to buy four and see what I get. If they are 4TB or 8TB, I'd be happy. Except I don't need them.

 

[Fastline]

I built one a few months ago. Very nice by the way.

Here is the link to it.

My New All NVMe Virtualized TrueNAS Server - Story and a Guide

PART 1: WHY A ESXi SERVER? I have a personal preference for virtualization and it started a long time ago while using VMWare Workstation, it was easy to operate on my Windoze computer and I'm accustom to it. While this thread is mainly how to select the right components and put together a...

www.truenas.com

That depends on what the goal is.

Fast data throughput, M.2 Gen 5 today, but if you wait a few years Gen 7 will be much faster. You can search the internet for data on these things. I'm not sure if Gen 6 is even available, I have not looked into it.

With speed comes heat, be very aware of that. If you want a cooler system a Gen 3 M.2 is good as you should not need a heatsink for those, but jump up to Gen 4 or Gen 5, a heatsink is required.

If you want cost effective then look into SATA 2.5" drives, they are easy to use and make connectivity easy.

Either way you will need to determine what the purpose of the drives are for and capacity, then figure it out through a lot of research, that is what I needed to do. My 4TB M.2 Gen 4 modules cost me $200 each. BUT I just located 8TB Gen 4 M.2 for $58.25 (USD) from AliExpress, Samsung 980 Pro. That definitely sounds wrong. I want to buy four and see what I get. If they are 4TB or 8TB, I'd be happy. Except I don't need them.

I have gone through your build and learned a lot. However, i still have a few questions:

1. Should i go for TLC/QLC/MLC SSD?
2. What is better? An SSD with DRAM Cache or Cacheless?

 

[Fastline]

I was looking at High Point SSD7540 which has space for 8*M.2 drives. A Gen4 NVMe typically requires 4 lanes to operate at its full/max speed. So that means 8*4 lanes = 32 lanes. And the card is only X16. So does this mean the SSD will perform at 2 lanes only? If that's the case, it means even by having a Gen4 SSD, the SSD will perform at half of the speed or maybe even less than that.

Or does it mean that it can pull a total of 32 lanes using the X16 slot? Provided that the CPU is at least 48 lanes or higher.

 

[joeschmuck]

1. Should i go for TLC/QLC/MLC SSD?

The only issue here is how long they last. SLC is the longer lasting format and comes at a price. The three you listed have a Program/Erase cycle significantly lower than SLC. Google is your friend to research it. But as for is they react differently with TrueNAS, they are all the same in that respect.

What is better? An SSD with DRAM Cache or Cacheless?

Generally Cache is faster however there are very fast cacheless devices. Again, Google. TrueNAS could care less, it's storage.

As for the High Point SSD7540, below is the company link, take a read.

https://www.highpoint-tech.com/nvme-individual/ssd7540

My concern:

• Supports RAID 0, 1, 10 and Single-Disk Configurations

Does this mean it does not expose each NVMe as its own single drive, showing 8 drives for a full card? This is KEY to know. If you cannot expose all 8 drives individually then this is not for TrueNAS. What very little I have read, this device only support RAID which is a no-go.

To answer your question... It appears to expose 16 lanes to an entire RAID drive configuration so individual NVMe lanes are not a factor.

 

[Morris]

I've been using a pair of Crucial MX500 2TB SATA drives in a mirror for a few years. They have been my go to for SATA SSDs for some time and there very reliable and will saturate the SATA interface on reads and writes. I store video on them including a PLEX library.

 

[Fastline]

The only issue here is how long they last. SLC is the longer lasting format and comes at a price. The three you listed have a Program/Erase cycle significantly lower than SLC. Google is your friend to research it. But as for is they react differently with TrueNAS, they are all the same in that respect.

Yes, i'm on it :)

Generally Cache is faster however there are very fast cacheless devices. Again, Google. TrueNAS could care less, it's storage.

Okay. But what about the Random Read/Write? I heard that even on an SSD, it can be slower than spinning rust. So what about that?

As for the High Point SSD7540, below is the company link, take a read.
https://www.highpoint-tech.com/nvme-individual/ssd7540
My concern:

• Supports RAID 0, 1, 10 and Single-Disk Configurations

Does this mean it does not expose each NVMe as its own single drive, showing 8 drives for a full card? This is KEY to know. If you cannot expose all 8 drives individually then this is not for TrueNAS. What very little I have read, this device only support RAID which is a no-go.

To answer your question... It appears to expose 16 lanes to an entire RAID drive configuration so individual NVMe lanes are not a factor.

Good catch ;)

I have asked High Point. As soon as i have info, I'll update the thread. Meanwhile, i was looking at NV9547-4I from 10Gtek. Any thoughts on this?

 

[Fastline]

I've been using a pair of Crucial MX500 2TB SATA drives in a mirror for a few years. They have been my go to for SATA SSDs for some time and there very reliable and will saturate the SATA interface on reads and writes. I store video on them including a PLEX library.

Cool!

 

[joeschmuck]

Okay. But what about the Random Read/Write? I heard that even on an SSD, it can be slower than spinning rust. So what about that?

I don't think that a spinning rust drive put against a decent SSD would be faster at all in any test. But you could create a vdev that can be very fast for IOPS, but maybe a 3-way mirror of HDDs vs a single SSD. I have not really looked into this aspect much because I would not use it but one to one, I find that hard to believe. There are some old SSD that are fairly slow but anything in the last 5 years or possibly more should be fast. I would say that if you need a high IOPS, figure out what you need and then look for a solution that meets that need. Just wanting high IOPS is not a requirement. SSDs are small, use little power, create little heat, and are not very expensive these days. If you plan to do a lot of writing data (I mean a lot!), then a SSD/NVMe does have a finite life where the spinning rust does not. There is a disadvantage to SSD/NVMe and that is it. this is why there is a Wear Level value. Each erase cycle takes life away.

NV9547-4I

I have no idea, I did not see a user manual. It does use a PLX chip so I would be cautious that is isn't a RAID card only as well. When you buy something you need to make sure that it can expose all your drives as individual drives, that is the important part. Find the user manual for it, read it. If it starts talking about bifurcation, before buying it, make sure your motherboard support the proper bifurcation needed.

My NVMe adapter requires bifurcation and places each NVMe (4 lanes each) on the PCIe bus. This is a direct connection, nothing in the middle like a RAID controller, except there will be line drivers between the PCIe slot and the NVMe cards. This is the normal way to connect electronic data paths, nothing fancy, it's very basic.

 

[Fastline]

I don't think that a spinning rust drive put against a decent SSD would be faster at all in any test.

Not me either. When i was searching yesterday, i found this:

DRAM in SSD zpools?

Is it generally a good idea to buy SSDs with dram if they're going to be used in raidz pools? Also should TLC be preferred to QLC like with regular PC builds?(at least for the OS drive)

www.truenas.com

But you could create a vdev that can be very fast for IOPS, but maybe a 3-way mirror of HDDs vs a single SSD.

Wow. Does this mean higher IOPS means faster read/write i.e. providing more throughput?

I have not really looked into this aspect much because I would not use it but one to one, I find that hard to believe. There are some old SSD that are fairly slow but anything in the last 5 years or possibly more should be fast.

Yes, indeed.

I would say that if you need a high IOPS, figure out what you need and then look for a solution that meets that need. Just wanting high IOPS is not a requirement. SSDs are small, use little power, create little heat, and are not very expensive these days. If you plan to do a lot of writing data (I mean a lot!), then a SSD/NVMe does have a finite life where the spinning rust does not. There is a disadvantage to SSD/NVMe and that is it. this is why there is a Wear Level value. Each erase cycle takes life away.

Yeah, how sad is that, but then the speed ;)

I have no idea, I did not see a user manual. It does use a PLX chip so I would be cautious that is isn't a RAID card only as well. When you buy something you need to make sure that it can expose all your drives as individual drives, that is the important part. Find the user manual for it, read it. If it starts talking about bifurcation, before buying it, make sure your motherboard support the proper bifurcation needed.

Gotcha!

Have asked them details.

My NVMe adapter requires bifurcation and places each NVMe (4 lanes each) on the PCIe bus. This is a direct connection, nothing in the middle like a RAID controller, except there will be line drivers between the PCIe slot and the NVMe cards. This is the normal way to connect electronic data paths, nothing fancy, it's very basic.

Yes, yes. Getting to know things :)

 

[joeschmuck]

Wow. Does this mean higher IOPS means faster read/write i.e. providing more throughput?

For random operations. There are many factors when it comes to throughput. Any decent SSD will provide high throughput and high IOPS, but hard drives are limited by rotational delay (latency) so if you create a 3-way mirror, whichever drive finds the data first will provide it thus increasing IOPS. A higher RPM drive will have less latency, but the tradeoff if more heat, most cost, more power required.

As I said, you need to know what your requirements are. If you just want a super fast machine, build a full NVMe machine like I did, it is fast. I don't need that but I wanted low power, noise, heat, and a long life which I can achieve in my use case where I do not write a lot of data. If I were a data center, my build would be very wrong.