Drive Replacement

[Fastline]

My impression, which can be totally wrong, is the following:

• PCIe 3 x8 can transfer around 8 GB/s or close to 64 Gbps of data.
• Each port supports SAS 12 Gbps, which results in a total of 48 Gbps (close to the total amount of 64 Gbps, the difference could be the overhead)
• Each drive can get up to 12 Gbps (you can connect up to four NVMe drives with x4 lanes, one per port, or eight in x2 lanes, two per port)
• If you connect more than a single drive to a port, the connected drives split the available 12 Gbps: connecting four drives to each port results in each having up to 3 Gbps or close to 375 MB/s

But I have never used an HBA and my knowledge about them is shaky; not having much time to reasearch doesn't help.
Other users will be able to tell you more accurate info.

I'm so confused now ;(

@Etorix @Patrick M. Hausen any insights?

 

[Patrick M. Hausen]

Are we talking spinning disks and SATA or NVMe here? No SATA disk will ever reach that 6G/s ...

A WD Red Pro has got a sustained read rate of "up to 265 MB/s". (1)
A WD Ultrastar SATA SSD has got a sustained read rate of "510 MB/s" (2)

The interface speed of SATA is in no way related to what the drive is actually able to deliver.

I know that NVMe SSDs which are essentially PCIe devices can - when used in larger numbers - outperform the PCIe lanes and bandwidth of your system. So you need to plan accordingly. But nothing with a SATA or SAS interface will ever saturate that interface bandwidth.

(1) https://www.westerndigital.com/de-de/products/internal-drives/wd-red-pro-sata-hdd?sku=WD221KFGX
(2) https://www.westerndigital.com/en-u...rives/ultrastar-dc-sa210-sata-ssd?sku=0TS1652

We had this with another forum beginner a couple of months ago. How on earth do people get the impression that 6G/s is the actual disk speed? Never was and probably never will be.

 

[Fastline]

Are we talking spinning disks and SATA or NVMe here? No SATA disk will ever reach that 6G/s ...

A WD Red Pro has got a sustained read rate of "up to 265 MB/s". (1)
A WD Ultrastar SATA SSD has got a sustained read rate of "510 MB/s" (2)

The interface speed of SATA is in no way related to what the drive is actually able to deliver.

I know that NVMe SSDs which are essentially PCIe devices can - when used in larger numbers - outperform the PCIe lanes and bandwidth of your system. So you need to plan accordingly. But nothing with a SATA or SAS interface will ever saturate that interface bandwidth.

(1) https://www.westerndigital.com/de-de/products/internal-drives/wd-red-pro-sata-hdd?sku=WD221KFGX
(2) https://www.westerndigital.com/en-u...rives/ultrastar-dc-sa210-sata-ssd?sku=0TS1652

We had this with another forum beginner a couple of months ago. How on earth do people get the impression that 6G/s is the actual disk speed? Never was and probably never will be.

Actually, here's what i'm trying to understand:

I guess the normal SATA III is capable of 6Gb/s which means 600MB/s. HDDs can go up to 300MB/s (enterprise). So, for my use case, 16x300MB/s = 4800MB/s or 4.8Gb/s approx. So, as the bandwidth for the HBA (LSI9400-16i) is 12Gb/s, which means one can connect up to 40HDDs with almost zero impact on the HDD speed. Is that correct?

However, like you mentioned, an SSD can easily saturate this speed. Considering a normal SATA SSD which is capable of 500-600MB/s approx. If i try to install up to 20 SSDs:

20x600MB/s = 12000MB/s or 12Gb/s approx. I'm under the limit and probably no impact on either of the SSD speeds. Or is my calculation wrong here?

And if that's the case, an U.2 NVMe installed on an HBA rated for 12Gb/s speed, it can just push 1.5GB/s only? That's so much under utilized then.

Please don't freak out if i'm wrong here :)

 

[Patrick M. Hausen]

1. In what usage scenario on this earth do you get 20 SATA SSD to fire at 600MB/s all simultaneously?
2. U.2 NVMe is PCIe already. It needs special enclosures and will be wired directly to the main board. Then it's a matter of how many devices, how many PCIe lanes, etc. But you cannot wire a U.2 SSD to a SAS/SATA HBA.

If you want to use U.2 you need to buy a barebone system designed for that, anyway. We have two of these: https://www.supermicro.com/en/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm

 

[Etorix]

And if that's the case, an U.2 NVMe installed on an HBA rated for 12Gb/s speed, it can just push 1.5GB/s only? That's so much under utilized then.

Don't attache U.2 drives to a tri-mode HBA; put them directly on PCIe lanes, or use a PLX switch.
What is your hardware by the way?

 

[Davvo]

I think his question is more about how the bandwidth is managed by the HBA.

 

[Fastline]

1. In what usage scenario on this earth do you get 20 SATA SSD to fire at 600MB/s all simultaneously?

It was approx. They can go upto roughly 500MB/s mostly. Isn't it?

2. U.2 NVMe is PCIe already. It needs special enclosures and will be wired directly to the main board. Then it's a matter of how many devices, how many PCIe lanes, etc. But you cannot wire a U.2 SSD to a SAS/SATA HBA.

Are you sure? Cause LSI itself says that you can connect NVMe SSD (U.2) to the HBA using a backplane.

Of course, i know you cannot use a SATA port for connecting a U.2 Drive.

If you want to use U.2 you need to buy a barebone system designed for that, anyway. We have two of these: https://www.supermicro.com/en/Aplus/system/1U/1113/AS-1113S-WN10RT.cfm

Are you sure my friend? It is really not possible to connect a U.2 Drive on a HBA with SFF8643 SAS port?

 

[Patrick M. Hausen]

Are you sure my friend?

No, sorry.

It is really not possible to connect a U.2 Drive on a HBA with SFF8643 SAS port?

I just would not do that. I prefer to buy complete systems - at least bare bones - that match the intended application.

 

[Fastline]

Don't attache U.2 drives to a tri-mode HBA; put them directly on PCIe lanes, or use a PLX switch.

Umm, would you please explain what is a PLX switch? The Motherboard I'm looking does not have an U.2 switch and i need to use like 8 SSD Drives which are U.2. for fast storage.

What is your hardware by the way?

The main one regarding which thread is about, I'm trying to figure out the hardware so that i don't have to buy the new hardware anytime soon because of lack of bandwidth or any similar issues.

 

[Fastline]

No, sorry.


I just would not do that. I prefer to buy complete systems - at least bare bones - that match the intended application.

Hmm. Quite sad. But thanks for explaining. As always, your insight is always helpful. Thank you

So, is there really not a way where i can use some U.2 Drives using the HBA Card?

 

[Etorix]

I think his question is more about how the bandwidth is managed by the HBA.

This was indeed the question (and I think the answer is that each drive has its 6 Gb/s link, of which it may actually use 150-200 MB/s, and then HBA will then repackage everything into the x PCIe lanes), but we're already seeing a sub-question with NVMe drives.

So, is there really not a way where i can use some U.2 Drives using the HBA Card?

You could, with one of the "Tri-mode HBA" that Broadcom wants to sell you for the usual hefty amount to keep their HBA business afloat. But they somewaht suck at that, and I think you cannot use the tri-mode for SAS/SATA and NVMe at the same time. Better stay with a good old (and cheap) 9200/9300 for spinners and direct PCI lanes (or a PLX) for the NVMe.
PLX chips are PCIe switches: They take X lanes on one end and serve Y lanes on the other, just like your Ethernet switch.
Here is an example, which could serve your eight U.2 for a single x16 slot:

https://nl.aliexpress.com/item/4000029811733.html

The main one regarding which thread is about, I'm trying to figure out the hardware so that i don't have to buy the new hardware anytime soon because of lack of bandwidth or any similar issues.

Fair enough. What's the board you're looking at?
From what you describe, you need (at least) 8*4=32 lanes for the "fast" NVMe pool, plus enough SATA ports or a x8 slot for a 9200/9300 HBA for the "slow" spinning pool. That's typical Xeon Scalable/EPYC territory.
For instance, a X11SPM-TPF provides:

• 12 SATA ports;
• a x8 slot for a HBA if the above is not enough, or you have SAS drives;
• two x16 slots, each of which can address 4 U.2 drives with bifurcation;
• 10 GbE already on-board, and there's still a M.2 slot to boot from.

That should cover the requirements. For more lanes, look at dual socket boards and/or EPYC.

 

[Fastline]

This was indeed the question (and I think the answer is that each drive has its 6 Gb/s link, of which it may actually use 150-200 MB/s, and then HBA will then repackage everything into the x PCIe lanes), but we're already seeing a sub-question with NVMe drives.

Yes. Can you please verify the calculation below?

1. The normal SATA III is capable of 6Gb/s which means 600MB/s. HDDs can go up to 300MB/s (enterprise). So, for my use case, 16x300MB/s = 4800MB/s or 4.8Gb/s approx. So, as the bandwidth for the HBA (LSI9400-16i) is 12Gb/s, which means one can connect up to 40HDDs with almost zero impact on the HDD speed. Is that correct?

2. Considering a normal SATA SSD which is capable of 500-600MB/s approx. If i try to install up to 20 SSDs:

20x500MB/s = 10000MB/s or 10Gb/s approx. I'm under the limit and probably no impact on either of the SSD speeds. Or is my calculation wrong here?

You could, with one of the "Tri-mode HBA" that Broadcom wants to sell you for the usual hefty amount to keep their HBA business afloat. But they somewaht suck at that, and I think you cannot use the tri-mode for SAS/SATA and NVMe at the same time. Better stay with a good old (and cheap) 9200/9300 for spinners and direct PCI lanes (or a PLX) for the NVMe.

Hmm. So i guess one mode at one time then? Interestingly, the 9400=16i datasheet says, you can use all together.

PLX chips are PCIe switches: They take X lanes on one end and serve Y lanes on the other, just like your Ethernet switch.
Here is an example, which could serve your eight U.2 for a single x16 slot:
https://nl.aliexpress.com/item/4000029811733.html

The product link you attached, i looked for a same when i was figuring out the U.2 Drives. It was from High Point. I have used their RAID cards in past, quite reliable. But it is also SFF8643 so how different is this from a normal HBA Card from LSI which also offers 8643 SAS ports?

Fair enough. What's the board you're looking at?
From what you describe, you need (at least) 8*4=32 lanes for the "fast" NVMe pool, plus enough SATA ports or a x8 slot for a 9200/9300 HBA for the "slow" spinning pool. That's typical Xeon Scalable/EPYC territory.
For instance, a X11SPM-TPF provides:

OMG. I'm planning to buy the same board you had in mind.

In short, here's what i plan

18x16TB SATA HDD
8xU.2

Plus, i want to reserve a slot for future to upgrade the network to 25GbE or 40GbE. I can do that even now once i figure out the rest of the hardware.

How do i figure out the bandwidth and the lanes?

 

[Etorix]

1. The normal SATA III is capable of 6Gb/s which means 600MB/s. HDDs can go up to 300MB/s (enterprise). So, for my use case, 16x300MB/s = 4800MB/s or 4.8Gb/s approx. So, as the bandwidth for the HBA (LSI9400-16i) is 12Gb/s, which means one can connect up to 40HDDs with almost zero impact on the HDD speed. Is that correct?

That's 12 Gb/s per SAS lane, or 6 Gb/s per SATA lane, although the HBA may not have the total capacity to handle all 16 lanes at full (theoretical) bandwidth simultaneously.

The product link you attached, i looked for a same when i was figuring out the U.2 Drives. It was from High Point. I have used their RAID cards in past, quite reliable. But it is also SFF8643 so how different is this from a normal HBA Card from LSI which also offers 8643 SAS ports?

A PLX is just a switch. It passes PCIe signals, bifurcating as required, and works with any PCIe device (GPU, NIC, SSD, whatever). It does not add an extra layer of running NVMe drives through a SCSI stack.

OMG. I'm planning to buy the same board you had in mind.

That was just an almost minimal example. But it helps to talk about an actual piece of hardware, owned or planned.

In short, here's what i plan

18x16TB SATA HDD
8xU.2

With so many drives, I think you should think about the case, and how you're going to attach, wire and cool these drives before even thinking of a motherboard. And it may be better to have one HDD NAS and one NVMe NAS, each with its own dedicated enclosure, rather than looking for a single jack-of-all-trades.

 

[Davvo]

Why going U.2 instead of M.2 though?

 

[Fastline]

That's 12 Gb/s per SAS lane, or 6 Gb/s per SATA lane, although the HBA may not have the total capacity to handle all 16 lanes at full (theoretical) bandwidth simultaneously.

OMG. So, what do you suggest here to utilize 18 HDDs at once with full bandwidth? Per SAS lane here means 12Gb/s per SAS port or the whole HBA?

Also, i wonder how those so many drives like 40-60 drives are connected in a rackmount server. Do they use something different than a HBA? I don't think any Motherboard will have so many onboard SATA ports.

PLX is just a switch. It passes PCIe signals, bifurcating as required, and works with any PCIe device (GPU, NIC, SSD, whatever). It does not add an extra layer of running NVMe drives through a SCSI stack.

Hmm. Does it also has something like IT firmware or something?

That was just an almost minimal example. But it helps to talk about an actual piece of hardware, owned or planned.

I had that in my mind as someone suggested that before in the other thread i created ;)

With so many drives, I think you should think about the case, and how you're going to attach, wire and cool these drives before even thinking of a motherboard.

Well, I'm having a Tower Case which has space for 18 HDDs in a single row. 14 in a single row and then 2x2 at the bottom. My friend did some CNC work and he has made a custom cage for the 8 separate cage with a 80mm FAN with 2500RPM. The front has 4x120mm which has 2500RPM FANs installed and then there are 4x120mm at top. When the case is fully closed and at load, the HDD temps are 35-37c during summers. Is that okay or i really need to upgrade my case too?

And it may be better to have one HDD NAS and one NVMe NAS, each with its own dedicated enclosure, rather than looking for a single jack-of-all-trades.

Well, the workload i have, i need one single one where i can do faster read/writes on the SSD pool and the bulk storage and rendering on the HDD

 

[Fastline]

Why going U.2 instead of M.2 though?

Umm, i thought i could easily connect U.2 Drives to the HBA but seems like i cannot ;(

My friend planned to give me a couple of U.2 Drives for decent price. So i thought it would be all good. After everything was sorted, I'm just stuck with the bandwidth and the lanes thingy ;(

 

[Etorix]

Why going U.2 instead of M.2 though?

Capacity and cooling, which goes with better performance as well. I can totally understand that.

OMG. So, what do you suggest here to utilize 18 HDDs at once with full bandwidth? Per SAS lane here means 12Gb/s per SAS port or the whole HBA?

Per lane means 4*12 Gb/s per SFF-8643 connector (4 lanes). 16*12 Gb/s for a -16i/16e HBA (though the controller may throttle below that).
I'm not suggesting anything, just pointing that a system is going to have more theoretical bandwith available than you can possibly use.
Design around your use case, not around the idea of "maxxing out everything for the sake of maxxing it out".

Also, i wonder how those so many drives like 40-60 drives are connected in a rackmount server. Do they use something different than a HBA? I don't think any Motherboard will have so many onboard SATA ports.

SAS HBA connecting to SAS expanders. (In this situation, a 4*12 Gb/s link between the HBA and one expander may be shared by more than four drives.)

Hmm. Does it also has something like IT firmware or something?

A PLX switch is not a controller. It doesn't have a driver or "IT/IR firmware". To the OS, it's essentially an extra hop on the PCI path.
@NickF actually found that a PLX can increase performance of an all-NVMe pool by taking over bifurcation duties from the CPU.

Well, the workload i have, i need one single one where i can do faster read/writes on the SSD pool and the bulk storage and rendering on the HDD

Do you mean bulk on the HDD and rendering on the SSD? Otherwise I don't know what the fast pool is intended for.
Anyway, since it's two pools, and two shares, it does not really matter whether they come from one or two systems.

 

[Fastline]

Capacity and cooling, which goes with better performance as well. I can totally understand that.

Yep!

Per lane means 4*12 Gb/s per SFF-8643 connector (4 lanes). 16*12 Gb/s for a -16i/16e HBA (though the controller may throttle below that).

Umm, so are you sure that the 12Gb/s bandwidth is per SAS port not all the ports combined together?

If that's the case, that means each Drive gets 3Gb/s which HDD can never use at all. So, would it still throttle? Seems like I'm missing something here.

Once this thing gets sorted, i can see what CPU will i need so that i don't run out of bandwidth. We'll come back to it later.

I'm not suggesting anything, just pointing that a system is going to have more theoretical bandwith available than you can possibly use.

Gotcha

Design around your use case, not around the idea of "maxxing out everything for the sake of maxxing it out".

Of course, of course. It's not about maxing out. But build one which can serve my needs :)

SAS HBA connecting to SAS expanders. (In this situation, a 4*12 Gb/s link between the HBA and one expander may be shared by more than four drives.)

Umm, and would that not throttle?

A PLX switch is not a controller. It doesn't have a driver or "IT/IR firmware". To the OS, it's essentially an extra hop on the PCI path.
@NickF actually found that a PLX can increase performance of an all-NVMe pool by taking over bifurcation duties from the CPU.

Will read that soon!

Do you mean bulk on the HDD and rendering on the SSD? Otherwise I don't know what the fast pool is intended for.

The 4 U.2 will be used for Metadata. The rest 4 drive will have a single RAID Z2 pool where i plan to store weekly temp data which needs fast access for recovery and restore of the medias and then weekly back to the HDD pool.

 

[Davvo]

Capacity and cooling, which goes with better performance as well. I can totally understand that.

I can't see how a U.2 drive would have better capacity and cooling, or performance, than a M.2 with heatsink

 

[Fastline]

I can't see how a U.2 drive would have better capacity and cooling, or performance, than a M.2 with heatsink

Why not. U.2 SSD will less heat, have space for cooling, no additional sink required and does not cost too much when you go for enterprise segment. My friend is having P5600 and I'll be getting four of that (each 6.4TB) and then 4xOptane 900p for Metadata.