Hardware upgrade questions

[dror]

Hey friends,
I have this system:
System: Supermicro 1028U-E1CR4+
CPU: 2 x Intel Xeon 8C E5-2620V4 2.1G 20M 8GT QPI
RAM: 4 x 16GB DDR4-2666 2RX8 ECC RDIMM (Total 64GB)
Disks: 6 x MZ7KM1T9HMJP-00005 Samsung SM863a,1.92TB,SATA 6Gb/s,VNAND,V48,2.5",7mm (3.6 DWPD)
HBA: AOC-S3008L-L8e HBA IT mode
LAN: AOC-STG-i4S 4-port10Gbe Standard LP with SFP+, Intel XL710-AM1

This is for my VT environment (XCP-NG) ISCSI.
I want to upgrade my RAM to 256GB and replace the Disk with this disk:
SAMSUNG PM1643 SAS Enterprise SSD 1.92 TB internal 2.5 inch SAS 12Gb/s TLC OEM (MZILT1T9HAJQ-00007)

Total 8 disks (4 group mirror for best performance ISCSI), This is a better option and I will see better performance?
Does Anyone Know if this Disk is Reliable and Durable for Multiple Writing to Disk? Is there a better alternative?

I also want to add SLOG device via this part:
Supermicro Add-on Card AOC-SLG3-2M2 https://www.supermicro.com/en/products/accessories/addon/AOC-SLG3-2M2.php

And connect it to the NVME m.2 INTEL Optane SSD DC P4801X 200GB M.2 110MM PCIe x4 3D Xpoint is this a good device to use as SLOG ? Is there a better alternative?

It will better if I connect it 2 X NVME m.2 INTEL Optane SSD DC P4801X 100GB M.2 110MM PCIe x4 3D XPoint ?
will it give more performance and more survival ? Or do i not need strip/mirror SLOG? what do you think ?
can I disable one CPU and still upgrade the RAM and use this "Supermicro Add-on Card AOC-SLG3-2M2" ?


Thanks.

 

[dror]

Anyone?

 

[Bozon]

Did you read this thread? https://www.ixsystems.com/community/threads/slog-benchmarking-and-finding-the-best-slog.63521/ It seems like a good place to start.

 

[Bozon]

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you...

www.ixsystems.com

It appears this is now in the resources section. Hit the join discussion button.

 

[dror]

I read a lot about SLOG and would still love to advise according to my situation and details.
I would love if you could give your opinion.

 

[Bozon]

I am not a hardware expert. My opinion would be to test both ways with the benchmarking command that was provided in the thread, and then choose the better of the 2. Also, post your results in the thread of both ways so that people can learn from you. The test that is indicated precludes the device being set up as a SLOG. So put the device in the way you think will be slowest. Test, then change the arrangement, and then test again. If this way is fastest then leave it that way and use it as a SLOG. If it isn't then switch it back.

 

[HoneyBadger]

There's a lot of questions here, so let's hit them one at a time.

Upgrading 64GB to 256GB of RAM
Good choice, you'll have much better read performance from the quadrupled ARC size.

Drive configuration (4x2-way mirrors)
Mirrors will provide the best performance for random I/O workloads like VMs over iSCSI.

SM863a vs PM1643
The PM drives are much newer and should provide better overall performance, but it's important to note that they have a lower DWPD rating - they are rated for a single drive write per day vs. the 3.6x of the older SM drives. Make sure that this is accounted for in your usage, and monitor the SMART statistics for the media wearout indicator.

Optane as SLOG
Short of using NVRAM or NVDIMM devices, Optane is pretty much the best you can get. The P4801X is a good option since it is less costly than the P4800X series cards, while still providing good performance.

Single SLOG vs Mirrored SLOG
This question is about "how long will it take to replace a failed SLOG" and "how significant is the impact of lost performance or data"? Consider if you have only a single SLOG device, and it fails, your write speeds will be reduced to your pool speed. With SAS SSD that won't be nearly as punishing as with spinning disks, but you likely will still notice a significant performance hit. If this array can be easily powered off and have the failed SLOG replaced within four hours, you may be able to go with a single, larger, faster SLOG like the P4801X 200G or even a single P4800X 375G (if it's in the budget) - but if you will need 24h+ or longer to make that downtime happen, you will likely want to go with mirrored 100G P4801X sticks, so that the loss of a single stick doesn't cause the performance loss for a long period of time.

Can you disable one CPU and still use all that RAM?
Each CPU has 12 DIMM slots - you would have to use 32GB or larger sticks to get 256GB in that space. But my concern would be that in a 1U system you are likely using PCIe lanes from both CPUs to fill those three slots, which is what you have (HBA, 10GbE, NVMe card)

 

[Bozon]

There's a lot of questions here, so let's hit them one at a time.
Single SLOG vs Mirrored SLOG
This question is about "how long will it take to replace a failed SLOG" and "how significant is the impact of lost performance or data"? Consider if you have only a single SLOG device, and it fails, your write speeds will be reduced to your pool speed. With SAS SSD that won't be nearly as punishing as with spinning disks, but you likely will still notice a significant performance hit. If this array can be easily powered off and have the failed SLOG replaced within four hours, you may be able to go with a single, larger, faster SLOG like the P4801X 200G or even a single P4800X 375G (if it's in the budget) - but if you will need 24h+ or longer to make that downtime happen, you will likely want to go with mirrored 100G P4801X sticks, so that the loss of a single stick doesn't cause the performance loss for a long period of time.

I thought his question was about making his 2 drives a stripe pair to get the maximum performance possible, if this is actually something that mattered. I had no idea, so I suggested the time honored solution of testing both setups, and dispensing with the hand waving.

 

[HoneyBadger]

I thought his question was about making his 2 drives a stripe pair to get the maximum performance possible, if this is actually something that mattered. I had no idea, so I suggested the time honored solution of testing both setups, and dispensing with the hand waving.

Ah, must have glossed over that part. With SSDs, the larger drives tend to have better performance, so the single 200G drive might still be faster than 2x100G in a stripe, without adding in the complexity of having a striped vdev as your "safe write location."

 

[Bozon]

Ah, must have glossed over that part. With SSDs, the larger drives tend to have better performance, so the single 200G drive might still be faster than 2x100G in a stripe, without adding in the complexity of having a striped vdev as your "safe write location."

Sure, but I still contend that testing and sharing would benefit mankind. Or at least, FreeNasKind. :)

 

[dror]

There's a lot of questions here, so let's hit them one at a time.

Upgrading 64GB to 256GB of RAM
Good choice, you'll have much better read performance from the quadrupled ARC size.

Drive configuration (4x2-way mirrors)
Mirrors will provide the best performance for random I/O workloads like VMs over iSCSI.

SM863a vs PM1643
The PM drives are much newer and should provide better overall performance, but it's important to note that they have a lower DWPD rating - they are rated for a single drive write per day vs. the 3.6x of the older SM drives. Make sure that this is accounted for in your usage, and monitor the SMART statistics for the media wearout indicator.

Optane as SLOG
Short of using NVRAM or NVDIMM devices, Optane is pretty much the best you can get. The P4801X is a good option since it is less costly than the P4800X series cards, while still providing good performance.

Single SLOG vs Mirrored SLOG
This question is about "how long will it take to replace a failed SLOG" and "how significant is the impact of lost performance or data"? Consider if you have only a single SLOG device, and it fails, your write speeds will be reduced to your pool speed. With SAS SSD that won't be nearly as punishing as with spinning disks, but you likely will still notice a significant performance hit. If this array can be easily powered off and have the failed SLOG replaced within four hours, you may be able to go with a single, larger, faster SLOG like the P4801X 200G or even a single P4800X 375G (if it's in the budget) - but if you will need 24h+ or longer to make that downtime happen, you will likely want to go with mirrored 100G P4801X sticks, so that the loss of a single stick doesn't cause the performance loss for a long period of time.

Can you disable one CPU and still use all that RAM?
Each CPU has 12 DIMM slots - you would have to use 32GB or larger sticks to get 256GB in that space. But my concern would be that in a 1U system you are likely using PCIe lanes from both CPUs to fill those three slots, which is what you have (HBA, 10GbE, NVMe card)

Thanks for your response!
What about 905p 380 m.2 ? I see this is the best device for SLOG but its not enterprise device, What you think ? Its safe to use it? It's really the best for my situation?
I'm going to mirror because DB VMs not love to loss data, Even if it's for 5 seconds :)
You right about the DWPD, You may have a better alternative (SAS SSD 12Gbps) ?
Again, You right.

Thanks!

 

[HoneyBadger]

The use of this storage for a transaction DB makes me plan cautiously.

If you plan to use this for a business/enterprise setting I would suggest sticking with the P4801X cards - the 905p might be faster, but Intel's warranty wording specifically states that they won't cover the 900p/905p if they are used outside of a single-user/workstation scenario.

You want mirrored SLOG for a similar reason - losing data on a DB isn't generally allowed, so you want to reduce risk as far as possible. ;)

If you need to have 3 DWPD in SAS format consider the Samsung PM1635A, Toshiba PX04SV/PX05SV, HGST SSD1600MM, or WD DC SS530/SS540 (you can get varying levels of DWPD ratings on these) but consider how heavily you will use the database itself. You can also continue using the SM863a drives, unless there's a reason you're moving away?

1 DWPD means what it says - one drive write per day. Do you plan to write 7.68T (1.92x4) to your XCP-NG VMs? That's definitely a "large business" level of transactions; if you're a smaller shop I would wager you'll be more than fine with the 1 DWPD that the PM1643 allows you, but only you know the intensity of your workload. Whatever you buy, keep an eye on the wearout indicators and replace drive proactively rather than reactively (don't wait for 10% or SMART errors; cycle them out somewhat fresh.)

 

[dror]

The use of this storage for a transaction DB makes me plan cautiously.

If you plan to use this for a business/enterprise setting I would suggest sticking with the P4801X cards - the 905p might be faster, but Intel's warranty wording specifically states that they won't cover the 900p/905p if they are used outside of a single-user/workstation scenario.

You want mirrored SLOG for a similar reason - losing data on a DB isn't generally allowed, so you want to reduce risk as far as possible. ;)

If you need to have 3 DWPD in SAS format consider the Samsung PM1635A, Toshiba PX04SV/PX05SV, HGST SSD1600MM, or WD DC SS530/SS540 (you can get varying levels of DWPD ratings on these) but consider how heavily you will use the database itself. You can also continue using the SM863a drives, unless there's a reason you're moving away?

1 DWPD means what it says - one drive write per day. Do you plan to write 7.68T (1.92x4) to your XCP-NG VMs? That's definitely a "large business" level of transactions; if you're a smaller shop I would wager you'll be more than fine with the 1 DWPD that the PM1643 allows you, but only you know the intensity of your workload. Whatever you buy, keep an eye on the wearout indicators and replace drive proactively rather than reactively (don't wait for 10% or SMART errors; cycle them out somewhat fresh.)

I don't think with at least 10DWPD (905p 380GB) it will fail any time soon.
The environment usually consists of web hosting servers that include databases.
I'm not expecting a load like the Giants (Google Cloud, Amazon Cloud etc) :)
I think, though, I'll go for the 905P because right now it's the fastest and relatively cheap.
The P4801X looks more reliable and of course they are an enterprise version but slower and insanely expensive!
You think for this environment I need 3/5DWPD OR P4801X? , I think 1DWPD and 905P its enough.
SM863a are excellent but since I already have a 12Gb / s controller I will already upgrade for better performance :)
What do you think ?

 

[HoneyBadger]

As mentioned in your other thread, have you identified the current drives as being a bottleneck? The higher bandwidth, deeper queues, and full-duplex nature of SAS drives can be beneficial vs SATA, but only if you're hitting those limits in your current config.

Remember that iSCSI doesn't usually request sync writes by default as well, so unless you've forced them on you may be seeing inflated performance from "writing to RAM."

 

[HoneyBadger]

Sorry I don't understand your answer :(
You see I can't upgrade the disks?

The question is "Are your current disks slowing you down?" If not, you won't get any faster by changing them. It's like putting Z-rated tires on an economy car.

 

[dror]

The question is "Are your current disks slowing you down?" If not, you won't get any faster by changing them. It's like putting Z-rated tires on an economy car.

Why? With the upgrade to RAM and 2 OPTANE 905P M.2 the server should be very fast right?

 

[HoneyBadger]

Why? With the upgrade to RAM and 2 OPTANE 905P M.2 the server should be very fast right?

Will adding new tires make an economy car into a Ferrari?
No, because the tires were not what made it slow.
By the same logic, unless your current SM863a drives are "slowing down" your system, replacing them will not change anything.
Add the RAM first. Add the SLOG devices if you are using sync writes. Then look at the sustained performance to consider if you need to replace the SSDs.