Resource icon

SLOG benchmarking and finding the best SLOG

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
Now that PCIe 4.0 is available we see ALOT of bandwidth, now it's not always the disk(arrays) that's the bottleneck anymore.
So how should one think when using 10+ PCIe 4.0 NVMe disks, is there even a SLOG disk option anymore?
Would a RMS-300 or any other highperformance NVMe disk help, or just slow the array down (when having sync writes that is)?

The only option i see i NVDIMM like Intel DCPMM or likewise.
Or would one be better of just running ZIL?
Any thouhts on this?
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
It will always depend on the use case.

An all-flash NAS certainly can get the job done quicker but a SLOG consisting of Optane or battery-backed NVDIMMs may still be justified in certain use cases. Not all flash is as fast as Optane. :smile:

I don't consider PCIe 4.0 some sort of savior, ditto the number of bus lanes. It comes down to how the system is laid out / implemented and how said implementation meets the needs of the customer. It's why I switched from the Atom series to the Intel D-15xx series of microprocessors. The latter offered much better flexibility for the future, in part by having more bus lanes for things like a dedicated m.2 PCIe 3.0 x4 interface, which was perfect for my SLOG. None of the Atom boards have a dedicated (not muxed) interface like that simply because the Atom boards lack the lanes.

PCIe 4.0 can help mitigate the lane issue by offering more speed per bus lane (i.e. allowing a PCI4 x 2 to operate as fast as a PCI 3.0 x 4) but between the premiums, etc. for the time being, a higher bus lane quantity PCIe 3 system is likely more cost effective.
 

ehsab

Dabbler
Joined
Aug 2, 2020
Messages
45
It will always depend on the use case.

An all-flash NAS certainly can get the job done quicker but a SLOG consisting of Optane or battery-backed NVDIMMs may still be justified in certain use cases. Not all flash is as fast as Optane. :smile:

I don't consider PCIe 4.0 some sort of savior, ditto the number of bus lanes. It comes down to how the system is laid out / implemented and how said implementation meets the needs of the customer. It's why I switched from the Atom series to the Intel D-15xx series of microprocessors. The latter offered much better flexibility for the future, in part by having more bus lanes for things like a dedicated m.2 PCIe 3.0 x4 interface, which was perfect for my SLOG. None of the Atom boards have a dedicated (not muxed) interface like that simply because the Atom boards lack the lanes.

PCIe 4.0 can help mitigate the lane issue by offering more speed per bus lane (i.e. allowing a PCI4 x 2 to operate as fast as a PCI 3.0 x 4) but between the premiums, etc. for the time being, a higher bus lane quantity PCIe 3 system is likely more cost effective.
But the slog device should be faster then the array of nvme disks right?`What kind of a device is faster then 10 disk striped mirrors?
With PCIe Gen 4 the lanes are the same amount, just more bandwidth in each lane.
The latency is ofcourse lower on NVDIMM, perhaps thats the benefit for the SLOG then havinga ZIL.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
Not all flash is "fast"... as the SLOG benchmarks here have proven time and again. How the pool is organized is another factor (hence my comment "how the system is laid out"). Inexpensive flash tends to be less-performant and the use of striped anything for applications other than scratch would concern me. Speaking of scratch, unless the data has to be accessed by multiple users at once, I prefer local scratch per a DAS vs. scratch on a NAS. But I also understand that lots of folk are happy with striped mirrored arrays in business-sensitive applications because the performance boost is so significant over a Zx-VDEV pool. All comes down to risk vs. reward (with a hopefully healthy dose of contingency planning thrown in)

As for PCIe 4, currently only one of the two major FreeNAS hardware platforms (AMD) supports PCIe4 and Intel likely will continue to drag its heels as they frantically try to address the more pressing issue in their business, i.e. the inability of their former core-competency fabs to scale the circuits footprints down and yet produce usable yields. The current disconnect between TSMC and like fabs vs. Intel is striking and if Intel cannot figure it out, it's likely that Intel will exit the fab business altogether. For the sake of all of us, I hope that does not happen as the shrinking competition of top-tier CPU fabs will invariably lead to rent-seeking and/or oligopolistic behavior just as we saw in the memory and like businesses.

Anyhow, all-flash arrays are a rarified world that I have little insight into. Once storage becomes that fast, other factors such as the CPU, network adapter(s), and so on may become a gating factor again. I'll chip away here at my array with a special VDEV when the time comes to boost metadata performance and to a lesser extent small file performance (TrueNAS 12).
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
What are you trying to do with your SLOG configuration here? If you want more performance, use better devices; lose the complicated multi-split setup.

Optane 16G sticks are okay, but the SATA SSDs are not suitable SLOG devices, A400s do not have the endurance or performance for this. You're also sharing them between multiple pools which will degrade things further. Having ZFS flip between the two of them for SLOG may also be resulting in some inconsistent throughput depending on where your records land.

The VM pool is also a RAIDZ1 and parity performs poorly for block/small-recordsize IO. Mirrors are strongly suggested for that use case.


Hello,

First thanks for the reply.
Well I was just testing for posterity how different type of disks/setups would perform, and to get mine finely tuned and got baffled by low performance on SSD side.

1st I know SATA SSD´s are not the thing for a SLOG job, especially not TLC´s, have some MLC´s at thand but performance was almost the same.
Was a little baffled to see 60MB/S on when mounted as SLOG on disks that normally achive 550MB/s easily, but I guess this is because using them as SLOG circumvents onboard RAM for data security purpose and thus you only get 10% of the actual disk performance.
Optane on the other hand surpasses the 130MB/s specified for 16G sticks and reaches around 140-160 MB/s in my case as it does not have RAM and does not need a battery in order to ACK data as written(dunno how bbu m.2 would perform)

2nd The benchmarks where done to see from a performance point of view which is better for SLOG Optane or NVME (as in the end will end up with 900p/800P or some capacitor backed SSD´s eg. Intel p3700/Seagate nytro 5000 most probably) but seeing SSD performance, I think Optane is the safest bet.


3rd, As I only have 2 nvme ports so I have to go along with multi split setup, as I do not want to dedicate an fully fledged server for NAS purpose, this was built to house my storage arrays (around 25 disks- 100TB rough capacity).
Tests where done only activating sync writes on one array although disks where partitioned and assigned as multi split.
Test where done to see if this would be enough for 10-20G throughput which 16G optane is lacking in performane it seems, the 800p would be a good match for 8G roughly so think that would be the sweet spot for me price/performance or if there are better options please let me know.

Looking for something on M.2 and not extremely expensive...
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Hello,

First thanks for the reply.
Well I was just testing for posterity how different type of disks/setups would perform, and to get mine finely tuned and got baffled by low performance on SSD side.

Fair enough; I suppose this is still the "SLOG benchmarking thread" after all.

1st I know SATA SSD´s are not the thing for a SLOG job, especially not TLC´s, have some MLC´s at thand but performance was almost the same.
Was a little baffled to see 60MB/S on when mounted as SLOG on disks that normally achive 550MB/s easily, but I guess this is because using them as SLOG circumvents onboard RAM for data security purpose and thus you only get 10% of the actual disk performance.
Optane on the other hand surpasses the 130MB/s specified for 16G sticks and reaches around 140-160 MB/s in my case as it does not have RAM and does not need a battery in order to ACK data as written(dunno how bbu m.2 would perform)

Correct; the SLOG workload follows the writes by a "flush cache" command. Drives with capacitors or other power-loss-protection circuitry (or "no DRAM at all" like the Optane cards) get to reply to that command instantly with "the data is non-volatile already" because their internal firmware will handle flushing the RAM to NAND. SSDs without it unfortunately have to actually push their data to NAND which is varying degrees of painful.

2nd The benchmarks where done to see from a performance point of view which is better for SLOG Optane or NVME (as in the end will end up with 900p/800P or some capacitor backed SSD´s eg. Intel p3700/Seagate nytro 5000 most probably) but seeing SSD performance, I think Optane is the safest bet.

Optane is the winner right now because of the underlying 3D XPoint tech. It's stupidly fast when comparing pure write speed/latency and also has virtually no penalty for concurrent R/W's which is a massive advantage for metadata/special vdevs. Other NVMe devices like the P3700 or NVRAM cards like the RMS-200 are also viable - and I recall there's an RMS-375 mentioned a few pages back that is U.2 form factor - but the former is outclassed by Optane and the latter are SLOG-specific due to sizing.

3rd, As I only have 2 nvme ports so I have to go along with multi split setup, as I do not want to dedicate an fully fledged server for NAS purpose, this was built to house my storage arrays (around 25 disks- 100TB rough capacity).
Tests where done only activating sync writes on one array although disks where partitioned and assigned as multi split.
Test where done to see if this would be enough for 10-20G throughput which 16G optane is lacking in performane it seems, the 800p would be a good match for 8G roughly so think that would be the sweet spot for me price/performance or if there are better options please let me know.

The "enterprise" option in M.2 is the P4801X but you'll have a very hard time saturating a 10G/20G link with VM traffic since that primarily pushes 4K/8K records. Larger I/O happens on svMotion and big copy operations but for general work inside VMs or small-record benchmarking there's more per-record overhead that needs to be accounted for.

Looking for something on M.2 and not extremely expensive...

You won't likely find a device in M.2 that can saturate 10/20Gbps with VM traffic at any price. The Optane P4801X family will get the closest and should suffice for file copying due to the larger recordsize in play. But that may or may not violate the "expensive" category if you're looking to buy two.

You could use the 800p or similar; SLOG partitions don't need to be large (4G with default tunables) and Optane does fairly well at concurrent workloads, but you'll still be splitting the peak bandwidth between all pools pointing to it.

Really hoping for some less expensive NVMe devices to bring full namespace/QOS support, but that's likely a pipe dream as most consumer workloads don't need that, and attaching "enterprise" to a drive name is a good reason to pump the cost up tenfold.
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
Fair enough; I suppose this is still the "SLOG benchmarking thread" after all.



Correct; the SLOG workload follows the writes by a "flush cache" command. Drives with capacitors or other power-loss-protection circuitry (or "no DRAM at all" like the Optane cards) get to reply to that command instantly with "the data is non-volatile already" because their internal firmware will handle flushing the RAM to NAND. SSDs without it unfortunately have to actually push their data to NAND which is varying degrees of painful.



Optane is the winner right now because of the underlying 3D XPoint tech. It's stupidly fast when comparing pure write speed/latency and also has virtually no penalty for concurrent R/W's which is a massive advantage for metadata/special vdevs. Other NVMe devices like the P3700 or NVRAM cards like the RMS-200 are also viable - and I recall there's an RMS-375 mentioned a few pages back that is U.2 form factor - but the former is outclassed by Optane and the latter are SLOG-specific due to sizing.

Thanks for clearing what I was alredy suspecting, anyway SSD is a bad choice due to endurance, the only thing that I worked with that where relatively safe on endurance Fusion-IO drives but they are still nand based.

The "enterprise" option in M.2 is the P4801X but you'll have a very hard time saturating a 10G/20G link with VM traffic since that primarily pushes 4K/8K records. Larger I/O happens on svMotion and big copy operations but for general work inside VMs or small-record benchmarking there's more per-record overhead that needs to be accounted for.

This looks pretty nice, I haven't had the pleasure of working with them, but it look the 375G one can pus an obscene write performance (2200 MB/s according to ARK, and knowing intel they might reach 2500 as they spec them 10-20% bellow the real throughput) so I think they can saturate 20gigabits or be on the limit of doing so.

You won't likely find a device in M.2 that can saturate 10/20Gbps with VM traffic at any price. The Optane P4801X family will get the closest and should suffice for file copying due to the larger recordsize in play. But that may or may not violate the "expensive" category if you're looking to buy two.

They are a little over budget (and that would not be the main issue)and although they are theoretically m.2 they are actually U.2 and I have runned out of phisical space for drives in my case, only way of putting them in is sacrificing boot SSD´s and moving OS to usb thumb-drives which kept failing on me in the past and are quite slow on boot for Truenas/BSD for some obscure reason.

You could use the 800p or similar; SLOG partitions don't need to be large (4G with default tunables) and Optane does fairly well at concurrent workloads, but you'll still be splitting the peak bandwidth between all pools pointing to it.
Yeah, that was the initial plan, but found the 16G ones for 11E/piece shipping included, and wanted to try them out, and do some benchmarking.
They seem ideal for 1-2gig setups so far, and extremely cheap at that too.
The 112G 800P can push roughly 650MB/s according to ARK and I doubt will need more than that in my home lab(I suspect the actual max would be 700-800MB/s but intel always specs them bellow for some reason). And a pair of them delivered will only set me back around 160E compared to the 600-700E for a pair of P4801X


Really hoping for some less expensive NVMe devices to bring full namespace/QOS support, but that's likely a pipe dream as most consumer workloads don't need that, and attaching "enterprise" to a drive name is a good reason to pump the cost up tenfold.


Yeah, we´ll see, but I doubt it on consumer side.
On DC side I worked with Samsung drives, but they where popping like Christmas tree lights was replacing them by the dozens in servers with I/O loads, would not trust them in my home setups :D.
Fusion-IO and Warp drives on the other hand seemed ok but they are expensive and slow for current standards.

Saw the 2tb nytro going for roughly 150-200E(200$+ shipping) on E-bay and seemed a nice drive but I do not know it´s durability, they state 1.5 drive write/day and are MLC, only down side is they are 22110 so one is ok for my board, the other one will have to "jerryrig" it as I do have the spacing to fit it, but not the screw hole, :)
Also do not know if they actually work in consumer boards... has anyone tried to put enterprise NVME on modern consumer boards?



But I´m a little skeptic about endurance/actual speed
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Thanks for clearing what I was alredy suspecting, anyway SSD is a bad choice due to endurance, the only thing that I worked with that where relatively safe on endurance Fusion-IO drives but they are still nand based.

All SSDs aren't necessarily bad for it - just those specific Kingston ones are. I have old HGST SAS drives that took 5PB of writes before gracefully letting me know via a SMART flag that they had another 500T or so to go. Optane just happens to have even higher endurance numbers, but as you can see from previous users, some write workloads could even cook those. At that stage is where you need NVRAM or NVDIMMs.

This looks pretty nice, I haven't had the pleasure of working with them, but it look the 375G one can pus an obscene write performance (2200 MB/s according to ARK, and knowing intel they might reach 2500 as they spec them 10-20% bellow the real throughput) so I think they can saturate 20gigabits or be on the limit of doing so.

Note that the "rated speed" on manufacturer sheets is often done with big block sizes, high levels of both parallelism, and deep queue depth; while recordsize can vary by workload (VMs generate small, files generate large, generally speaking) neither of the last two is seen in great amounts for an SLOG workload. You need to look more at things like "4K QD 1" or "sustained random write IOPS" to find a number closer to what you can expect from it in an SLOG role for virtual machines; you might see closer to the datasheet numbers for big records like media files or 1M recordsizes.

They are a little over budget (and that would not be the main issue)and although they are theoretically m.2 they are actually U.2 and I have runned out of phisical space for drives in my case, only way of putting them in is sacrificing boot SSD´s and moving OS to usb thumb-drives which kept failing on me in the past and are quite slow on boot for Truenas/BSD for some obscure reason.

P4800X is HHHL/U.2 but the P4801X is definitely available in NVMe M.2 format. Might be low availability now though.

Yeah, that was the initial plan, but found the 16G ones for 11E/piece shipping included, and wanted to try them out, and do some benchmarking.
They seem ideal for 1-2gig setups so far, and extremely cheap at that too.
The 112G 800P can push roughly 650MB/s according to ARK and I doubt will need more than that in my home lab(I suspect the actual max would be 700-800MB/s but intel always specs them bellow for some reason). And a pair of them delivered will only set me back around 160E compared to the 600-700E for a pair of P4801X

16G/32G Optane cards are excellent for cheap options at the 1-2Gbps range, you just have to bear in mind their rated endurance is 365TBW - most home setups won't push nearly that, but they could quickly run out of P/E cycles if asked to hold up an enterprise workload. The larger devices will fare better in both endurance and speed but they're still technically a "consumer" drive so Intel won't warranty it if it fails under than circumstance.

Yeah, we´ll see, but I doubt it on consumer side.
On DC side I worked with Samsung drives, but they where popping like Christmas tree lights was replacing them by the dozens in servers with I/O loads, would not trust them in my home setups :D.
Fusion-IO and Warp drives on the other hand seemed ok but they are expensive and slow for current standards.

I haven't had an issue with Samsung's datacenter drives per se, but I've had more consistent performance results from HGST/Intel/Toshiba. Aged FusionIO is a neat thing to have, but modern SSDs have evolved quite a bit.

Saw the 2tb nytro going for roughly 150-200E(200$+ shipping) on E-bay and seemed a nice drive but I do not know it´s durability, they state 1.5 drive write/day and are MLC, only down side is they are 22110 so one is ok for my board, the other one will have to "jerryrig" it as I do have the spacing to fit it, but not the screw hole, :)
Also do not know if they actually work in consumer boards... has anyone tried to put enterprise NVME on modern consumer boards?


But I´m a little skeptic about endurance/actual speed

NVMe is just a protocol/standard, older consumer boards may not boot from it but they'll generally work fine. I threw an Optane stick on an adaptor card into a Xeon-5600 based machine and it recognised it fine. I would be mildly skeptical about the numbers on the Nytro 5000 but the images do show the physical capacitors for PLP so it's not unreasonable to expect that it can do a good job from a performance basis. Bear in mind the previous caveats about recordsize as well though - recordsize tends to have a direct correlation with effective throughput.

Regarding endurance, 1.5 DWPD is definitely entry-level "mixed-use" though - write-intensive devices tend to start at 10DWPD and climb from there. You could just buy the cheap Optane 16G sticks and replace them periodically as they start to get close to their rated life. If they aren't fast enough, step up to the 32G or even the 800p 58G/118G.
 
Last edited:

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
All SSDs aren't necessarily bad for it - just those specific Kingston ones are. I have old HGST SAS drives that took 5PB of writes before gracefully letting me know via a SMART flag that they had another 500T or so to go. Optane just happens to have even higher endurance numbers, but as you can see from previous users, some write workloads could even cook those. At that stage is where you need NVRAM or NVDIMMs.

Yes, but enterprise drives are used and have little life at this point, or expensive.
And consumer flash does not have PLP, and tends to be TLC(I personally only own 3 SSD´s that are TLC, the kingstone you saw).
I bought them in the past, but had a couple of Toshiba´s dye on me from usage so not touching them again.
I have 2 OCZ MLC´s that are still chugging along after years of usage and some, but is not the same as using them for SLOG.
The optaneyu´we seen have already over 1tb written in 1 week....


P4800X is HHHL/U.2 but the P4801X is definitely available in NVMe M.2 format. Might be low availability now though.

Just seen that they are present in 110 and 2.5, was thrown off course as I hve seen all had 2.5inch stipulated on ARC and only saw 2.5 inch images while seaching for them, but they have 2 versions of each model.

16G/32G Optane cards are excellent for cheap options at the 1-2Gbps range, you just have to bear in mind their rated endurance is 365TBW - most home setups won't push nearly that, but they could quickly run out of P/E cycles if asked to hold up an enterprise workload. The larger devices will fare better in both endurance and speed but they're still technically a "consumer" drive so Intel won't warranty it if it fails under than circumstance.

Well if I leave them to the task they should chug along for around 3-5 years under current usage which is not terrible by my standards...
Think will buy 2-4 more just for kicks, bought them from china and they arrived in 2 weeks pretty amazing given current circumstances....

I haven't had an issue with Samsung's datacenter drives per se, but I've had more consistent performance results from HGST/Intel/Toshiba. Aged FusionIO is a neat thing to have, but modern SSDs have evolved quite a bit.

Depends how many you´we seen, used to handle an excess of 400k-1000k servers so I´we seen quite a lot, even dead Fusion and Optane drives.
But had an influx of dead SAMSUNG 2tb NVME so... after replacing around 1K you can understand I do not trust them any more....
Although only one model faild me so far, it still left a deep scar, as NVME do not have drive location light and vanish from OBM once failed,
Had to sent DC tech guys the S/N and they had to power down and manually look for them in servers that had literally dozens of them.

NVMe is just a protocol/standard, older consumer boards may not boot from it but they'll generally work fine. I threw an Optane stick on an adaptor card into a Xeon-5600 based machine and it recognised it fine. I would be mildly skeptical about the numbers on the Nytro 5000 but the images do show the physical capacitors for PLP so it's not unreasonable to expect that it can do a good job from a performance basis. Bear in mind the previous caveats about recordsize as well though - recordsize tends to have a direct correlation with effective throughput.

I know NVME is a protocol and should work, thing is I never tried enterprise grade PLP drives on consumer boards, so I just asked to double check.
As enterprise ones go in to canisters connected to carrier boards on the server front, so had no idea if they have some additional stuff except PLP.
The only thing driving me nuts, is that enetrprise uses 20110/22110 and consumer uses 2080, consumer boards mostly have one 22110 slot and the rest are 2280....


Regarding endurance, 1.5 DWPD is definitely entry-level "mixed-use" though - write-intensive devices tend to start at 10DWPD and climb from there. You could just buy the cheap Optane 16G sticks and replace them periodically as they start to get close to their rated life. If they aren't fast enough, step up to the 32G or even the 800p 58G/112G.
[/QUOTE]

Yes, I know, it´s pretty mainstream... Nytro is they´re prosumer range not actual top server grade, but rather SOHO and is cheap for some reason.
I think that for ZFS SLOG for home usage 58G OPTANE would be the sweet spot for 10G, but 112G are cheaper for some reason.
I say this as not all writes are Synk ones, and generally you do not have big workloads on the storage side like in the enterprise where you would actually need 10-100gig links, and if you have some wild expectations just build all flash :D and forget about the dyeing spinning rust altogether :)

In my case I have 2 pools one built form HGST Enterprise SAS, and the other with WD RED´s, and use them for VM´s and SMB/NFS shares.
Hybrid pools seem to work nicely so far, I activated dedup for the first time and it works without a big RAM/performance hit, and metadata SSD accelerate writes a bit.
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Yes, but enterprise drives are used and have little life at this point, or expensive.
And consumer flash does not have PLP, and tends to be TLC(I personally only own 3 SSD´s that are TLC, the kingstone you saw).
I bought them in the past, but had a couple of Toshiba´s dye on me from usage so not touching them again.
I have 2 OCZ MLC´s that are still chugging along after years of usage and some, but is not the same as using them for SLOG.
The optaneyu´we seen have already over 1tb written in 1 week....

Well if I leave them to the task they should chug along for around 3-5 years under current usage which is not terrible by my standards...
Think will buy 2-4 more just for kicks, bought them from china and they arrived in 2 weeks pretty amazing given current circumstances....

If they're doing the job on performance, and you can tolerate a once-per-year planned downtime, just buy a half-dozen and cycle through them. There's your "wear-leveling" ;)

Just seen that they are present in 110 and 2.5, was thrown off course as I hve seen all had 2.5inch stipulated on ARC and only saw 2.5 inch images while seaching for them, but they have 2 versions of each model.

...

I know NVME is a protocol and should work, thing is I never tried enterprise grade PLP drives on consumer boards, so I just asked to double check.
As enterprise ones go in to canisters connected to carrier boards on the server front, so had no idea if they have some additional stuff except PLP.
The only thing driving me nuts, is that enetrprise uses 20110/22110 and consumer uses 2080, consumer boards mostly have one 22110 slot and the rest are 2280....

The 22110 form factor does throw a wrench in the works for a lot of adaptors and motherboards certainly. SuperMicro's adaptor card fits them, as do some StarTechs, but the super-cheap Shenzhen specials cap out at 2280.

Depends how many you´we seen, used to handle an excess of 400k-1000k servers so I´we seen quite a lot, even dead Fusion and Optane drives.
But had an influx of dead SAMSUNG 2tb NVME so... after replacing around 1K you can understand I do not trust them any more....
Although only one model faild me so far, it still left a deep scar, as NVME do not have drive location light and vanish from OBM once failed,
Had to sent DC tech guys the S/N and they had to power down and manually look for them in servers that had literally dozens of them.

Consistent failure of a specific model usually makes me think there's bad firmware in play. Once bitten, twice shy for sure - one thousand times bitten, "screw that drive and the horse it rode in on."

Yes, I know, it´s pretty mainstream... Nytro is they´re prosumer range not actual top server grade, but rather SOHO and is cheap for some reason.
I think that for ZFS SLOG for home usage 58G OPTANE would be the sweet spot for 10G, but 112G are cheaper for some reason.
I say this as not all writes are Synk ones, and generally you do not have big workloads on the storage side like in the enterprise where you would actually need 10-100gig links, and if you have some wild expectations just build all flash :D and forget about the dyeing spinning rust altogether :)

At those prices, I'd be considering trying to find a way to rig up massive numbers of M.2 adaptors and using them as the vdev storage.

In my case I have 2 pools one built form HGST Enterprise SAS, and the other with WD RED´s, and use them for VM´s and SMB/NFS shares.
Hybrid pools seem to work nicely so far, I activated dedup for the first time and it works without a big RAM/performance hit, and metadata SSD accelerate writes a bit.

Be careful still with deduplication; hybrid pools don't eliminate the RAM hunger, it just makes it less painful because your spillover if you run out is now SSD vs "going back to spinning disks"

But I've mentioned this before and I'll say it again; hybrid pools and meta-on-SSD are going to be big news, especially for things like "VMs on spinning disk" - personally, I'd grab a few more of those 118G Optane sticks before everyone realises how awesome they are for special vdevs.
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
If they're doing the job on performance, and you can tolerate a once-per-year planned downtime, just buy a half-dozen and cycle through them. There's your "wear-leveling" ;)

You read my mind :)

The 22110 form factor does throw a wrench in the works for a lot of adaptors and motherboards certainly. SuperMicro's adaptor card fits them, as do some StarTechs, but the super-cheap Shenzhen specials cap out at 2280.

Unfortunately yes... anything beyond 2280 is a pain for now, even my threadripper board which in theory is prosumer and closer so server HW only has one 1x22110 slot and 2x2280

But saw some cheap ones:

Consistent failure of a specific model usually makes me think there's bad firmware in play. Once bitten, twice shy for sure - one thousand times bitten, "screw that drive and the horse it rode in on."
They did release a FW to correct that after 1 year of constant failing drives sigh... but it left a bad taste, HP as well had the whole sudden fail shananigan, and had to patch all servers... and so on... if I recall correctly even intel had a whole 3700 batch that went bad and we replaced them in droves...
All in all had seen a lot of SSD issues on DC side, that is why if I see a cheap DC standard ssh I always ask for people who had XP with that particular model...

At those prices, I'd be considering trying to find a way to rig up massive numbers of M.2 adaptors and using them as the vdev storage.
Thought o that myself, but I do not see a way without an actual server chassis with the whole carrier board shenanigan. And for the moment those things cost at least a couple of good kidney´s.

Be careful still with deduplication; hybrid pools don't eliminate the RAM hunger, it just makes it less painful because your spillover if you run out is now SSD vs "going back to spinning disks"

That is why I´m experimenting with one pool, have not noticed any negative impact so far with dedup set on verify and 2x480 SSD´s as dedup disks.
Metadata, have dedicated the other pair of 480(here I have put Sandisk PRO MLC´s just for good measure of safety)

But I've mentioned this before and I'll say it again; hybrid pools and meta-on-SSD are going to be big news, especially for things like "VMs on spinning disk" - personally, I'd grab a few more of those 118G Optane sticks before everyone realises how awesome they are for special vdevs.

Exactly that is why I´m already experimenting and thus far I see good results....

As for the 118, if we can sum up to 10drives we can order them at 65EURO a pop +2EURO on delivery for each batch depending how you bundle them(in my case would be individually, to avoid import fees)
 
Last edited:

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
So far so good, funny thing is dedup data is bigger then metadata :oops::eek:
Will let you know how dedup works in the long run, but all seem fine for now not a real impact so far, it´s set on verify, so the most punishing:

capacity operations bandwidth
pool alloc free read write read write
---------------------------------------------- ----- ----- ----- ----- ----- -----
Archive 1.40T 5.85T 5 7 1.40M 782K
gptid/62f66461-b2c5-11e9-af38-902b34d39495 1.40T 5.85T 5 7 1.40M 782K
---------------------------------------------- ----- ----- ----- ----- ----- -----
SAS 3.43T 52.5T 79 302 721K 10.6M
raidz2 3.41T 51.2T 58 158 635K 6.45M
gptid/50cad0f1-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 661K
gptid/511b4e3f-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.4K 660K
gptid/51294ecd-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 660K
gptid/517b5371-d0d7-11ea-8717-2cf05d07d39b - - 5 16 63.4K 660K
gptid/51907b32-d0d7-11ea-8717-2cf05d07d39b - - 5 16 63.5K 660K
gptid/51924d04-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 660K
gptid/51a8084b-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.6K 660K
gptid/5189a1b0-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 660K
gptid/51df64e8-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 661K
gptid/51e871f0-d0d7-11ea-8717-2cf05d07d39b - - 5 15 63.5K 660K
dedup - - - - - -
gptid/85c84ce9-e23b-11ea-ba69-2cf05d07d39b 7.14G 437G 15 57 60.4K 1.00M
gptid/863904ee-e23b-11ea-ba69-2cf05d07d39b 3.02G 441G 6 16 24.7K 390K
special - - - - - -
mirror 2.13G 442G 0 38 1.55K 591K
gptid/ded6ac5d-e224-11ea-ba69-2cf05d07d39b - - 0 19 792 295K
gptid/deea7ff4-e224-11ea-ba69-2cf05d07d39b - - 0 19 793 295K
logs - - - - - -
mirror 2.66M 6.50G 0 31 1 2.18M
gptid/dfd4c43d-e065-11ea-8ad9-2cf05d07d39b - - 0 15 0 1.09M
gptid/e26ced7c-e065-11ea-8ad9-2cf05d07d39b - - 0 15 0 1.09M
---------------------------------------------- ----- ----- ----- ----- ----- -----


1598484434240.png



Hmm, for some reason pool resilvered after adding special vdev and dedup:

pool: SAS
state: ONLINE
scan: resilvered 1.64M in 00:00:00 with 0 errors on Wed Aug 19 16:05:09 2020
config:

NAME STATE READ WRITE CKSUM
SAS ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/50cad0f1-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/511b4e3f-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51294ecd-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/517b5371-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51907b32-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51924d04-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51a8084b-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/5189a1b0-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51df64e8-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
gptid/51e871f0-d0d7-11ea-8717-2cf05d07d39b ONLINE 0 0 0
dedup
gptid/85c84ce9-e23b-11ea-ba69-2cf05d07d39b ONLINE 0 0 0
gptid/863904ee-e23b-11ea-ba69-2cf05d07d39b ONLINE 0 0 0
special
mirror-13 ONLINE 0 0 0
gptid/ded6ac5d-e224-11ea-ba69-2cf05d07d39b ONLINE 0 0 0
gptid/deea7ff4-e224-11ea-ba69-2cf05d07d39b ONLINE 0 0 0
logs
mirror-10 ONLINE 0 0 0
gptid/dfd4c43d-e065-11ea-8ad9-2cf05d07d39b ONLINE 0 0 0
gptid/e26ced7c-e065-11ea-8ad9-2cf05d07d39b ONLINE 0 0 0
spares
gptid/a8f6d298-d4d4-11ea-8717-2cf05d07d39b AVAIL
 
Last edited:

Rand

Guru
Joined
Dec 30, 2013
Messages
906
Just for completeness sake - PM 1725A 3.2 TB (Dell branded, A03 FW, new)

Code:
 smartctl -a /dev/nvme1
smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.3-RELEASE-p11 amd64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       Dell Express Flash PM1725a 3.2TB SFF
Serial Number:                      S3B0NX0KC04945
Firmware Version:                   1.1.2
PCI Vendor ID:                      0x144d
PCI Vendor Subsystem ID:            0x1028
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 3,200,631,791,616 [3.20 TB]
Unallocated NVM Capacity:           0
Controller ID:                      33
Number of Namespaces:               1
Namespace 1 Size/Capacity:          3,200,631,791,616 [3.20 TB]
Namespace 1 Formatted LBA Size:     512
Namespace 1 IEEE EUI-64:            002538 ec81b03d70
Local Time is:                      Thu Aug 27 17:14:29 2020 CEST
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x0006):   Format Frmw_DL
Optional NVM Commands (0x0016):     Wr_Unc DS_Mngmt Sav/Sel_Feat
Maximum Data Transfer Size:         32 Pages
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     77 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +    25.00W       -        -    0  0  0  0      100     100

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 -     512       0         1
 1 -     512       8         3
 2 -    4096       0         0
 3 -    4096       8         2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning:                   0x00
Temperature:                        36 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    2 [1.02 MB]
Data Units Written:                 44,270 [22.6 GB]
Host Read Commands:                 187
Host Write Commands:                515,750
Controller Busy Time:               0
Power Cycles:                       5
Power On Hours:                     1
Unsafe Shutdowns:                   3
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               36 Celsius
Temperature Sensor 2:               36 Celsius
Temperature Sensor 3:               33 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged







diskinfo -wS /dev/nvd1
/dev/nvd1
        512             # sectorsize
        3200631791616   # mediasize in bytes (2.9T)
        6251233968      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Dell Express Flash PM1725a 3.2TB SFF    # Disk descr.
        S3B0NX0KC04945  # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     17.2 usec/IO =     28.4 Mbytes/s
           1 kbytes:     17.0 usec/IO =     57.3 Mbytes/s
           2 kbytes:     17.9 usec/IO =    109.1 Mbytes/s
           4 kbytes:     17.8 usec/IO =    219.2 Mbytes/s
           8 kbytes:     19.9 usec/IO =    392.0 Mbytes/s
          16 kbytes:     24.4 usec/IO =    640.9 Mbytes/s
          32 kbytes:     33.2 usec/IO =    941.0 Mbytes/s
          64 kbytes:     45.5 usec/IO =   1372.8 Mbytes/s
         128 kbytes:     76.1 usec/IO =   1643.4 Mbytes/s
         256 kbytes:    123.2 usec/IO =   2029.9 Mbytes/s
         512 kbytes:    238.2 usec/IO =   2099.3 Mbytes/s
        1024 kbytes:    434.3 usec/IO =   2302.5 Mbytes/s
        2048 kbytes:    916.9 usec/IO =   2181.3 Mbytes/s
        4096 kbytes:   1768.0 usec/IO =   2262.5 Mbytes/s
        8192 kbytes:   3397.6 usec/IO =   2354.6 Mbytes/s
diskinfo -wS /dev/nvd2
/dev/nvd2
        512             # sectorsize
        3200631791616   # mediasize in bytes (2.9T)
        6251233968      # mediasize in sectors
        0               # stripesize
        0               # stripeoffset
        Dell Express Flash PM1725a 3.2TB SFF    # Disk descr.
        S3B0NX0KC05074  # Disk ident.
        Yes             # TRIM/UNMAP support
        0               # Rotation rate in RPM

Synchronous random writes:
         0.5 kbytes:     17.0 usec/IO =     28.7 Mbytes/s
           1 kbytes:     16.8 usec/IO =     58.0 Mbytes/s
           2 kbytes:     17.8 usec/IO =    109.8 Mbytes/s
           4 kbytes:     17.9 usec/IO =    218.4 Mbytes/s
           8 kbytes:     19.7 usec/IO =    395.7 Mbytes/s
          16 kbytes:     24.3 usec/IO =    642.9 Mbytes/s
          32 kbytes:     33.2 usec/IO =    941.9 Mbytes/s
          64 kbytes:     45.4 usec/IO =   1375.4 Mbytes/s
         128 kbytes:     76.2 usec/IO =   1640.3 Mbytes/s
         256 kbytes:    123.4 usec/IO =   2026.8 Mbytes/s
         512 kbytes:    238.8 usec/IO =   2094.0 Mbytes/s
        1024 kbytes:    434.5 usec/IO =   2301.3 Mbytes/s
        2048 kbytes:    911.0 usec/IO =   2195.5 Mbytes/s
        4096 kbytes:   1745.6 usec/IO =   2291.5 Mbytes/s
        8192 kbytes:   3421.0 usec/IO =   2338.5 Mbytes/s
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
16GB optane, for posteriority:


root@freenas[/mnt/SAS]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: INTEL MEMPEK1W016GA
Serial Number: PHBT813403S5016D
Firmware Version: K3110310
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 14,403,239,936 [14.4 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Aug 28 10:35:37 2020 CEST
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0046): Wr_Unc DS_Mngmt Timestmp
Maximum Data Transfer Size: 32 Pages

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 4.50W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 57 Celsius
Available Spare: 100%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 271 [138 MB]
Data Units Written: 2,919,304 [1.49 TB]
Host Read Commands: 6,427
Host Write Commands: 25,410,239
Controller Busy Time: 0
Power Cycles: 18
Power On Hours: 263
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

root@freenas[/mnt/SAS]# diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
14403239936 # mediasize in bytes (13G)
28131328 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1W016GA # Disk descr.
PHBT813403S5016D # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 13.7 usec/IO = 35.7 Mbytes/s
1 kbytes: 13.1 usec/IO = 74.4 Mbytes/s
2 kbytes: 19.2 usec/IO = 101.9 Mbytes/s
4 kbytes: 32.6 usec/IO = 120.0 Mbytes/s
8 kbytes: 59.7 usec/IO = 130.8 Mbytes/s
16 kbytes: 114.0 usec/IO = 137.0 Mbytes/s
32 kbytes: 224.6 usec/IO = 139.1 Mbytes/s
64 kbytes: 445.1 usec/IO = 140.4 Mbytes/s
128 kbytes: 897.9 usec/IO = 139.2 Mbytes/s
256 kbytes: 1770.4 usec/IO = 141.2 Mbytes/s
512 kbytes: 3513.3 usec/IO = 142.3 Mbytes/s
1024 kbytes: 6974.2 usec/IO = 143.4 Mbytes/s
2048 kbytes: 13872.6 usec/IO = 144.2 Mbytes/s
4096 kbytes: 27682.9 usec/IO = 144.5 Mbytes/s
8192 kbytes: 55192.2 usec/IO = 144.9 Mbytes/s
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
That actually looks really good for small writes when one considers how inexpensive those sticks are. My p4801x is 1/2 as fast at the small end (17.4 Mbytes/s @ 0.5kbytes); reaches parity at 2-4kbytes. Yes, it blows its less expensive cousin away for larger writes (1GB/s @ 1024kbytes) but it also costs a lot more. I wonder what applications could make use of 0.5kbyte writes though? Databases?
 

John Doe

Guru
Joined
Aug 16, 2011
Messages
635
wouldn't it make sense to put the results in a shared google spreadsheet?

that thread is grown big
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
wouldn't it make sense to put the results in a shared google spreadsheet?

that thread is grown big
I've been meaning to remake the thread as a shared resource/wiki for some time, since the original thread starter is busy with formal iX support duties. I do have a personally saved spreadsheet; just figuring on how to make it shareable or visible without Google wanting me to expose my personal information to the world.
 

Stilez

Guru
Joined
Apr 8, 2016
Messages
529
16GB optane, for posteriority:


root@freenas[/mnt/SAS]# smartctl -a /dev/nvme0
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: INTEL MEMPEK1W016GA
Serial Number: PHBT813403S5016D
Firmware Version: K3110310
PCI Vendor/Subsystem ID: 0x8086
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 14,403,239,936 [14.4 GB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Fri Aug 28 10:35:37 2020 CEST
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0046): Wr_Unc DS_Mngmt Timestmp
Maximum Data Transfer Size: 32 Pages

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 4.50W - - 0 0 0 0 0 0

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 57 Celsius
Available Spare: 100%
Available Spare Threshold: 0%
Percentage Used: 0%
Data Units Read: 271 [138 MB]
Data Units Written: 2,919,304 [1.49 TB]
Host Read Commands: 6,427
Host Write Commands: 25,410,239
Controller Busy Time: 0
Power Cycles: 18
Power On Hours: 263
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 0

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged

root@freenas[/mnt/SAS]# diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
14403239936 # mediasize in bytes (13G)
28131328 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1W016GA # Disk descr.
PHBT813403S5016D # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM

Synchronous random writes:
0.5 kbytes: 13.7 usec/IO = 35.7 Mbytes/s
1 kbytes: 13.1 usec/IO = 74.4 Mbytes/s
2 kbytes: 19.2 usec/IO = 101.9 Mbytes/s
4 kbytes: 32.6 usec/IO = 120.0 Mbytes/s
8 kbytes: 59.7 usec/IO = 130.8 Mbytes/s
16 kbytes: 114.0 usec/IO = 137.0 Mbytes/s
32 kbytes: 224.6 usec/IO = 139.1 Mbytes/s
64 kbytes: 445.1 usec/IO = 140.4 Mbytes/s
128 kbytes: 897.9 usec/IO = 139.2 Mbytes/s
256 kbytes: 1770.4 usec/IO = 141.2 Mbytes/s
512 kbytes: 3513.3 usec/IO = 142.3 Mbytes/s
1024 kbytes: 6974.2 usec/IO = 143.4 Mbytes/s
2048 kbytes: 13872.6 usec/IO = 144.2 Mbytes/s
4096 kbytes: 27682.9 usec/IO = 144.5 Mbytes/s
8192 kbytes: 55192.2 usec/IO = 144.9 Mbytes/s
I wrote up my own Optane experiences and real-world pool impact as a resource: A bit about SSD perfomance and Optane SSDs, when you're plannng your next SSD

Have a teaser, actual sustained usage during a 36 hour pool replication (this was about typical, not cherrypicked):
IMG_20200813_181408_648.jpg

1/4 million 4k IOPS being pulled off each Optane in the mirror.... ;) Think Im going to swoon! Latency miniscule too.
 

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
Well actually if you see my previous SSD looks better reaching 240MB/s-300MB/s, in contrast to optane which reaches a humble 140MB/s.
Price to performance, SSD was 40E before the whole shananigan which drove prices up, and optane costed me around 15E delivered in about 2weeks, so optane takes the crown without blinking on SLOG duties.

But testing on a VM level optane acully gives me 150MB/s in contrast to 480GB MLC Sandisk plus reaching a only 60MB/s(dedup on, compression on) bassically bear disk offer me the same throughput as they have SSD cache:

root@freenas[~]# smartctl -a /dev/da13
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Marvell based SanDisk SSDs
Device Model: SanDisk SSD PLUS 480GB
Serial Number: 190943801617
LU WWN Device Id: 5 001b44 8b9373c99
Firmware Version: UG5000RL
User Capacity: 480,113,590,272 bytes [480 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri Aug 28 21:47:38 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 32) The self-test routine was interrupted
by the host with a hard or soft reset.
Total time to complete Offline
data collection: ( 120) seconds.
Offline data collection
capabilities: (0x15) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
No Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 85) minutes.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 6359
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 196
165 Total_Write/Erase_Count 0x0032 100 100 000 Old_age Always - 298
166 Min_W/E_Cycle 0x0032 100 100 --- Old_age Always - 2
167 Min_Bad_Block/Die 0x0032 100 100 --- Old_age Always - 0
168 Maximum_Erase_Cycle 0x0032 100 100 --- Old_age Always - 6
169 Total_Bad_Block 0x0032 100 100 --- Old_age Always - 399
170 Unknown_Attribute 0x0032 100 100 --- Old_age Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 0
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Avg_Write/Erase_Count 0x0032 100 100 000 Old_age Always - 2
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 194
184 End-to-End_Error 0x0032 100 100 --- Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 --- Old_age Always - 0
194 Temperature_Celsius 0x0022 062 058 000 Old_age Always - 38 (Min/Max 12/58)
199 SATA_CRC_Error 0x0032 100 100 --- Old_age Always - 0
230 Perc_Write/Erase_Count 0x0032 100 100 000 Old_age Always - 85 40 85
232 Perc_Avail_Resrvd_Space 0x0033 100 100 005 Pre-fail Always - 100
233 Total_NAND_Writes_GiB 0x0032 100 100 --- Old_age Always - 997
234 Perc_Write/Erase_Ct_BC 0x0032 100 100 000 Old_age Always - 3353
241 Total_Writes_GiB 0x0030 100 100 000 Old_age Offline - 1552
242 Total_Reads_GiB 0x0030 100 100 000 Old_age Offline - 1355
244 Thermal_Throttle 0x0032 000 100 --- Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 6340 -
# 2 Extended offline Self-test routine in progress 10% 6340 -
# 3 Extended offline Self-test routine in progress 10% 6340 -
# 4 Extended offline Self-test routine in progress 10% 6340 -
# 5 Extended offline Self-test routine in progress 10% 6340 -
# 6 Extended offline Self-test routine in progress 10% 6340 -
# 7 Extended offline Self-test routine in progress 10% 6340 -
# 8 Extended offline Self-test routine in progress 20% 6340 -
# 9 Extended offline Self-test routine in progress 20% 6340 -
#10 Extended offline Self-test routine in progress 20% 6340 -
#11 Extended offline Self-test routine in progress 20% 6340 -
#12 Extended offline Self-test routine in progress 20% 6340 -
#13 Extended offline Self-test routine in progress 20% 6340 -
#14 Extended offline Self-test routine in progress 30% 6340 -
#15 Extended offline Self-test routine in progress 30% 6340 -
#16 Extended offline Self-test routine in progress 30% 6340 -
#17 Extended offline Self-test routine in progress 30% 6340 -
#18 Extended offline Self-test routine in progress 30% 6340 -
#19 Extended offline Self-test routine in progress 30% 6340 -
#20 Extended offline Self-test routine in progress 30% 6340 -
#21 Extended offline Self-test routine in progress 30% 6340 -

Selective Self-tests/Logging not supported


Forgot to mention underlying SAS disks also have internal SSD SLOG :D :
root@freenas[~]# smartctl -a /dev/da5
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.1-STABLE amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS726060AL4210
Revision: A7J0
Compliance: SPC-4
User Capacity: 6,001,175,126,016 bytes [6.00 TB]
Logical block size: 4096 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id:
Serial number:
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Fri Aug 28 22:22:16 2020 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature: 44 C
Drive Trip Temperature: 85 C

Manufactured in week 45 of year 2017
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 120
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 848
Elements in grown defect list: 0

Vendor (Seagate Cache) information
Blocks sent to initiator = 38472892500934656

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 661 0 661 3938334 68365.124 0
write: 0 324 0 324 1922660 311107.824 0
verify: 0 0 0 0 1250477 0.018 0


Non-medium error count: 0

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Self test in progress ... - NOW - [- - -]
# 2 Background long Completed - 13742 - [- - -]

Long (extended) Self-test duration: 49115 seconds [818.6 minutes]

root@freenas[~]#


Thing is that Freenas is not actually using it I guess:

FUA_NV(Force Unit Access Non-Volatile Cache) may be set to 0 or 1, but is ignored since NV_SUP=0 in Inquiry Page 86h
 
Last edited:

CrimsonMars

Dabbler
Joined
Aug 7, 2020
Messages
24
Wanted to benchmark the disk, but for some reason unable to remove it from dedup duties...

If dedup data is not volatile and necessary, why did it assign the disks in stripe and not mirror by default?

SAS 3.74T 52.1T 56 112 268K 2.74M
raidz2 3.73T 50.8T 40 51 204K 532K
gptid/50cad0f1-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.3K 53.2K
gptid/511b4e3f-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.4K 53.2K
gptid/51294ecd-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.2K 53.2K
gptid/517b5371-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.3K 53.2K
gptid/51907b32-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.3K 53.2K
gptid/51924d04-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.3K 53.3K
gptid/51a8084b-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.4K 53.2K
gptid/5189a1b0-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.4K 53.2K
gptid/51df64e8-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.3K 53.2K
gptid/51e871f0-d0d7-11ea-8717-2cf05d07d39b - - 4 5 20.5K 53.2K
dedup - - - - - -
gptid/85c84ce9-e23b-11ea-ba69-2cf05d07d39b 8.37G 436G 11 10 44.4K 520K
gptid/863904ee-e23b-11ea-ba69-2cf05d07d39b 3.18G 441G 4 3 17.7K 187K
special - - - - - -
mirror 2.42G 442G 0 10 2.54K 254K
gptid/ded6ac5d-e224-11ea-ba69-2cf05d07d39b - - 0 5 1.31K 127K
gptid/deea7ff4-e224-11ea-ba69-2cf05d07d39b - - 0 5 1.23K 127K
logs - - - - - -
mirror 123M 6.38G 0 39 13 1.35M
gptid/d73be473-e939-11ea-bd27-2cf05d07d39b - - 0 19 6 693K
gptid/d9886192-e939-11ea-bd27-2cf05d07d39b - - 0 19 6 693K
---------------------------------------------- ----- ----- ----- ----- ----- -----

---------------------------------------------- ----- ----- ----- ----- ----- -----
root@freenas[~]# gpart list da13 | egrep 'Mediasize|rawuuid'
Mediasize: 480113504256 (447G)
rawuuid: 863904ee-e23b-11ea-ba69-2cf05d07d39b
Mediasize: 480113590272 (447G)
root@freenas[~]#

1598645197585.png


1598645227860.png


1598645251445.png

1598645298505.png
 
Top