SMART errors that I am unable to understand

[nikade87]

Hi everyone,

I've got a Dell R730xd installed with TrueNAS-12.0-U8 and I am seeing some really weird SMART-errors here that I am unable to understand.

The TrueNAS is installed on a boot-drive which consists of 2x600Gb 10K SAS-drives that are configured in a RAID1 with the built in Perc H730 mini.
There are additional 15x1.8Tb 10K SAS-drives and 3x200Gb SSD-drives which are configured in "passthrough" mode so that the disks are presented directly in the TrueNAS gui under Storage -> Disks.

Ive then created a 3 vdevs with 5 drives in each vdev and the pool itself is configured in RAIDZ2, I then have 2 of the SSD disks configured as a mirrored slog and 1 of the SSD disks configured as a l2arc. So far so good.

After running a couple of days I am getting alerts from this machine that there are SMART failures with the code "ascq=0x9c" so I tried googling it, didnt really find anything usefull so I went a head and upgraded the firmware (Latest version PER730XD_BOOTABLE_21.12.00.24) and then after some more days I am getting more alerts about more disks failing.

TrueNAS @ hby-san3

New alerts:
* Device: /dev/da17, Self-Test Log error count increased from 0 to 1.

Current alerts:
* Device: /dev/da17, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da12, SMART Failure: FIRMWARE IMPENDING FAILURE DATA ERROR RATE TOO HIGH.
* Device: /dev/da11, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da10, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da9, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da5, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da4, SMART Failure: WARNING: ascq=0x9c.
* Device: /dev/da17, Self-Test Log error count increased from 0 to 1.

I've checked the iDRAC and it is only complaining about 1 of the disks, da11, the rest is fine according to iDRAC.
I then tried doing a SMART long test on da17 and I got an error that it didnt go that well, this is the output of smartctrl -a

root@hby-san3[/]# smartctl -a /dev/da17
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUC101818CS4204
Revision: FK45
Compliance: SPC-4
User Capacity: 1,800,360,124,416 bytes [1.80 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
Formatted with type 2 protection
8 bytes of protection information per logical block
LU is fully provisioned
Rotation Rate: 10000 rpm
Form Factor: 2.5 inches
Logical Unit id: 0x5000cca02c17a900
Serial number: 08GE0BTA
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Mon Apr 11 10:27:00 2022 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Current Drive Temperature: 39 C
Drive Trip Temperature: 50 C

Accumulated power on time, hours:minutes 52904:48
Manufactured in week 50 of year 2015
Specified cycle count over device lifetime: 50000
Accumulated start-stop cycles: 2022
Specified load-unload count over device lifetime: 600000
Accumulated load-unload cycles: 4216
Elements in grown defect list: 0

Vendor (Seagate Cache) information
Blocks sent to initiator = 143340003655680000

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 3797 0 40473 35022226 224017.218 2
write: 0 0 0 0 3211588 382554.413 0
verify: 0 0 0 0 3563076 0.684 0

Non-medium error count: 1

SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background long Failed in segment --> 100 52904 4294967295 [- - -]
# 2 Background short Completed 80 4 - [- - -]
# 3 Reserved(7) Completed 64 4 - [- - -]

Long (extended) Self-test duration: 11697 seconds [194.9 minutes]

I cant really tell if these errors are for real or not, so I am worried about putting any data on the pool for now.
Any pointers or suggestions would be really appreciated.

Many thanks!

 

[jgreco]

RAID controllers are unsupported and dangerous to your pools. See

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Yank the H730 Mini and replace it with an HBA.

 

[HoneyBadger]

Welcome to the forums. Beware of long post ahead!

Before we get into the RAID/HBA discussion that's already been had, I want to dig into other things a bit deeper:

# 1 Background long Failed in segment --> 100 52904 4294967295 [- - -]

According to this, the device failed a long SMART test and should be replaced.

* Device: /dev/da12, SMART Failure: FIRMWARE IMPENDING FAILURE DATA ERROR RATE TOO HIGH.

SAS devices can also throw "preemptive/predicted failure" codes though SMART which is what you're seeing here. Might want to plan on replacing this one too.

But now we get to the fun stuff.

Formatted with type 2 protection
8 bytes of protection information per logical block

This is telling me that those drives are formatted with 520-byte sectors, not 512-byte. Hardware RAID controllers and SAN arrays often use those extra bits for checksumming; however, ZFS includes that in-line with the normal blocks and won't write to the extra 8-bits of T10 protection space. This could be causing your controller/TrueNAS to disagree with your drives about what is/isn't corrupt or invalid on the drives. I'm not sure of how the H730 behaves with passthrough devices vs. true HBA mode. (More on that later.)

While ZFS and VMware vSAN are worlds apart, they both want the same thing from your disk controller, which is mostly "Get out of the way" so you'll want to:

Yank the H730 Mini and replace it with an HBA.

Specifically the Dell PERC HBA330 if you can get it. If you can't, the (somewhat cheaper) H330 can be cross-flashed into the HBA330 but it's a bit of a process.

In the interim, the H730 could be set to "HBA mode" through iDRAC which will make it use the somewhat-better mrsas driver - this will probably render your system unbootable though as it won't be able to understand the RAID1 boot device. It might also mean that the system can no longer talk to 520-byte-per-sector drives, which makes your pool unmountable.

Thankfully most drives can be changed between sector sizes although it requires a full reformat:

How to reformat HDD & SSD to 512B Sector Size

For those of you who have purchased OEM rebranded HDD or SSD from major storage vendor such as NetApp, EMC, or even HP, you might find out that the drive that you've purchased will not initialized in windows. The main reason is that the drive come formatted from the factory with a non standard...

forums.servethehome.com

Which of course means "back up any data, erase, and try again" - to complicate things, you might only be able to do this after you either switch the H730 to HBA mode in iDRAC or replace it entirely with the HBA330 or reflashed H330.

Ive then created a 3 vdevs with 5 drives in each vdev and the pool itself is configured in RAIDZ2, I then have 2 of the SSD disks configured as a mirrored slog and 1 of the SSD disks configured as a l2arc.

Once we're done, let's revisit this piece. RAIDZ2 is usually needed when you want single-stream sequential I/O, which doesn't typically cross over the use case where SLOG is as critical. If you're planning to serve databases/VMs off of this, the answer is mirrors. The make/model of the SSDs also matter as SLOG duties can burn out a read-intensive or even mixed-use drive in a hurry. If they're write-intensive then they'll be fine.

 

[nikade87]

Welcome to the forums. Beware of long post ahead!

Before we get into the RAID/HBA discussion that's already been had, I want to dig into other things a bit deeper:



According to this, the device failed a long SMART test and should be replaced.



SAS devices can also throw "preemptive/predicted failure" codes though SMART which is what you're seeing here. Might want to plan on replacing this one too.

But now we get to the fun stuff.



This is telling me that those drives are formatted with 520-byte sectors, not 512-byte. Hardware RAID controllers and SAN arrays often use those extra bits for checksumming; however, ZFS includes that in-line with the normal blocks and won't write to the extra 8-bits of T10 protection space. This could be causing your controller/TrueNAS to disagree with your drives about what is/isn't corrupt or invalid on the drives. I'm not sure of how the H730 behaves with passthrough devices vs. true HBA mode. (More on that later.)

While ZFS and VMware vSAN are worlds apart, they both want the same thing from your disk controller, which is mostly "Get out of the way" so you'll want to:



Specifically the Dell PERC HBA330 if you can get it. If you can't, the (somewhat cheaper) H330 can be cross-flashed into the HBA330 but it's a bit of a process.

In the interim, the H730 could be set to "HBA mode" through iDRAC which will make it use the somewhat-better mrsas driver - this will probably render your system unbootable though as it won't be able to understand the RAID1 boot device. It might also mean that the system can no longer talk to 520-byte-per-sector drives, which makes your pool unmountable.

Thankfully most drives can be changed between sector sizes although it requires a full reformat:

How to reformat HDD & SSD to 512B Sector Size

For those of you who have purchased OEM rebranded HDD or SSD from major storage vendor such as NetApp, EMC, or even HP, you might find out that the drive that you've purchased will not initialized in windows. The main reason is that the drive come formatted from the factory with a non standard...

forums.servethehome.com

Which of course means "back up any data, erase, and try again" - to complicate things, you might only be able to do this after you either switch the H730 to HBA mode in iDRAC or replace it entirely with the HBA330 or reflashed H330.


Once we're done, let's revisit this piece. RAIDZ2 is usually needed when you want single-stream sequential I/O, which doesn't typically cross over the use case where SLOG is as critical. If you're planning to serve databases/VMs off of this, the answer is mirrors. The make/model of the SSDs also matter as SLOG duties can burn out a read-intensive or even mixed-use drive in a hurry. If they're write-intensive then they'll be fine.

Thanks for all your replies, I really appreciate it.
I'll try and make a reply that hopefully gives you a detailed and informative answer.

1) I actually noticed that some of the devices passed the short tests, but not the long tests, which leads me to believe this is a "predictive failure" but I wasnt able to confirm that. I think you helped me to confirm that, luckily we've got support on this machine since it was running Nexentastor and ZFS for many years in production until a couple of weeks ago when I re-installed it with TrueNAS since it is what I prefer to use.

2) When it comes to the PERC H730 controller we didnt change it since the Nexenta team initially configured it. Their recommendation was RAID1 for boot-drive and then passthrough for the rest of the disks accordingly to their best practice.
I just had a look and I noticed in the iDRAC that I can go to Settings -> Storage -> Controllers and then under "Controller Mode" I have 2 choices: RAID and HBA. I assume your recommendation is to switch it to HBA and then re-install TrueNAS and choose the 2 induvidual disks (600Gb SAS now in RAID1) as a mirrored boot-drive - Correct?

We have not started using this machine yet, so there are just a couple of VM's for testing and benchmarking the array. In other word no data to lose or backup

3) Regarding RAIDZ2 this was my choice since we've got a couple of other machines running TrueNAS and RAIDZ2 pools, those perform really well so I didnt really do any research to be honest, thanks for the pointer on that.
We'll be using this for VM's and some SQL-servers, is a mirrored pool the way to go? I guess in pair of 2 disks then? Since I have 15 disks there will be 7 mirrored vdev's and the last disk will be a spare I guess.

Please correct me if im wrong and thanks for replying!

 

[jgreco]

When it comes to the PERC H730 controller we didnt change it since the Nexenta team initially configured it. Their recommendation was RAID1 for boot-drive and then passthrough for the rest of the disks accordingly to their best practice.
I just had a look and I noticed in the iDRAC that I can go to Settings -> Storage -> Controllers and then under "Controller Mode" I have 2 choices: RAID and HBA. I assume your recommendation is to switch it to HBA and then re-install TrueNAS and choose the 2 induvidual disks (600Gb SAS now in RAID1) as a mirrored boot-drive - Correct?

No, this is not correct. The H730 is not usable. The linked article above about "What's all the noise" was written in response to people coming in here with the misbegotten idea that "HBA mode" was somehow actually making an HBA out of it. It's not. It's just making it a lobotomized RAID controller. A RAID controller that is no longer able to do RAID but still uses the same firmware and drivers as the RAID is not the same thing as an HBA. It's just lobotomized. It doesn't fully address the issue.

 

[nikade87]

No, this is not correct. The H730 is not usable. The linked article above about "What's all the noise" was written in response to people coming in here with the misbegotten idea that "HBA mode" was somehow actually making an HBA out of it. It's not. It's just making it a lobotomized RAID controller. A RAID controller that is no longer able to do RAID but still uses the same firmware and drivers as the RAID is not the same thing as an HBA. It's just lobotomized. It doesn't fully address the issue.

Allright, I understand. Thanks for clearing that out.
We actually use some LSI 2008 or similar in our other FreeNAS/TrueNAS machines and they do not have this issue.

Maybe this box cant run TrueNAS at all? I probably wont be able to replace the controller, the budget for this machine from 2016 is €0 so maybe I should look at alternatives.

 

[HoneyBadger]

No, this is not correct. The H730 is not usable. The linked article above about "What's all the noise" was written in response to people coming in here with the misbegotten idea that "HBA mode" was somehow actually making an HBA out of it. It's not. It's just making it a lobotomized RAID controller. A RAID controller that is no longer able to do RAID but still uses the same firmware and drivers as the RAID is not the same thing as an HBA. It's just lobotomized. It doesn't fully address the issue.

@jgreco Are you quoting a ghost post, or does OP need his approved still? I can't see the actual source there. It's there now.

 

[HoneyBadger]

1) I actually noticed that some of the devices passed the short tests, but not the long tests, which leads me to believe this is a "predictive failure" but I wasnt able to confirm that. I think you helped me to confirm that, luckily we've got support on this machine since it was running Nexentastor and ZFS for many years in production until a couple of weeks ago when I re-installed it with TrueNAS since it is what I prefer to use.

2) When it comes to the PERC H730 controller we didnt change it since the Nexenta team initially configured it. Their recommendation was RAID1 for boot-drive and then passthrough for the rest of the disks accordingly to their best practice.
I just had a look and I noticed in the iDRAC that I can go to Settings -> Storage -> Controllers and then under "Controller Mode" I have 2 choices: RAID and HBA. I assume your recommendation is to switch it to HBA and then re-install TrueNAS and choose the 2 induvidual disks (600Gb SAS now in RAID1) as a mirrored boot-drive - Correct?

We have not started using this machine yet, so there are just a couple of VM's for testing and benchmarking the array. In other word no data to lose or backup

3) Regarding RAIDZ2 this was my choice since we've got a couple of other machines running TrueNAS and RAIDZ2 pools, those perform really well so I didnt really do any research to be honest, thanks for the pointer on that.
We'll be using this for VM's and some SQL-servers, is a mirrored pool the way to go? I guess in pair of 2 disks then? Since I have 15 disks there will be 7 mirrored vdev's and the last disk will be a spare I guess.

Please correct me if im wrong and thanks for replying!

1) If by support you mean "hardware support and warranty through Dell" then go ahead and get any drive that even blinks its LED funny replaced.

2) Not sure why they would have done that (passthrough vs. at least pure HBA) or not supplying you with the HBA330 to begin with. The H730 is a card for bare-metal performance, not SDS or hypervisor bootstrapping. Regarding the budget question a replacement HBA330 (or H330 pre-flashed to one) should be around USD$150 at worst if you're in NA. Petition your higher-ups for the dollars and justify it by saying TrueNAS alerted you to premature failure of one of the 1.8T HGST's.

3) Definitely mirrored but I'm a little worried about the data safety, as RAIDZ2 and "perform really well" don't tend to go together for VMs. I'm going to hazard a guess that you're running iSCSI ZVOLs off of them, which default to using asynchronous writes. Fast, but unsafe in a power-loss or unexpected shutdown scenario. See the resource:

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

Depending on the make/model of your SLOG SSDs, enabling sync=always on iSCSI ZVOLs can range from "noticeable but manageable" to "OUR SYSTEM IS DOWN HALP" levels of performance hits. Post make/model of SSDs if you can, should be visit from SMART data.

 

[nikade87]

1) If by support you mean "hardware support and warranty through Dell" then go ahead and get any drive that even blinks its LED funny replaced.

2) Not sure why they would have done that (passthrough vs. at least pure HBA) or not supplying you with the HBA330 to begin with. The H730 is a card for bare-metal performance, not SDS or hypervisor bootstrapping. Regarding the budget question a replacement HBA330 (or H330 pre-flashed to one) should be around USD$150 at worst if you're in NA. Petition your higher-ups for the dollars and justify it by saying TrueNAS alerted you to premature failure of one of the 1.8T HGST's.

3) Definitely mirrored but I'm a little worried about the data safety, as RAIDZ2 and "perform really well" don't tend to go together for VMs. I'm going to hazard a guess that you're running iSCSI ZVOLs off of them, which default to using asynchronous writes. Fast, but unsafe in a power-loss or unexpected shutdown scenario. See the resource:

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

Depending on the make/model of your SLOG SSDs, enabling sync=always on iSCSI ZVOLs can range from "noticeable but manageable" to "OUR SYSTEM IS DOWN HALP" levels of performance hits. Post make/model of SSDs if you can, should be visit from SMART data.

Hi,

1) Yes exactly, we have this machine under hardware support so I'll go a head and request a replacement for the disk that atleast iDRAC complains about. I'll see what they can tell me about the others, they usually dont want to swap drives out if iDRAC isnt complaining :(

2) I took at look at ebay and you're right - they are only about $150 so I will go a head and order a HBA330(I guess this one is fine? https://www.ebay.com/itm/325121896275?hash=item4bb2c66f53:g:~0sAAOSwM0tiRtHR). This machine will be serving NFS to a XCP-NG pool which mounts the storage over NFS. Previously when it was running Nexentastor it used to be both NFS and iSCSI.

3) I also had a chat with the Nexentastor technician and he said we were using a "ZFS-Mirror which is like a RAID10" in the previous configuration so you are correct about that as well. It was the MD1220 that was configured in a RAIDZ2, not the primary storage.

The SSD's are 2.5" Sandisk D417 enterprise SAS SSD's "mixed use" if that tells you anything. They've been running for about 6 years and the 2 that was used for slog has 85% remaining endurance accordingly to iDRAC. The one used for l2arc has 92% remaining endurance so hopefully I dont have to do anything about those.

 

[nikade87]

One more thing, we used to have a MD1220 enclosure connected to this machine and I realized that we connected it with 2 SAS-cables, when I look in the iDRAC under Storage -> Controllers I see this:

I am hoping this is the kind of HBA that I need but I am unable to see any information about it. Expanding it just show this and no partnumber:

It would be convenient if all I had to do was swap the cables from the PERC H730 to the HBA.

 

[HoneyBadger]

Hi,

1) Yes exactly, we have this machine under hardware support so I'll go a head and request a replacement for the disk that atleast iDRAC complains about. I'll see what they can tell me about the others, they usually dont want to swap drives out if iDRAC isnt complaining :(

You should be able to get the iDRAC flagged disk easily, I'd argue for the one throwing "predictive failure" as well if you can get a Dell utility to show you that (either iDRAC or the PERC H730 array config util)

2) I took at look at ebay and you're right - they are only about $150 so I will go a head and order a HBA330(I guess this one is fine? https://www.ebay.com/itm/325121896275?hash=item4bb2c66f53:g:~0sAAOSwM0tiRtHR).

That one will be fine.

This machine will be serving NFS to a XCP-NG pool which mounts the storage over NFS. Previously when it was running Nexentastor it used to be both NFS and iSCSI.

3) I also had a chat with the Nexentastor technician and he said we were using a "ZFS-Mirror which is like a RAID10" in the previous configuration so you are correct about that as well. It was the MD1220 that was configured in a RAIDZ2, not the primary storage.

I don't have any real experience with XCP but you'll definitely want the mirror configuration here for virtual machines. If you're running NFS it should be sending the data either as sync writes or following it with a flush - might depend on the virtual disk behaviour at a per-machine level, if XCP behaves like KVM in that sense.

The SSD's are 2.5" Sandisk D417 enterprise SAS SSD's "mixed use" if that tells you anything. They've been running for about 6 years and the 2 that was used for slog has 85% remaining endurance accordingly to iDRAC. The one used for l2arc has 92% remaining endurance so hopefully I dont have to do anything about those.

Tells me they're most likely Sandisk Lightning Gen II (and it's a good thing you're on firmware D417 as the earlier ones would bug out when they crossed 32767 operating hours.) Those do have PLP and the "mixed use" drives are still rated for 10 DWPD (drive writes per day) so they'll be fine for your use case here.

One more thing, we used to have a MD1220 enclosure connected to this machine and I realized that we connected it with 2 SAS-cables, when I look in the iDRAC under Storage -> Controllers I see this:

I am hoping this is the kind of HBA that I need but I am unable to see any information about it. Expanding it just show this and no partnumber:

It would be convenient if all I had to do was swap the cables from the PERC H730 to the HBA.

You're in luck, that's basically an external HBA330. Hook the MD1220 into there, throw some drives in, and they should populate automatically without need for any RAID configuration.

 

[nikade87]

You should be able to get the iDRAC flagged disk easily, I'd argue for the one throwing "predictive failure" as well if you can get a Dell utility to show you that (either iDRAC or the PERC H730 array config util)



That one will be fine.



I don't have any real experience with XCP but you'll definitely want the mirror configuration here for virtual machines. If you're running NFS it should be sending the data either as sync writes or following it with a flush - might depend on the virtual disk behaviour at a per-machine level, if XCP behaves like KVM in that sense.



Tells me they're most likely Sandisk Lightning Gen II (and it's a good thing you're on firmware D417 as the earlier ones would bug out when they crossed 32767 operating hours.) Those do have PLP and the "mixed use" drives are still rated for 10 DWPD (drive writes per day) so they'll be fine for your use case here.



You're in luck, that's basically an external HBA330. Hook the MD1220 into there, throw some drives in, and they should populate automatically without need for any RAID configuration.

1) I have contacted support and uploaded SupportAssist dump, hoping they will let me replace all of those drives.

2) I discussed this with a colleague - the Perc H730 is probably an integrated card which sits on a slot on the motherboard. If we want to replace that one we should get an integrated H330 instead so we found this (https://www.ebay.com/itm/224657028779?hash=item344e9a36ab:g:CtMAAOSwg8FiSufM) H330 which by what I can tell is the same one as recommended earlier, right? Just integrated instead of a stick-in card.

3) Regarding the Sandisk's we got a alert from Dell like 2 years ago about the firmware bug so we upgraded them. So far they are doing pretty well so I am going to keep using them. We'll also do a ZFS-mirror instead of the RAIDZ2 as you advised earlier once we get the H330 and have it installed.

4) We'll use the MD1220 as well, we have some SATA disks for that pool which will basically work as an archive for backups. From what I can tell you are saying this will work thanks for the external HBA card? No TrueNAS incompability or simular?

If I am able to get this rolling I am looking to replace another R730xd that is currently running Nexentastor, I personally prefer the TrueNAS gui and I am more comfortable working with TrueNAS over Nexentastor.

Thanks for all your help, I really do appreciate it!

 

[HoneyBadger]

1) I have contacted support and uploaded SupportAssist dump, hoping they will let me replace all of those drives.

Reach out to your sales rep or VAR if they balk at replacing a drive in a "predictive failure" state and play a little bit of politics.

2) I discussed this with a colleague - the Perc H730 is probably an integrated card which sits on a slot on the motherboard. If we want to replace that one we should get an integrated H330 instead so we found this (https://www.ebay.com/itm/224657028779?hash=item344e9a36ab:g:CtMAAOSwg8FiSufM) H330 which by what I can tell is the same one as recommended earlier, right? Just integrated instead of a stick-in card.

The H330 and HBA330 are not the same, but the H330 can be flashed to become an HBA330. The H330 on its own is a (very poor) RAID card that's really only good for booting a hypervisor. Check the link below and read carefully before continuing, but the "Mini" is flashable.

Flash/Crossflash DELL H330 RAID Card to HBA330/12Gbps HBA IT Firmware

Success! You can now flash the Dell PERC H330 (All Models) to IT Mode (HBA330). Big Thanks to BLinux for asking if it could be done! (Inspiration!) So I initially had a super long write-up, but that's way too long, you just need the compact steps. I linked the write up at the end of the...

forums.servethehome.com

If you have rear 2.5" bays in your R730XD though you could also use the H730 to control those (for hardware-mirrored boot) and then use a regular PCIe form factor HBA330.

3) Regarding the Sandisk's we got a alert from Dell like 2 years ago about the firmware bug so we upgraded them. So far they are doing pretty well so I am going to keep using them. We'll also do a ZFS-mirror instead of the RAIDZ2 as you advised earlier once we get the H330 and have it installed.

Sounds good. Keep an eye on the endurance on the SLOG drives.

4) We'll use the MD1220 as well, we have some SATA disks for that pool which will basically work as an archive for backups. From what I can tell you are saying this will work thanks for the external HBA card? No TrueNAS incompability or simular?

For optimal compatibility you could flash generic LSI firmware onto the card, but the Dell HBA firmware has been fine for other users. Being an HBA there's no way for it to throw RAID in the mix.

If I am able to get this rolling I am looking to replace another R730xd that is currently running Nexentastor, I personally prefer the TrueNAS gui and I am more comfortable working with TrueNAS over Nexentastor.

Thanks for all your help, I really do appreciate it!

No problem. Enable compression, don't use deduplication, and try setting your dataset record size to 32K if you're going to run VMs that want lower disk latency.

 

[nikade87]

Reach out to your sales rep or VAR if they balk at replacing a drive in a "predictive failure" state and play a little bit of politics.



The H330 and HBA330 are not the same, but the H330 can be flashed to become an HBA330. The H330 on its own is a (very poor) RAID card that's really only good for booting a hypervisor. Check the link below and read carefully before continuing, but the "Mini" is flashable.

Flash/Crossflash DELL H330 RAID Card to HBA330/12Gbps HBA IT Firmware

Success! You can now flash the Dell PERC H330 (All Models) to IT Mode (HBA330). Big Thanks to BLinux for asking if it could be done! (Inspiration!) So I initially had a super long write-up, but that's way too long, you just need the compact steps. I linked the write up at the end of the...

forums.servethehome.com

If you have rear 2.5" bays in your R730XD though you could also use the H730 to control those (for hardware-mirrored boot) and then use a regular PCIe form factor HBA330.



Sounds good. Keep an eye on the endurance on the SLOG drives.



For optimal compatibility you could flash generic LSI firmware onto the card, but the Dell HBA firmware has been fine for other users. Being an HBA there's no way for it to throw RAID in the mix.



No problem. Enable compression, don't use deduplication, and try setting your dataset record size to 32K if you're going to run VMs that want lower disk latency.

Great feedback, ive noted that there are both H330 and HBA330 on ebay. When I look at this specific HBA330 (https://www.ebay.com/itm/224657028779?hash=item344e9a36ab:g:CtMAAOSwg8FiSufM) it says HBA330 in the name and "IT-mode" which is what I think is what I am looking for. Atleast that is what i've found on this forum and checking reddit.
Totally replacing the H730 is no problem for me, I'd run the boot in a mirror, I think that is how our other FreeNAS machines are setup and they have been running fine for many years.

Regarding compression and dedup we always use compression but never dedup, I never tried but I think our machines would handle it.
They all have 128Gb ECC ram and 2x Xeon hexacores, at variating clock-speed.

I have never changed the dataset record size, any guide/link on where and how we change that? We got 4 FreeNAS machines running and I have never before changed this on any of them. Might be worth checking!

 

[HoneyBadger]

Deduplication is much bigger than just CPU and RAM, it's pretty strongly discouraged unless you are willing to throw lot of money and resources at it just to make it "acceptable performance."

I have never changed the dataset record size, any guide/link on where and how we change that? We got 4 FreeNAS machines running and I have never before changed this on any of them. Might be worth checking!

It's under "Advanced Dataset Options" as "Record Size"

/core/coretutorials/storage/pools/datasets/

It will only affect newly written data though.

 

[nikade87]

Deduplication is much bigger than just CPU and RAM, it's pretty strongly discouraged unless you are willing to throw lot of money and resources at it just to make it "acceptable performance."



It's under "Advanced Dataset Options" as "Record Size"

/core/coretutorials/storage/pools/datasets/

It will only affect newly written data though.

Yeah we only have dedup enabled on 1 of our systems and it is on our Dell EMC PowerStore 1000T because it is enabled by default and cant be turned off.

Thanks for the pointer, I will look in to that on our other boxes running TrueNAS.

Regarding the HBA I'll make the order this week, hopefully its an easy switch and then I'll reinstall this R730xd and cross my fingers.
If it all goes well we will be reinstalling another R730xd that is currently running Nexentastor.

Thanks for all your help and pointers, I really do appreciate it!