SOLVED Server upgrade for ECC RAM

[chuck32]

Thank you! You are awesome!

For clarity: I own 4 drives I specifically purchased for the server, they have seen < 20 spin downs. For whatever reason I purchased tow of the same drives two years ago for my personal PC, they have seen probably around 600-750 spin downs. Consensus seems to be to avoid too many spin down events. One of that is still in my PC the other one is currently a hot spare.
Currently we're using 42% of the available space in raidz2 with 1 hot spare and 1 drive still in my PC.
Additionally I own a 4 TB external drive.
No one can predict the future, both drives from my PC, that have seen more spin downs and are 2 years older are the most likely to fail the first (not entirely true though). That's I would be a bit uneasy with anything less than a fault tolerance of two drives. With that knowledge in mind, let's say I have 6 x 4 TB disks available right now.
Also it needs to be added: yes we should be able to take down the server, wait for the new drive(s) to arrive, burn them in, and then resilver. I would say we could work on internal drives in the meantime and just migrate the changed files later. Most of our stuff is long term storage (photos, videos) and day to day business couldn't be much more than a few documents / photos etc.

5 disk, z3, well, in case it's your idea that means you won't need backups, not so.

No, I just read somewhere that you it is better to already have the redundancy of the hot spare incorporated in RAIDZ3 rather than using RAIDZ2 + hot spare.
I currently have the server, my personal machine and B2 cloud backup as my backup solution. I didn't come up with that kind of redundancy to avoid taking actual backups. If avoidable I'd rather not restore from backup ;) but RAIDZ2 is the level of redundancy I can live with.

Number of drives	4	5	6
RAIDZ2, capacity in TiB	6,81	10,4	14
RAIDZ3, capacity in TiB	X	7,25	9,9
Striped Mirror 2 way, capacity in TiB	7,25	X	10,87
Striped Mirror 3 way, capacity in TiB	X	X	~ 7,25

1. I would be uneasy with RAIDZ2 and 6 disks, because I fear that two of the disks are more prone to failure then the others. In case of a disk failure and resilvering things could get dangerous quickly. Also 14 TiB worth of storage are not really needed right now.
2. Striped mirrors: a three-way mirror would get me 2 drives of redundancy but wouldn't get me any meaningful advantage in terms of space or speed. The latter because if I keep using 1 Gbe ethernet as of right now I wouldn't really profit from faster speeds. For a two way mirror the fault tolerance would still be 1 drive (assuming the next drive that dies kills the mirror), but now I use 6 drives instead of 5 (RAIDZ2). However this is easily expandable.
3. RAIDZ3 is probably overkill in terms of redundancy, I agree now.

So basically I see the following options for me:

1. Expand to 6 drives raidz2, the drives should fail before I reach the capacity limit. If I'm not mistaken I could then go ahead and buy 6 new drives of larger capacity, if I need more space or just replace a single drive with 4 TB.
2. Go for a 2 way striped mirror but only have an 80 % chance that the second drive that dies won't kill the pool.
3. Go for a 3 way striped mirror and achieve the same redundancy I have now (if not higher because there still is a 75 % chance a third failure will not kill the pool) and have the same capacity. But expanding now only costs me 3 drives each time and increases the chances that a third failure is not fatal. Again, as soon as I currently need to store more than 4 TB I need to buy new drives to maintain my backup on my PC.

It's a rock and a hard place for me, but Option 3 a three way mirror seems intriguing now. If I choose to upgrade my network I could profit from faster speeds and expanding the pool in the future gets cheaper. Question is: how much do I want to avoid to restore from backup? I guess restoring from my PC is the least preferable option due to the lack of ECC RAM. So restoring from Backblaze would be the worst that can happen when I go with Option 2. Maybe I should accept that the fact that I do not need that much redundancy and restoring from backup is a viable option. In that case Option 2 would be the best. I should make sure I don't put the two drives from my PC (which I trust the least) into the same mirror though. I even went the extra mile when I purchased the drives to make sure I don't end up with the same batch. I got 2 from the same batch and 2 from 2 different batches.

In any case in that I don't just move my existing pool to the new server but change the layout, I guess restoring from backblaze would be the safest bet?

Really, thank you so much for all your input!

 

[sfatula]

I can tell you I ran raid 5 (not zfs) for close to 20 years on close to 50 systems (guessing), never lost an array. I ran raidz1 for a few years, 3 drive z1, 3 vdevs, never lost a pool. Can it happen? Of course, you see people posting in these forums. But not very often compared to the total set of users. I actually read more about cable failures, controllers failures than I do true multiple drive failures at close to the same time. The material out there on the internet would suggest you have a huge chance of losing a pool to a second failed drive in the same vdev. As in over 50%! Not so. With tler on the drives, a second drive set of errors should not kill the pool at all. Might kill a few files. However, total drive failure, sure that would do it. It's certainly always possible with any level of redundancy.

Go with whatever level of redundancy that makes you happy and meets your budget. For me, I went with easy expansion. So, mirrors but raidz2 not bad. Yes, if the drives get old and you buy those 6 new drives, you can replace them one by one until all replaced and then you have more space if the drives are larger.

 

[chuck32]

I actually read more about cable failures, controllers failures than I do true multiple drive failures at close to the same time. The material out there on the internet would suggest you have a huge chance of losing a pool to a second failed drive in the same vdev. As in over 50%! Not so.

That's what always got me. Most of the time you read about a total drive failure it seems that everyone suggests you will not be able to resilver in time before another drive fails (due to the added stress). I think it's a worst case scenario, but possible.

With tler on the drives, a second drive set of errors should not kill the pool at all. Might kill a few files. However, total drive failure, sure that would do it. It's certainly always possible with any level of redundancy.

Killing a few files sounds horrible, I mean how would you check that / recover from that? It's not that I access all the files all the time and would know if a file is damaged. I know for a fact, that some backups I have had for 15 years have damaged files. Some old pictures I just couldn't open anymore. But that I only discovered years later.

Not all the parts have arrived yet and I still got that 24 h memory testing to do before I can swap parts (and I also need some time because I'm sure my VMs will not work after the CPU Swap (the Home Assistant VM runs in passthrough mode, the others in host model)). So I still have some time to decide.

Go with whatever level of redundancy that makes you happy and meets your budget. For me, I went with easy expansion. So, mirrors but raidz2 not bad. Yes, if the drives get old and you buy those 6 new drives, you can replace them one by one until all replaced and then you have more space if the drives are larger.

It really comes down to what I want. I would think people here will tell me to go for striped mirrors because:

• we don't need high availability
• we have backups (I know the mantra: RAID is not a backup)

As I said we mostly store documents and family photos. Neither of us will pick up videography, photography or the likes as a hobby. We don't hoard linux isos. So the space requirement will only gradually increase over time.

Alternatively I could stick to a 4 disk raidz2 and bite the bullet of purchasing 4 larger drives if needed. I'm pretty confident that this is like 2-3 years down the road.

Only using 4 drives would open the possibility to move the last PC drive to the server and just create a 2 drive pool with no redundancy. Backdraw would be that now my 2nd copy is in the same machine as the originals rather than a separate machine on site. Advantage would be that this copy also enjoys ZFS.

So many options...

 

[sfatula]

Killing a few files sounds horrible, I mean how would you check that / recover from that? It's not that I access all the files all the time and would know if a file is damaged. I know for a fact, that some backups I have had for 15 years have damaged files. Some old pictures I just couldn't open anymore. But that I only discovered years later.

Actually.... ZFS tells you which files so it's not horrible at all!

I think you may have read about the debunked article raid 5 is dead, that mathematically "proves" it. It has a lot of believers, some here. ZFS is not raid anyway. More resilient.

 

[chuck32]

Actually.... ZFS tells you which files so it's not horrible at all!

Very nice!

I think you may have read about the debunked article raid 5 is dead, that mathematically "proves" it. It has a lot of believers, some here. ZFS is not raid anyway. More resilient.

No, I'll see if I can find it.

Some humor for you: Today I learned there are narrow and square 2011 sockets. I was stoked that the CPU cooler arrived in time but now discovered that I need to purchase a different one. Well, maybe I can still lay hands on it before the weekend

 

[chuck32]

IPMI is suprisingly useful

Indeed necessary from what I gathered. No way to set the fan control from the BIOS. I'm still amazed at this feature.

I purchased the noctua u12dx i4 as my CPU cooler and it triggered the lower threshold of fan speed, thus it was ramping up all the time.

I thought there was some setup to do, but no, it even displayed the IP during boot. After correcting the fan thresholds everything seemed to work.

One minor bummer is, that my memory is seen as 2133 instead of 2400. I can probably override the frequency in BIOS but I honestly I doubt it will have any real life Impact on me.

Next step is to plan out the migration to the new hardware, I still need to figure out if my VMs will work after swapping the hardware and whether I need to recreate everything from scratch or whether it would be enough to create a new VM and just use the old zvols.

I also figured out why I have such a large pile of fans - none of them were PWM.. I'll probably replace some of the fans for PWM ones, once I dive deeper into IPMI and how to adjust the fan control, this will be the next logical step.
Paired with the ability to receive notifications of alerts, at least that's why I understood at a glance the system should should be pretty resilient against failures.

Thank you all for your support so far! I feel much better to have this configuration looked over by you.

I will report back again once the migration is done. Currently I'm waiting for memtest, will probably leave it running until Monday after work.

 

[sfatula]

Definitely PWM fans are a good thing, just like IPMI! And yes, setting up alerts is a good thing. I am continually amazed when someone posts on these forums how they've been running Truenas for a year, and happened to login and happened to notice 3 failed drives that had failed months ago. What!? You should be taking action the day you get an alert!

 

[PhilD13]

In an older chassis I put a little pwm fan temperature sensor controller in there and connected the fans to that. Works like a charm.

I don't think you will ever actually notice the speed difference but you can verify on your memory that you actually have all the same speed and type, and the sticks are installed in the proper slots if the memory slots are not all filled. Improper placement on some boards (some Supermicro ones) can cause memory to run slower according to their manuals. Some BIOS's in the Northbridge memory section have an Enforce POR setting which enforces or caps memory speed usually to a lower value than stated on the memory. It is enabled by default. On my Supermicro board it's here

Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Enforce POR -> Disabled (default Enabled)
Advanced >> Chipset Configuration >> North Bridge >> Memory Configuration >> Frequency >> 2400 (default Auto)

There is no guarantee for this function, since it's not following the Intel POR.

Alerts are a good thing, a very good thing. I think though sometimes people don't set the email up or turn off alerts because they are tired of getting emails about something that could be an issue but is not currently one and so they take the easy way out and disable the alerts (I've had people at work disable alerts on robots that then failed later on during production).

 

[chuck32]

The system is mostly running know, but I'd require your insights again.

I did 42 hours / 4 passes of memtest86+ and checked via IPMI if any errors occured.
I ran mprime under lubuntu for an hour to monitor temps (awesome results, I never exceeded 50 °C after an hour with 500 RPM fan speed (300 - 1400 RPM is the range of the cooler). Right now treunas is idling at 26 °C with 21,4 °C ambient temperature.

I had trouble installing truenas in the first place. At first I had trouble finding my USB as a bootmedia in the first place, when I it finally showed up I received the following error messages before the truenas install screen came up:

Code:

crypto/fips/fips_post.c:235: FIPS self test failure

The messages were to fast for me, I typed this from a google result. But it was definitely that.

Then the installer came up. I tried different USB sticks, different USB ports (for creating the image on my PC and for booting on the server) and tried rufus in ISO mode, dd mode and balena etcher. The version I installed was UEFI with rufus and dd mode.

The installation went well except for a few warnings that the GPT partition table maybe garbage. In the end the installation went through without any problems, I was able to boot up truenas. I rebooted a few times, no errors. I took the plunge and reloaded my config and attached all drives, because I think a failed boot medium shouldn't hurt my pools.

After some problems getting the network back up (in the end it was obvious though, I needed to set the new interface to dchp auto for both ipv4 ands ipv6, and then attach the correct to the bridge i created earlier).

Now I have the correct network settings and my pools are accessible.

1) Are the fips and gpt warnings during install of any concern if the system seems to work?
I installed it on two old SSDs (one with a windows install and one with a drive from my VM pool which I had replaced prior to migrating.
I ran smart tests and these are the results:

Code:

root@truenas[~]# smartctl -A /dev/sdb
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.107+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   000    Pre-fail  Always       -       10
  5 Reallocate_NAND_Blk_Cnt 0x0033   100   100   000    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       2170
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3907
171 Program_Fail_Count      0x0032   100   100   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   000    Old_age   Always       -       0
173 Ave_Block-Erase_Count   0x0032   092   092   000    Old_age   Always       -       243
174 Unexpect_Power_Loss_Ct  0x0032   100   100   000    Old_age   Always       -       287
180 Unused_Reserve_NAND_Blk 0x0033   000   000   000    Pre-fail  Always       -       1036
183 SATA_Interfac_Downshift 0x0032   100   100   000    Old_age   Always       -       0
184 Error_Correction_Count  0x0032   100   100   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       92
194 Temperature_Celsius     0x0022   069   060   000    Old_age   Always       -       31 (Min/Max 16/40)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       16
197 Current_Pending_ECC_Cnt 0x0032   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   000    Old_age   Always       -       0
202 Percent_Lifetime_Remain 0x0031   092   092   000    Pre-fail  Offline      -       8
206 Write_Error_Rate        0x000e   100   100   000    Old_age   Always       -       0
210 Success_RAIN_Recov_Cnt  0x0032   100   100   000    Old_age   Always       -       0
246 Total_LBAs_Written      0x0032   100   100   000    Old_age   Always       -       34309214753
247 Host_Program_Page_Count 0x0032   100   100   000    Old_age   Always       -       1077341318
248 FTL_Program_Page_Count  0x0032   100   100   000    Old_age   Always       -       936901314

root@truenas[~]# smartctl -A /dev/sda
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.107+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       29417
 12 Power_Cycle_Count       0x0032   095   095   000    Old_age   Always       -       4420
177 Wear_Leveling_Count     0x0013   080   080   000    Pre-fail  Always       -       719
179 Used_Rsvd_Blk_Cnt_Tot   0x0013   100   100   010    Pre-fail  Always       -       0
181 Program_Fail_Cnt_Total  0x0032   100   100   010    Old_age   Always       -       0
182 Erase_Fail_Count_Total  0x0032   100   100   010    Old_age   Always       -       0
183 Runtime_Bad_Block       0x0013   100   100   010    Pre-fail  Always       -       0
187 Uncorrectable_Error_Cnt 0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0032   073   053   000    Old_age   Always       -       27
195 ECC_Error_Rate          0x001a   200   200   000    Old_age   Always       -       0
199 CRC_Error_Count         0x003e   253   253   000    Old_age   Always       -       0
235 POR_Recovery_Count      0x0012   099   099   000    Old_age   Always       -       857
241 Total_LBAs_Written      0x0032   099   099   000    Old_age   Always       -       93706165163

Should I get new SSDs for the boot pool? Could this be related to the errors? I can't recall if I encountered the errors when I originally setup truenas at the beginning of the year.

2) I can't start my home assistant VM (ubuntu vm with CPU mode set to model booted just fine)

Code:

[EFAULT] Unable to define domain for homeassistant: cannot serialize None (type NoneType)

Probably because the CPU mode was set to passthrough and I swapped CPUs (or the whole system for that matter). Do I need to detach the zvol, recreate the VM and reattach the zvol in order to solve this?

I'll respond to your other replies later, it's way too late for the night - but at least I wanted to see my pools up and running before I could sleep well.

Thanks in advance for your time!

 

[chuck32]

As promised the update after migration is complete:

I had CRC errors on an SSD but smart reported no errors, but they disappeared as soon as I ditched the y-splitter for power (which I'm pretty sure was connected to molex in the old server instead of sata power), so I guess there's the validity of the recommendation: don't use them ;)

1) Are the fips and gpt warnings during install of any concern if the system seems to work?
This is still relevant, I concluded it does not indicate bad SSDs for the bootpool as the error persisted even with the SSDs physically detached. I could not really find any relevant information for it. I created a new boot medium with the latest scale version TrueNAS-SCALE-22.12.4.2 instead of the one I had prior to the upgrade and the messages (including the warnings about the GPT header) persistet. However I have yet to find any problems in stability with the system. Maybe it is not relevant at all.

2) I can't start my home assistant VM
I deleted the VM and attached the zvol for the new VM, even with cpu mode passthrough the VM always started after creation. However it seems that it does not break at every reboot but at every other. For now I set the cpu mode to model but have not rebooted since.
Note on that: I do not plan many reboots at all, since long uptime does not exhibit any negative behaviour (friend of mine reboots his synology every week, otherwise it becomes sluggish). Because I always plan some downtime for reboots it would not be the end of the world if I need to recreate the VM, attach the USB devices again and set the MAC address of the NIC so it will receive the same IP again.
CPU mode is the only thing I can image that plays a role since all other VMs survived every reboot. I realized they do not autostart but the dataset where their zvols live are encrypted. I can't remember whether I needed to unlock them first before too, truenas does not give me an alarm on that though.
At least for ubuntu it did not matter what mode was selected, sysbench results were within margin or error).

3) I had major problems with my SMB shares, they would not persist on windows machine reboots and the credentials disappeared from the credential manager.
After longer that I'd like to admit the solution was to create new users, add them (or their respective group [also newly created] to the ACL. Since then the shares never stopped working. Windows through a different error message for every dataset (no access, device name already in use, multiple connections are not allowed). My ubuntu VM didn't have trouble accessing its shares though.

If I look at an existing ACL it looks like this:

Newly created ACLs were shown with Obj permissions also:

I double checked, since restarting the smb service all entries not visible in the first screenshot seem to have disappeared also (mask, - Obj. etc.).

So I can't really pin the issue, however it seems reimporting the configuration has broken some of the permissions. I found similiar threads where this happened due to upgrades of truenas. Should I open a ticket for that?

Otherwise I can only thank you again and conclude thart I am pretty happy with the new system. IPMI is just freaking awesome ;) Everyone who told me I'd love the feature was right.

Thermals are fine, usually 5-9 °C above ambient with all VMs running.

Power draw is also okay, I added another SSD with the migration and I am now at 70-75 W, the old system drew ~ 55-60 W. This includes approx. 10 W for the USP (I measure what the USP inkl. server draws). I have yet to dive into some CPU heavy tasks (importing multiple documents to paperless for example) but from what I saw until now, it does seem to perform at least as good as the old server.

 

[sos_nz]

Currently the server is living in our office which is currently also used most of the time as a bedroom.

My Fractal R5 server isn't too loud or too hot. It lives in the garage outside :)