Partialy crash after "heavy" load

[Pixi]

Hi i have some trouble with my freenas . When i have and " heavy " and particulary continous load on server , within 3-4 hours of writing, folder becomes slow for accesse and it's impossible to write anything like it's corrupt , only solution i finded it's reboot Truenas server. After this 'crash', parent folder of writed folder it's verry slow , accesse time ~2-3min !

We write many RIB files from Renderman ( env 800mo each) with 7 or 8 workstation in sam time , this server is renderfarm storage for 3d studio.

I don't understand , if you need more information do not hesitate


Pool configuration :
6 RAIDZ2 Vdevs of 10 disk
9 SSD In Cache
Dedup : Off
Compression : lz4
Exported in SMB

Hardware :

TrueNAS-12.0-RC1
Network : Intel 82599Es 2x10Gb Fibre
CPU : Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
RAM : 64GB ECC
MotherBoard : SuperMicro X11DPH-T
SAS Card : AOC-S3108L-H8iR
Disk : 60 Mixed SAS and SATA 4To HDD ( i know is not best practice but i have no choice)
Cache : 9 SSD 1 To

 

[sretalla]

TrueNAS-12.0-RC1
It's probably time to update to 12.0 U2.1

Network : Intel 82599Es 2x10Gb Fibre
Network is fine.

CPU : Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
CPU is fine.

RAM : 64GB ECC
RAM is probably very low for what you are doing... see cache.

MotherBoard : SuperMicro X11DPH-T
Motherboard is fine (although may be your limit for RAM).

SAS Card : AOC-S3108L-H8iR
I don't know a lot specifically about that HBA, but that model number looks a bit wrong to me on the broadcom side (3108). Have you made sure you got the latest firmware? or are you running a broadcom native firmware?

Disk : 60 Mixed SAS and SATA 4To HDD ( i know is not best practice but i have no choice)
That's somewhere in the region of 200TB of storage... you need more RAM to handle the metadata (or consider a multi-way mirror of SSDs for a metadata VDEV)

Cache : 9 SSD 1 To
This is probably the source of the problems... that's 9 Terabytes of L2ARC... way too much for your 64GB of RAM to handle the mappings for, so I imagine your problems begin when you fill enough of the L2ARC to mean that the ARC is entirely dedicated to the mapping for the L2ARC and then you see performance drop to a miserable speed. L2ARC is emptied on reboot, so that would explain why things go back to normal after reboot until you fill the L2ARC sufficiently again to bring back the problem.

You could probably see at least some improvement in stability if you just remove the L2ARC entirely or vastly reduce the amount of it (maximum one of those 1TB SSDs... ideally even a smaller one unless you can add more RAM).

The "ideal" amount of RAM (available to use for ARC... so less than your total installed RAM) needed for L2ARC mapping is about 1:10 as far as I can tell (maybe even a bit worse than that)... so for 1TB of L2ARC, you really should have 100GB of "free" RAM.... for 9TB, that's 900GB...

 

[HoneyBadger]

You should definitely update to the latest TrueNAS 12 release as if I recall there were some SMB bugs in early releases (and you're on a release candidate, not even production) that would need squashing.

Your RAID card is also a likely pain point. Do you know if it is using the mfi driver or the mrsas driver? Running dmesg | grep 'mfi\|mrsas' from a shell should give you the answer.

The 9TB of L2ARC is going to be a weight certainly but it should be manageable given the large files.

Even if default recordsize of 128K is used:

9T * 1024 (G) * 1024 (M) * 1024 (K) / 128K = 75497472 records * 70 bytes/record = 5284823040 bytes = 5040M RAM to hold headers. Certainly big but manageable on a system with 64G of RAM.

Switching the dataset to 1M recordsize would offer significant gains (8x less RAM or 630M).

My question here would be "is the L2ARC actually populating, because 9TB at the default fill rates will take a long time"

 

[Pixi]

Thanks for your reply ! i have mrsas driver , i know im on RC release but im an IT who has 2h/week for manage 50 servers , network and NAS , so it's hard to find time for check update ^^ . I know 9To off cache it's too high we realy use 5/6To im going to reduce that .
I have 2 questions , how i can update from RC to U2 release , it's obligatory to reinstall from scratch ? It's not dangerous for my Pool ?
I can change recordsize in production ?

Sorry for my bad English

Many thank for your help !

 

[HoneyBadger]

Your English is much better than my French. :)

I definitely understand that updating can be challenging when you have limited amounts of time and resources. I would still suggest trying to make time as there were several valuable fixes that could help address the freezing/stalling.

The amount of cache (9To) is not a big concern while it is running, but it may take a very long time for it to actually fill. I would not suggest enabling the L2ARC rebuild (or "persistent L2ARC") feature either, as there is a bug filed right now that suggests a very large L2ARC could cause performance issues on rebuild. 9To would seem to qualify as a "very large L2ARC."

Have you considered removing some of the SSDs, and making a separate pool out of them for some of the more frequently-accessed or "hot" data?

You do not need to reinstall TrueNAS to take patches or updates, but you will need to reboot.

Recordsize can be changed in production and on a live dataset without interruption, but only new data will be written with 1M recordsize. Older data will remain at 128K.

 

[sretalla]

how i can update from RC to U2 release , it's obligatory to reinstall from scratch ? It's not dangerous for my Pool ?

You should be able to switch trains to 12.0-STABLE and your system will see the update.

No risk to data in your pools and no need to reinstall.

 

[Pixi]

Hi , Many thanks guys , i have updatated fine my Truenas , i have increase recordsize to 512kb , and i have remove all SSD cache for rebuild them with new organisation of SSD in my DAS but when i try to re add my SSD on new cache Vdev , at format step i have this error :

Code:

  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/disk_/format.py", line 25, in format
    raise CallError(f'Failed to wipe disk {disk}: {job.error}')
middlewared.service_exception.CallError: [EFAULT] Failed to wipe disk da4: [EFAULT] Command dd if=/dev/zero of=/dev/da4 bs=1M count=32 failed (code 1):
dd: /dev/da4: Input/output error
1+0 records in
0+0 records out
0 bytes transferred in 0.001677 secs (0 bytes/sec)

But , at this time with this modification all work great, i tried in intensive charge and at this time, i have no issue. if a have new issue , i comeback to you, but i wish to reuse my SSD cache at 2To and fix this format error

 

[HoneyBadger]

Code:

dd: /dev/da4: Input/output error

A generic I/O error doesn't give us a lot to work with. Can you connect the drive to another machine and see if it is recognized and can be formatted there?

 

[sretalla]

Can you connect the drive to another machine and see if it is recognized and can be formatted there?

Or maybe run the command at CLI and get some different error report?
dd if=/dev/zero of=/dev/da4 bs=1M count=32

 

[Pixi]

Actualy i cant connect this drive on other machine , but when i try with CLI i have same error :

Code:

root@mother[~]# dd if=/dev/zero of=/dev/da4 bs=1M count=32
dd: /dev/da4: Input/output error
1+0 records in
0+0 records out
0 bytes transferred in 0.001657 secs (0 bytes/sec)

With value with no M in bs value have no issue :

Code:

root@mother[~]# dd if=/dev/zero of=/dev/da4 count=32 bs=8192
32+0 records in
32+0 records out
262144 bytes transferred in 0.018975 secs (13815330 bytes/sec)


root@mother[~]# dd if=/dev/zero of=/dev/da4 count=32 bs=131072
32+0 records in
32+0 records out
4194304 bytes transferred in 0.033296 secs (125969204 bytes/sec)

I have this issue with all ssd :/

When i try fsdisk i have this report , i tried to recontruct partion table with no sucesse :

Code:

root@mother[~]# fdisk da4
******* Working on device /dev/da4 *******
parameters extracted from in-core disklabel are:
cylinders=116737 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=116737 heads=255 sectors/track=63 (16065 blks/cyl)

fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 63, size 1875379842 (915712 Meg), flag 80 (active)
        beg: cyl 0/ head 1/ sector 1;
        end: cyl 0/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>