Extremly long startup after adding new pool, kubernetes not starting

U

ubergosu

Cadet
Joined
Feb 26, 2022
Messages
9
Hi!

I had a home lab with 2 pools. One pool RAIDZ for some work-related backups and one pool with 1 cheap SMR for personal backups. I had some running kubernetes pods and virtual machines, they had virtual disks on my SMR drive. System is installed on a separate SSD drive.

Recently I've added new pool with mirrored SSD devices and decided to move there personal backups, kubernetes and VMs. I did it with replication tasks - cloned backup datasets and VM's zvols to new pool. Everything seemingly worked fine.

But after I rebooted (I wanted to be sure everyithing starts up fine) mounting pools took awfuly long time (maybe hours, after waiting few hours just went to sleep). VM didn't start automatically, kubernetes didn't run, CPU, disks etc charts in Reporting and Dashboard are missing.

Not sure were to look, even didn't find any useful logs. The only strange error message in dmesg is:


Code:
[Mon Jan 15 22:24:34 2024] INFO: task middlewared (wo:1748 blocked for more than 120 seconds.
[Mon Jan 15 22:24:34 2024]       Tainted: P           OE      6.1.55-production+truenas #2
[Mon Jan 15 22:24:34 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Jan 15 22:24:34 2024] task:middlewared (wo state:D stack:0     pid:1748  ppid:1681   flags:0x00000002
[Mon Jan 15 22:24:34 2024] Call Trace:
[Mon Jan 15 22:24:34 2024]  <TASK>
[Mon Jan 15 22:24:34 2024]  __schedule+0x2ed/0x860
[Mon Jan 15 22:24:34 2024]  schedule+0x5a/0xb0
[Mon Jan 15 22:24:34 2024]  schedule_preempt_disabled+0x14/0x30
[Mon Jan 15 22:24:34 2024]  __mutex_lock.constprop.0+0x3b4/0x700
[Mon Jan 15 22:24:34 2024]  spa_open_common+0x65/0x440 [zfs]
[Mon Jan 15 22:24:34 2024]  spa_get_stats+0x4a/0x210 [zfs]
[Mon Jan 15 22:24:34 2024]  ? spl_kmem_alloc_impl+0x87/0xd0 [spl]
[Mon Jan 15 22:24:34 2024]  zfs_ioc_pool_stats+0x3c/0x90 [zfs]
[Mon Jan 15 22:24:34 2024]  zfsdev_ioctl_common+0x67f/0x770 [zfs]
[Mon Jan 15 22:24:34 2024]  ? __kmalloc_node+0xbf/0x150
[Mon Jan 15 22:24:34 2024]  zfsdev_ioctl+0x4f/0xd0 [zfs]
[Mon Jan 15 22:24:34 2024]  __x64_sys_ioctl+0x90/0xd0
[Mon Jan 15 22:24:34 2024]  do_syscall_64+0x5b/0xc0
[Mon Jan 15 22:24:34 2024]  ? handle_mm_fault+0xdb/0x2d0
[Mon Jan 15 22:24:34 2024]  ? preempt_count_add+0x47/0xa0
[Mon Jan 15 22:24:34 2024]  ? up_read+0x37/0x70
[Mon Jan 15 22:24:34 2024]  ? do_user_addr_fault+0x1bb/0x570
[Mon Jan 15 22:24:34 2024]  ? exc_page_fault+0x70/0x170
[Mon Jan 15 22:24:34 2024]  entry_SYSCALL_64_after_hwframe+0x64/0xce
[Mon Jan 15 22:24:34 2024] RIP: 0033:0x7f1d03928afb
[Mon Jan 15 22:24:34 2024] RSP: 002b:00007ffe61da17f0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[Mon Jan 15 22:24:34 2024] RAX: ffffffffffffffda RBX: 0000000002ee3d10 RCX: 00007f1d03928afb
[Mon Jan 15 22:24:34 2024] RDX: 00007ffe61da1870 RSI: 0000000000005a05 RDI: 0000000000000019
[Mon Jan 15 22:24:34 2024] RBP: 00007ffe61da4e60 R08: 0000000000000007 R09: 0000000000000013
[Mon Jan 15 22:24:34 2024] R10: 000000000150b010 R11: 0000000000000246 R12: 00007ffe61da1870
[Mon Jan 15 22:24:34 2024] R13: 0000000002ee3d10 R14: 000000000433da20 R15: 00007ffe61da4e74
[Mon Jan 15 22:24:34 2024]  </TASK>
[Mon Jan 15 22:24:34 2024] INFO: task middlewared (wo:1824 blocked for more than 121 seconds.
[Mon Jan 15 22:24:34 2024]       Tainted: P           OE      6.1.55-production+truenas #2
[Mon Jan 15 22:24:34 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

I rebooted multiple times and this persists. After sshd goes online during boot sequence I see that zpools are not ready and not mounted for a long time. After they are finally mounted, I manually boot VM and it works fine. No kube and no charts though.

I can live with it, of course, charts are not so important and reboots are rare, but it troubles me that seemingly trivial operation like adding new pool and replicating datasets to it broke something.

Where can I look for problems? Smartctl and hdsentinel show that disks are good. Maybe some specific logs of kubernetes?
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Hardware list please. Are you perhaps using persistent L2ARC?
 
U

ubergosu

Cadet
Joined
Feb 26, 2022
Messages
9
Hello!

Thank you for your reply.

No, no l2arc. For raidz pool (main) i have 2 dedup mirrored ssd.
Pools :
main (raidz) - work backups
home_data (mirror) - new ssd pool , moved private data here
temp_smr (smr single drive) - old private data



Code:
00:00.0 Host bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 (rev 01)
00:01.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 01)
00:03.0 PCI bridge: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 01)
00:05.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 01)
00:05.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug (rev 01)
00:05.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 01)
00:05.4 PIC: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC (rev 01)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5)
00:1c.7 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #8 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
02:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 430] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF108 High Definition Audio Controller (rev a1)
04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9215 PCIe 2.0 x1 4-port SATA 6 Gb/s Controller (rev 11)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
ff:0b.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0b.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0b.2 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 01)
ff:0c.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0c.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0d.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers (rev 01)
ff:0f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent (rev 01)
ff:0f.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:0f.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:0f.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 01)
ff:10.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 01)
ff:10.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface (rev 01)
ff:10.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:10.6 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:10.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 01)
ff:12.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 01)
ff:12.1 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 (rev 01)
ff:12.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 01)
ff:12.5 Performance counters: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 (rev 01)
ff:13.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 01)
ff:13.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 01)
ff:13.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 01)
ff:13.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 01)
ff:13.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 01)
ff:13.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 01)
ff:14.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 01)
ff:14.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 01)
ff:14.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 01)
ff:14.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 01)
ff:14.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:14.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 01)
ff:16.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 01)
ff:16.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 01)
ff:16.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 01)
ff:16.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder (rev 01)
ff:16.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 01)
ff:16.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 01)
ff:17.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 01)
ff:17.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control (rev 01)
ff:17.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers (rev 01)
ff:17.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers (rev 01)
ff:17.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.5 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.6 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:17.7 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 01)
ff:1e.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.1 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.3 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1e.4 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit (rev 01)
ff:1f.0 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 01)
ff:1f.2 System peripheral: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU (rev 01)


Code:
NAME     FSTYPE     FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1
└─sda2   zfs_member 5000  temp_smr  4215456177573269261
sdb
├─sdb1
└─sdb2   zfs_member 5000  main      4435232535313325120
sdc
├─sdc1
└─sdc2   zfs_member 5000  main      4435232535313325120
sdd      zfs_member 5000
└─sdd1   zfs_member 5000  main      4435232535313325120
sde
├─sde1
├─sde2   vfat       FAT32 EFI       6BF8-DEFE
├─sde3   zfs_member 5000  boot-pool 1290263006053816833
└─sde4
  └─sde4 swap       1               e8b3d4ed-f95a-470f-94c7-9b81c8328262                [SWAP]
sdf      zfs_member 5000
└─sdf1   zfs_member 5000  main      4435232535313325120
sdg
├─sdg1
└─sdg2   zfs_member 5000  main      4435232535313325120
sdh
└─sdh1   zfs_member 5000  home_data 3254388083323955506
sdi
└─sdi1   zfs_member 5000  home_data 3254388083323955506



Code:
zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
boot-pool   216G  22.9G   193G        -         -    10%    10%  1.00x    ONLINE  -
home_data  1.81T  1.14T   693G        -         -     0%    62%  1.00x    ONLINE  /mnt
main       11.0T  4.89T  6.09T        -         -     9%    44%  1.90x    ONLINE  /mnt
temp_smr   3.62T  1.25T  2.37T        -         -    10%    34%  1.00x    ONLINE  /mnt


Code:
zpool status -v
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:55 with 0 errors on Tue Jan 16 03:45:57 2024
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sde3      ONLINE       0     0     0

errors: No known data errors

  pool: home_data
 state: ONLINE
  scan: scrub repaired 0B in 01:43:18 with 0 errors on Sun Jan 21 01:43:19 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        home_data                                 ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            b01dfc68-ad6d-40ed-9b0b-c0f027a4a737  ONLINE       0     0     0
            ebe92af9-6937-42d6-9a11-bb46d7d021c7  ONLINE       0     0     0

errors: No known data errors

  pool: main
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 07:50:34 with 0 errors on Mon Jan  1 07:50:36 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        main                                      ONLINE       0     0     0
          520981dd-5f0e-4484-9546-06189625420c    ONLINE       0     0     0
          a8ae6e12-183c-4b52-8363-f12fcbae3dec    ONLINE       0     0     0
          f7682e70-f523-42c6-ae6e-d36ce208a4fe    ONLINE       0     0     0
        dedup
          mirror-3                                ONLINE       0     0     0
            f5bbba26-d6b4-469c-970b-8e5082cef094  ONLINE       0     0     0
            e686a9d0-2448-41b9-b7ff-e44d8ff4ec9d  ONLINE       0     0     0

errors: No known data errors

  pool: temp_smr
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 23:32:55 with 0 errors on Mon Jan  1 23:33:40 2024
config:

        NAME                                    STATE     READ WRITE CKSUM
        temp_smr                                ONLINE       0     0     0
          75e3d9ff-0c90-4b44-bbaf-69d87622b386  ONLINE       0     0     0

Is it enough hardware info or should i proivde something else?
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
ubergosu said:
I have 2 dedup mirrored ssd.
Click to expand...
I think we found the culprit. There are a few troubleshooting commands you should run to confirm this, but I have no time right now to try recalling them/searching for a dedup thread in order to confirm those. Will try coming back to you with something this evening (CEST).

In the meanwhile please look at the hardware list example in the forum rules thread and provide as much info as you can; I'm specially interested in RAM size and the SSDs model.
 
U

ubergosu

Cadet
Joined
Feb 26, 2022
Messages
9
Thank you for your reply.

Oh, didn't think that dedup could be the problem. If it's the problem I can even destroy that pool and recreate.

Thanks for pointing me to the rules and still replying, I'll read them more carefully.

My RAM is 16 Gb,

SSDs that are used for dedup vdev:

Code:
smartctl -a /dev/sdf
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.55-production+truenas] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Phison Driven SSDs
Device Model:     Patriot Burst
Serial Number:    A47207190B0C00598715
Firmware Version: SBFM61.5
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jan 22 17:51:02 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


and another one:

Code:
root@truenas[~]# smartctl -a /dev/sdd
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.55-production+truenas] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Phison Driven SSDs
Device Model:     Patriot Burst
Serial Number:    A45A07180B8600544249
Firmware Version: SBFM61.5
User Capacity:    120,034,123,776 bytes [120 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jan 22 17:45:01 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED



System SSD:


Code:
root@truenas[~]# smartctl -a /dev/sde
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.55-production+truenas] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 EVO 250GB
Serial Number:    S6P2NF0R531471H
LU WWN Device Id: 5 002538 f4150c815
Firmware Version: SVT01B6Q
User Capacity:    250,059,350,016 bytes [250 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jan 22 17:54:28 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


And new pool SSDs:


Code:
root@truenas[~]# smartctl -a /dev/sdh
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.55-production+truenas] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 QVO 2TB
Serial Number:    S5RPNJ0W503908K
LU WWN Device Id: 5 002538 f3355e1de
Firmware Version: SVQ02B6Q
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jan 22 17:58:08 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


and another one:


Code:
root@truenas[~]# smartctl -a /dev/sdi
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.55-production+truenas] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Samsung based SSDs
Device Model:     Samsung SSD 870 QVO 2TB
Serial Number:    S5RPNJ0W503911Y
LU WWN Device Id: 5 002538 f3355e1e1
Firmware Version: SVQ02B6Q
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
TRIM Command:     Available, deterministic, zeroed
Device is:        In smartctl database 7.3/5319
ATA Version is:   ACS-4 T13/BSR INCITS 529 revision 5
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Jan 22 17:58:43 2024 MSK
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Extremly long startup after adding new pool, kubernetes not starting"

Similar threads

E
  • Locked
SOLVED Kubernetes does not start after having defined the Apps pool
Replies
1
Views
7K
ebatsin
E
N
Kritical Kubernetes error on new TrueNAS SCALE install
Replies
4
Views
2K
nicholas-h
N
A
SOLVED Kubernetes service is not running - RC2 or Hardware Upgrade Issue
Replies
5
Views
3K
Antoshka
A
defendos
Restored configuration docker and kubernetes keep crashing
Replies
1
Views
650
morganL
M
P
Trying to understand Kubernetes implementation and how to use in multiple OSs
2
Replies
21
Views
20K
ornias
ornias
Top