spa_sync timeout on boot up

[sehang]

* Updated on Jan 6, 2018.
========
I used a new USB stick to install a fresh FreeNAS 11.0-U4. Trying to import the pool #2 from the fresh FreeNAS. The fresh FreeNAS does not work, the problem is the same.

I then go to the local shop and buy a new LSI HBA card (very expensive.....). The new card is LSI SAS 9207-4i4e. I can then able to import the pool from the new card. At that time, I though it may be a HBA card issue. ACTUALLY, IT IS NOT ROOT CAUSE.

My FreeNAS is now very slow. Probably the last crash caused by the same problem. I received many errors similar to the following:

Code:

> tap2: link state changed to DOWN
> tap2: Ethernet address: 00:bd:96:4f:18:02
> tap2: promiscuous mode enabled
> tap2: link state changed to UP
> tap1: link state changed to DOWN
> tap1: Ethernet address: 00:bd:90:91:19:01
> tap1: promiscuous mode enabled
> tap1: link state changed to UP
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (435 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (642 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1112 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1195 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1267 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (823 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (652 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1195 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1260 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1173 occurrences)
> sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow: 193 already in queue awaiting acceptance (1129 occurrences)
> Limiting open port RST response from 245 to 200 packets/sec

I have attempted to reproduce these errors. It is caused by the deletion of large number of files. For example, I created 300,000 files, they are images, HTML files, javascript files, etc. These are small files (around few KBytes). The total size of 300,000 files is less than 500GB. I created a SMB share and access these files from a Windows PC. "sonewconn: pcb 0xfffff8077507e1d0: Listen queue overflow" will occur sometime if I delete all these files.

Is it a network issue? Because I cannot fine any listen queue related to 0xfffff8077507e1d0. I used

Code:

netstat -Lan | grep fffff8077507e1d0

but no result is returned. I also run

Code:

netstat -Lan | grep fffff8077507e1d0

in all my jails, all returns empty.




========
My FreeNAS server was suddenly rebooted a week ago (it never happen in the past). After that, one of pool cannot import on boot up.

My hardware information and FreeNAS information are shown as follows:
- FreeNAS version is: 11.0-U4
- Motherboard: Supermicro X10DRi-T
- Memory: 150GB ECC RAM
- HBA card: LSI SAS 9207-8i 6Gb
- SAS Expansion Card: HP 468406-B21 32 port

I have two pool (pool #1 and pool #2), the pool #1 is OK to import.
pool #1: raid-z1 (8 x 3TB SEAGATE)
pool #2: raid-z1 (8 x 3TB HGST) + raid-z1 (4 * 6TB WD RED) + raid-z1 (4 * 8TB WD RED)

pool #2 is failed to import.

My resolution:
1. I unpluged all HDDs from pool #1 and let the pool #2 run alone. Same problem.
2. I login the FreeNAS using the single user mode and import the pool #2, it works. Therefore, I run the zpool scrub on the pool #2. I take around 60 hours (about 3 days) and everything is OK. I then reboot the system again, the problem is the same...

I have no idea right now, any help and suggestion would be appreciated.

Sorry for my bad English.

Thank you very much!

 

[Nick2253]

I wonder if there is a problem with your boot device. Can you try backing up your config, installing FreeNAS to a new boot device, restore your config, and see if that changes anything?

 

[sehang]

I wonder if there is a problem with your boot device. Can you try backing up your config, installing FreeNAS to a new boot device, restore your config, and see if that changes anything?

My FreeNAS server is using the SanDisk 120GB SSD for the boot drive.

I am preparing a USB stick to install a fresh FreeNAS, will update later.

I have updated the latest status in the first post.

Thank you.