SOLVED Concerned about getting ahcichX: Timeout on XX port 0

Status
Not open for further replies.

Zpid

Cadet
Joined
Dec 28, 2014
Messages
9
Hi

I have just made a new build after weeks of research of what hardware to choose, it feels like it's not to bad when comparing to for example "Freenas Mini" (i think same motherboard as i choose.)

Motherboard: ASRock C2750D4I
Memory: 4 x Kingston ValueRAM TS DDR3 PC12800/1600MHz ECC CL11 8GB (KVR16E11/8)
Disks: 6 x WD Red WD40EFRX 64MB 4TB in raidz2
Case: Fractal Design Node 804
PSU: Corsair 450W
Build: FreeNAS-9.3-STABLE-201412312006

However, when running scrub and i suspect when i started to move files to it from my old Qnap i saw reports in the mailed log and also when checking /var/log/messages like these:


Code:
Jan  5 20:19:24 freenas ahcich2: Timeout on slot 28 port 0
Jan  5 20:19:24 freenas ahcich2: is 00000000 cs 10000000 ss 00000000 rs 10000000 tfd 50 serr 00000000 cmd 10009c17

Jan  6 15:19:24 freenas ahcich4: Timeout on slot 30 port 0
Jan  6 15:19:24 freenas ahcich4: is 00000000 cs 40000000 ss 00000000 rs 40000000 tfd 40 serr 00000000 cmd 10009e17

Jan  6 16:49:23 freenas ahcich5: Timeout on slot 28 port 0
Jan  6 16:49:23 freenas ahcich5: is 00000000 cs 10000000 ss 00000000 rs 10000000 tfd 40 serr 00000000 cmd 10009c17
Jan  6 17:19:24 freenas ahcich3: Timeout on slot 17 port 0
Jan  6 17:19:24 freenas ahcich3: is 00000000 cs 00020000 ss 00000000 rs 00020000 tfd 40 serr 00000000 cmd 10009117


No errors reported in the pool:
Code:
[root@freenas] ~# zpool status
  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: redpool
state: ONLINE
  scan: scrub in progress since Tue Jan  6 13:14:57 2015
        6.26T scanned out of 13.1T at 386M/s, 5h10m to go
        0 repaired, 47.75% done
config:

        NAME                                                STATE     READ WRITE CKSUM
        redpool                                             ONLINE       0     0     0
          raidz2-0                                          ONLINE       0     0     0
            gptid/9675d4ca-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0
            gptid/96ef2f30-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0
            gptid/97654887-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0
            gptid/97dc5cdf-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0
            gptid/9851e6e7-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0
            gptid/98ca791e-9057-11e4-82f3-d05099504507.eli  ONLINE       0     0     0

errors: No known data errors


Code:
[root@freenas] ~# camcontrol devlist
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus1 target 0 lun 0 (pass1,ada1)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus3 target 0 lun 0 (pass3,ada3)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus4 target 0 lun 0 (pass4,ada4)
<WDC WD40EFRX-68WT0N0 82.00A82>    at scbus5 target 0 lun 0 (pass5,ada5)
<Marvell Console 1.01>             at scbus9 target 0 lun 0 (pass6)
<Kingston DataTraveler 3.0 PMAP>   at scbus17 target 0 lun 0 (pass7,da0)


Before i started to use the system i did run SMART smartctl tests on all disks, both short and long and that did not show any errors.

Are these "ahcich" timeouts something to be concerned of that i should try to fix before using this NAS as my primary?
 

Zpid

Cadet
Joined
Dec 28, 2014
Messages
9
To answer my own question, this was when all 6 disks was connected to the marwell controllers because of short cables, i found a new firmware that disables the HW raid because of "stability problems"

http://www.asrockrack.com/support/ipmi.asp#Marvell9230

Marvell 9230 FW update Procedure
Since we found Marvell 9230 Raid Mode get some problem at some OS, we already highlighted this issue to Marvell.
In the short term solution, can disable Marvell 9230 HW Raid function to make Marvell 9230 port more stable, please follow below procedure to flash Marvell 9230 FW to disable HW Raid function.

There is no use for HW raid in freenas anyway, and after some heavy file moves and scrub running all night, no new error.

The update seems to have updated the BIOS to 2.80 also, so i think if one would flash the normal bios it would enable again.

Also, updated the BMC with the brand new 00.23.001/7/2015 version, so now IPMI works again with newer Java versions.

Now to my next problem with horrible NFS performance..

EDIT: NFS performance problem solved!

So after reading up alot on NFS and especially when using some of the storage for ESXi to try and resolve the horrible performance when migrating machines or just starting one machine on the storage.
What would happend is the performance when streaming a low bitrate video via NFS to XBMC while VMWare ESXi was using the storage it would buffer every minute or more often, also copy operations from another machine using NFS was really slow.

The solution was adding a Intel 530 SSD as zil cache! Now it's smooth and quick even in vmware :)

So to anyone using the storage with NFS mounts to ESXi, you will need ZIL!

I set up 10GB as ZIL, this might be overkill, the creator of ZFS stated that 4GB should be good i heard, hoping that this intel disk will do wear leveling and not die too quick.

I reused my desktop SSD (good reason to upgrade that one to 250GB), i did also erase it in freenas, and found a few posts on how to do it in these forums so i will repeat it quick to aid if someone needs to find instructions:

To check how hammered the disks are and if ZIL is in use:

Code:
gstat -I 20000 -f gpt


To create GPT and partition a smaller size and add it to the pool, change ada6 to what your SSD is called and <poolname> to what you named the pool, as i said maybe 10GB is overkill..:

Code:
gpart create -s GPT ada6
gpart add -t freebsd-zfs -l myzil -s 10G -b 1M ada6
zpool add <poolname> log /dev/gpt/myzil
 
Last edited:

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
To answer my own question, this was when all 6 disks was connected to the marwell controllers because of short cables, i found a new firmware that disables the HW raid because of "stability problems"

http://www.asrockrack.com/support/ipmi.asp#Marvell9230



There is no use for HW raid in freenas anyway, and after some heavy file moves and scrub running all night, no new error.

The update seems to have updated the BIOS to 2.80 also, so i think if one would flash the normal bios it would enable again.

Also, updated the BMC with the brand new 00.23.001/7/2015 version, so now IPMI works again with newer Java versions.

Now to my next problem with horrible NFS performance..

EDIT: NFS performance problem solved!

So after reading up alot on NFS and especially when using some of the storage for ESXi to try and resolve the horrible performance when migrating machines or just starting one machine on the storage.
What would happend is the performance when streaming a low bitrate video via NFS to XBMC while VMWare ESXi was using the storage it would buffer every minute or more often, also copy operations from another machine using NFS was really slow.

The solution was adding a Intel 530 SSD as zil cache! Now it's smooth and quick even in vmware :)

So to anyone using the storage with NFS mounts to ESXi, you will need ZIL!

I set up 10GB as ZIL, this might be overkill, the creator of ZFS stated that 4GB should be good i heard, hoping that this intel disk will do wear leveling and not die too quick.

I reused my desktop SSD (good reason to upgrade that one to 250GB), i did also erase it in freenas, and found a few posts on how to do it in these forums so i will repeat it quick to aid if someone needs to find instructions:

To check how hammered the disks are and if ZIL is in use:

Code:
gstat -I 20000 -f gpt


To create GPT and partition a smaller size and add it to the pool, change ada6 to what your SSD is called and <poolname> to what you named the pool, as i said maybe 10GB is overkill..:

Code:
gpart create -s GPT ada6
gpart add -t freebsd-zfs -l myzil -s 10G -b 1M ada6
zpool add <poolname> log /dev/gpt/myzil
Good post sir. I expect it will help many users.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
It's good to see ASRock is willing to troubleshoot these issues and work on solving them.
 

samesong

Cadet
Joined
Jan 8, 2015
Messages
8
Thank you for posting this Zpid. I have almost the same setup as you and am also migrating files. Hopefully this patch sorts things out.
 

Wim Mistiaen

Dabbler
Joined
Mar 21, 2015
Messages
22
Thanks for this info. I ran into the same issue last night with the i5 equivalent of this mobo (c2550d4i). i'm going to flash this controller later today.

I just hope my disks still be readable after the flash :)

cheers!
 
Status
Not open for further replies.
Top