FreeNAS Keeps Shutting Down - "Unable to extract pool GUID from vdev label data"

[Jakg]

Hi, have an odd problem where my FreeNAS box keeps shutting down (gracefully).

It seems to be related to load - it can idle for days, but if used hard it will happen in a couple of hours.

It's complaining about a drive that's being used as a cache / ZIL drive.

If I restart, it all works fine... for a while.

It looks to me like there was some kind of problem with the ZIL drive (maybe it crashed, maybe there was a read error, maybe the code that does caching just broke) and a graceful shutdown was initiated?

I'd really appreciate some thoughts!

Specifically, I've got the following excerpt from the log at the time when the fault happened. There was nothing in the log in the 14 hours before this. I've also tried to include any relevant other information.

Code:

Dec 24 18:44:14 freenas zfsd: DevfsEvent::ReadLabel: /dev/ada4p2: Unable to extract pool GUID from vdev label data.
Dec 24 18:44:15 freenas afpd[2966]: AFP Server shutting down
Dec 24 18:44:15 freenas cnid_metad[2967]: shutting down on SIGTERM
Dec 24 18:44:15 freenas netatalk[2951]: Netatalk AFP server exiting
Dec 24 18:44:15 freenas smbd[14822]: dnssd_clientstub DNSServiceProcessResult called with DNSServiceRef with no ProcessReply function
Dec 24 18:44:17 freenas winbindd[14826]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=1)
Dec 24 18:44:17 freenas winbindd[14979]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 24 18:44:17 freenas winbindd[14827]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 24 18:44:17 freenas winbindd[14980]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 24 18:44:18 freenas nmbd[14818]: [2015/12/24 18:44:18.599678,  0] ../source3/nmbd/nmbd.c:57(terminate)
Dec 24 18:44:18 freenas nmbd[14818]:   Got SIGTERM: going down...
Dec 24 18:44:18 freenas ntpd[2349]: ntpd exiting on signal 15
Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:19 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:20 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:21 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:22 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:25 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:26 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:27 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:28 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:31 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:34 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:37 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:38 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:40 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:43 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:44 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:45 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:46 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:49 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:50 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:51 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 24 18:44:52 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:57 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 24 18:44:58 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3

Debug File:
https://www.dropbox.com/s/4q5fqheag1tg7dx/debug-freenas-20151226003325..tgz?dl=0

PC Info:

Code:

Make / Model: HP Microserver N54L.
CPU: AMD Turion II Neo N54L - 2.2 GHz.
Memory: 16GB DDR3 ECC, 2x 8GB DDR1600.
Hard Drives: 14.3TB RAID-Z1, 1x OCZ Vertex 2 120GB SSD (Cache), 5x Seagate 7200.15 4TB, 1x Sandisk Cruzer Contour 8GB.
Power Supply: XFX Pro 450w Core Edition.
Network Card: Intel PRO/1000 PT PCI-E Gigabit Network Card.
Operating System: FreeNAS 9 64-bit (FreeNAS-9.3-STABLE-201512121950)

 

[tvsjr]

You indicate the system is under heavy load. Have you looked at the system temperatures? Also, the results of a long SMART test on the cache drive?

 

[gpsguy]

Given what you've said and looking at the output of "zpool status", it appears that you put both the logs and cache on the same OCZ SSD drive.

From the manual, these should have been separate drives. And, OCZ is not recommended.

With just 16GB of RAM, having a cache "drive" is probably hurting your performance.

Are you using the ODD and eSATA connections on your N54L for the 5th drive and SSD? If so, are you running one of the "hacked BIOS's"? If not, the connections are basically IDE, instead of AHCI.

Code:

• log device: requires at least one dedicated device, where an SSD is recommended
• cache device: requires at least one dedicated device, where an SSD is recommended

Code:

  pool: Storage
state: ONLINE
  scan: scrub repaired 0 in 38h34m with 0 errors on Mon Dec  7 14:34:27 2015
config:

   NAME  STATE  READ WRITE CKSUM
   Storage  ONLINE  0  0  0
    raidz1-0  ONLINE  0  0  0
    gptid/fad60e46-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fb3a82a4-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fb8b998b-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fbd7a5ea-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fc63e143-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
   logs
    gpt/log  ONLINE  0  0  0
   cache
    gpt/cache  ONLINE  0  0  0 

It's complaining about a drive that's being used as a cache / ZIL drive.

 

[Jakg]

You indicate the system is under heavy load. Have you looked at the system temperatures? Also, the results of a long SMART test on the cache drive?

When I said "heavy load", I was perhaps stretching things a little.

As a test, when I started testing the crash I looped playback of a few films off the drive to keep it alive and busy, but no real load. In this state it worked for days.

I then tried to copy about 10,000 files amounting to 55GB. In this state, it shut down after a couple of hours.

I'm not able to say for sure it's not the temperature, but in my experience overheating would lead to an instant, not graceful, shutdown.

In any case I'm now monitoring the CPU temp - it's currently at 43° C while doing a scrub which I think is fine.

I've run the SMART test, which is below

Code:

[root@freenas] ~# smartctl -A /dev/ada4
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   104   104   050    Pre-fail  Always       -       0/8305896
  5 Retired_Block_Count     0x0033   100   100   003    Pre-fail  Always       -       0
  9 Power_On_Hours_and_Msec 0x0032   100   100   000    Old_age   Always       -       20637h+20m+50.530s
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       413
171 Program_Fail_Count      0x0032   000   000   000    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   000   000   000    Old_age   Always       -       0
174 Unexpect_Power_Loss_Ct  0x0030   000   000   000    Old_age   Offline      -       185
177 Wear_Range_Delta        0x0000   000   000   000    Old_age   Offline      -       6
181 Program_Fail_Count      0x0032   000   000   000    Old_age   Always       -       0
182 Erase_Fail_Count        0x0032   000   000   000    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
194 Temperature_Celsius     0x0022   030   030   000    Old_age   Always       -       30 (Min/Max 30/30)
195 ECC_Uncorr_Error_Count  0x001c   104   104   000    Old_age   Offline      -       0/8305896
196 Reallocated_Event_Count 0x0033   100   100   000    Pre-fail  Always       -       0
231 SSD_Life_Left           0x0013   096   096   010    Pre-fail  Always       -       0
233 SandForce_Internal      0x0000   000   000   000    Old_age   Offline      -       42496
234 SandForce_Internal      0x0032   000   000   000    Old_age   Always       -       45888
241 Lifetime_Writes_GiB     0x0032   000   000   000    Old_age   Always       -       45888
242 Lifetime_Reads_GiB      0x0032   000   000   000    Old_age   Always       -       20544

Nothings jumping out at me as a problem there, but I can't say I understand it all.

Given what you've said and looking at the output of "zpool status", it appears that you put both the logs and cache on the same OCZ SSD drive.

From the manual, these should have been separate drives. And, OCZ is not recommended.

With just 16GB of RAM, having a cache "drive" is probably hurting your performance.

Are you using the ODD and eSATA connections on your N54L for the 5th drive and SSD? If so, are you running one of the "hacked BIOS's"? If not, the connections are basically IDE, instead of AHCI.

Code:

• log device: requires at least one dedicated device, where an SSD is recommended
• cache device: requires at least one dedicated device, where an SSD is recommended

Code:

  pool: Storage
state: ONLINE
  scan: scrub repaired 0 in 38h34m with 0 errors on Mon Dec  7 14:34:27 2015
config:

   NAME  STATE  READ WRITE CKSUM
   Storage  ONLINE  0  0  0
    raidz1-0  ONLINE  0  0  0
    gptid/fad60e46-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fb3a82a4-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fb8b998b-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fbd7a5ea-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
    gptid/fc63e143-bcf3-11e3-904d-6805ca076dc5  ONLINE  0  0  0
   logs
    gpt/log  ONLINE  0  0  0
   cache
    gpt/cache  ONLINE  0  0  0

You are correct, they are on the same drive. My understanding was that this isn't ideal, but is far from a problem. And shouldn't cause the issues I'm experiencing.

I also don't see why OCZ isn't recommended and can't find any reference for this, although I do know my model's reliability history isn't great. My understanding was that in the event of it failing, I will lose all of the writes that hadn't been flushed (maybe 10 seconds worth) and the pool would fall over until I could remove the drive, which I was happy with.

You raise a good point regarding memory - now I understand it a bit more I think that just having a L2ARC cache incorporates a memory overhead. I can't say I've ever run out of memory, however:

Unfortunately I can't add any more memory.

I would be interested in working out a way of benchmarking if performance is actually improved with the L2ARC in my case - I had the SSD spare and thought it would be better in the system than in a drawer somewhere.

And yes, I am using the additional SATA ports on my Microserver, and I have a modified BIOS to allow it to run in AHCI mode.

I think it's interesting you've focussed in on the SSD so as a test I've removed the L2ARC / ZIL and unplugged the SSD to see if this improves stability.

 

[Jakg]

Unfortunately, even with the caching stuff disabled, and the drive unplugged - it's still doing it. Log below:

Code:

Dec 26 19:21:28 freenas afpd[2862]: AFP Server shutting down
Dec 26 19:21:28 freenas cnid_metad[2863]: shutting down on SIGTERM
Dec 26 19:21:28 freenas netatalk[2847]: Netatalk AFP server exiting
Dec 26 19:21:29 freenas winbindd[2613]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=1)
Dec 26 19:21:29 freenas winbindd[2619]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 26 19:21:29 freenas winbindd[2618]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 26 19:21:29 freenas winbindd[2617]:   STATUS=daemon 'winbindd' finished starting up and ready to serve connectionsGot sig[15] terminate (is_parent=0)
Dec 26 19:21:29 freenas nmbd[2607]:   STATUS=daemon 'nmbd' finished starting up and ready to serve connectionsGot SIGTERM: going down...
Dec 26 19:21:29 freenas upsmon[2436]: upsmon parent: read
Dec 26 19:21:30 freenas upsd[2430]: mainloop: Interrupted system call
Dec 26 19:21:30 freenas ntpd[2307]: ntpd exiting on signal 15
Dec 26 19:21:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:32 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:33 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:34 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:34 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:34 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:35 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:21:35 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 26 19:21:35 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:21:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:39 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:40 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:40 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:40 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:41 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:41 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:41 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:42 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:21:42 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 26 19:21:42 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:21:46 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:46 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:46 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:47 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:47 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:47 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:48 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:48 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:48 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:49 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:21:49 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:21:49 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 26 19:21:53 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:53 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:53 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:21:54 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:54 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:55 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:21:55 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:55 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:56 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:21:56 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:21:57 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:21:57 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 26 19:22:01 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:01 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:01 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:02 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:02 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:02 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:03 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:03 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:03 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:04 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:22:04 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused
Dec 26 19:22:04 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:22:07 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:07 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:07 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 26 19:22:08 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:08 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:08 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 26 19:22:09 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:09 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:09 freenas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 26 19:22:10 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:4 Err:-1 Errno:61 Connection refused
Dec 26 19:22:10 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:7 Err:-1 Errno:61 Connection refused
Dec 26 19:22:10 freenas python: dnssd_clientstub ConnectToServer: connect() failed Socket:8 Err:-1 Errno:61 Connection refused

 

[gpsguy]

You thought that it might be an issue with your "ZIL drive" in original message. It wasn't until I drilled down into your debug file, that I realized that you were using the same SSD for both the log and cache devices.

Again, quoting from the manual - "When creating a dedicated log device, it is recommended to use a fast SSD with a supercapacitor or a bank of capacitors that can handle writing the contents of the SSD’s RAM to the SSD." Typically, it's an Intel 3700 series SSD or 750 series PCI/e SSD.

As far as what happens when you put both functions on the same drive, when the documentation clearly says to use a dedicated drive, well, I don't know what will happen.

As you've now discovered for yourself, using it for L2ARC is taking memory away from your base system RAM.

I realize the limitations of the N54L, since I also have one (and a N40L too).

I think it's interesting you've focussed in on the SSD so as a test I've removed the L2ARC / ZIL and unplugged the SSD to see if this improves stability.

 

[gpsguy]

Were you doing this today, when it shutdown?

I don't know a think about AFP, but I've heard that it's been deprecated and a lot of Apple users on the forum are using CIFS instead.

Could you try copying the files via CIFS instead?

I then tried to copy about 10,000 files amounting to 55GB. In this state, it shut down after a couple of hours.

 

[Jakg]

Were you doing this today, when it shutdown?

I don't know a think about AFP, but I've heard that it's been deprecated and a lot of Apple users on the forum are using CIFS instead.

Could you try copying the files via CIFS instead?

It was doing a scrub, that seems to accelerate things. I wasn't explicitly using AFP, but the service was running. It's restarted without any clients being connected in other instances.

When I was copying the files, that was via CIFS.

I don't know if I even use AFP - on my Mac I use smb to access the CIFS share, and Time Machine.

 

[jgreco]

As far as what happens when you put both functions on the same drive, when the documentation clearly says to use a dedicated drive, well, I don't know what will happen.

Nothing will happen, except lots of stupid.

The purpose and intent of the SLOG SSD is to provide data protection in the event of certain bad things happening. Using an SSD without the requisite capabilities means that you are slowing down your filer and also wearing out flash pages for no particularly good purpose other than to kill a crappy OCZ SSD, which, while admirable, isn't actually doing anything useful for you. Please read the SLOG sticky to understand the purpose of a SLOG, and why using a device without the requisite capabilities is effectively pointless.

Putting SLOG on the same SSD as L2ARC basically just wears the SSD out even faster. As a method for cooking an OCZ SSD to death, I give it rave reviews, but as for doing anything useful for your system, it is just going to cause you trouble.

 

[Jakg]

I think I owe FreeNAS an apology - looked in the BIOS log and could see errors relating to the fan speed.

I think the fan header had become loose and caused an intermittent loss of connection, which caused the server to do a graceful shutdown.

Having reseated the connection it's been working fine since.

 

[jgreco]

I think I owe FreeNAS an apology

Fortunately your FreeNAS is a very easygoing creature, just give it a little tender loving care and maybe a nice dusting off of its fans, and all will be forgiven.