Something is definitely wrong with GUI upgrades

[Daisuke]

I upgraded yesterday to 8.3.1 from 8.3.0 and now I face the same issues I previously had. The load stays low for a while and then it jumps to 1.3-1.5. I'm not a happy camper, it is a pain to open the case, pull the USB stick and redo all installation from scratch.

Also, is this considered noise? I don't see it in /var/log/messages or dmesg.

This is what I mean by hard to deal with the USB stick:

Edit: I reinstalled FreeNAS, everything is back to normal.

 

[cyberjock]

Not to sound rude, but you'll need to provide ALOT more information as to what is causing the load issues. Just saying "my server load goes really high suddenly" provides no information that the developers can use to find or fix the issue.

 

[Daisuke]

Hi Cyber,

Don't worry my friend (about the "rude" part). :)
Based on my experience, the GUI upgrade does something to network configuration if you don't configure the cards into GUI or terminal prompt and that is why I believe dhclient complains. I never setup the interfaces before.

I noticed that the load goes high as soon as I get (ada0:ahcich0:0:0:0): lost device into logs, g_event freaks. ada0 is my SSD cache disk:

Code:

# ataidle ada0                                                    
Model:                  M4-CT064M4SSD2                                          
Serial:                 000000001143031CF086                                    
Firmware Rev:           0009                                                  
ATA revision:           ATA-9                                                   
LBA 48:                 yes                                                     
Geometry:               16383 cyls, 16 heads, 63 spt                            
Capacity:               59GB                                                    
SMART Supported:        yes                                                     
SMART Enabled:          yes                                                     
Write Cache Supported:  yes                                                     
Write Cache Enabled:    yes                                                     
APM Supported:          yes                                                     
APM Enabled:            no                                                      
AAM Supported:          no

Crucial released a new firmware for this drive, I will upgrade it and see if the timeouts are gone. I remember a while ago when I reverted to 8.0.4 that they were gone like magic, which makes me believe it is something related to kernel.

So I will redo the process just to clear things up, as I have few USB sticks with different FreeNAS versions on it. First I will go back to 8.3.0, configure the interfaces into GUI then upgrade. Second, I will upgrade the SSD firmware to 040H. Hopefully this will cure all my "minor" yet nasty errors I get on each upgrade.

I will post the results tomorrow. :)

 

[cyberjock]

I noticed that the load goes high as soon as I get (ada0:ahcich0:0:0:0): lost device into logs, g_event freaks. ada0 is my SSD cache disk...

If the drive is being lost then the drive is having a problem communicating with the system. It could be a SATA controller, SATA cable, bad settings for the SATA controller BIOS or even something like a kernel driver bug with your SATA controller driver. If it doesn't come back online the SSD cache disk is providing no function at all and you should just keep it detached.

Could you please provide a SMART printout of your SSD drive? Your SSD may not be the cause of the high loading, but it is something that should be explored. Any drive being "lost" is a very bad sign if system reliability is important.

 

[Daisuke]

OK Cyber, some nice updates. :)
First and most important, is been 3 days I'm free of AHCI timeouts. Upgrading the SSD firmware solved all issues, I used to get many warnings during the day. The SMART report you requested:

Code:

# smartctl -a /dev/ada0
smartctl 5.43 2012-06-30 r3573 [FreeBSD 8.3-RELEASE-p6 amd64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron RealSSD C300/C400/m4
Device Model:     M4-CT064M4SSD2
Serial Number:    000000001143031CF086
LU WWN Device Id: 5 00a075 1031cf086
Firmware Version: 040H
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 6
Local Time is:    Tue Mar 26 23:57:48 2013 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (10235) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (   4) minutes.
Conveyance self-test routine
recommended polling time:        (   3) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       5288
 12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       91
170 Grown_Failing_Block_Ct  0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   001    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Wear_Levelling_Count    0x0033   098   098   010    Pre-fail  Always       -       75
174 Unexpect_Power_Loss_Ct  0x0032   100   100   001    Old_age   Always       -       3
181 Non4k_Aligned_Access    0x0022   100   100   001    Old_age   Always       -       792 583 208
183 SATA_Iface_Downshift    0x0032   100   100   001    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   001    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   001    Old_age   Always       -       0
189 Factory_Bad_Block_Ct    0x000e   100   100   001    Old_age   Always       -       62
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   001    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   001    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   001    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   001    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   001    Old_age   Always       -       0
202 Perc_Rated_Life_Used    0x0018   098   098   001    Old_age   Offline      -       2
206 Write_Error_Rate        0x000e   100   100   001    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Vendor (0xff)       Completed without error       00%      5282         -
# 2  Vendor (0xff)       Completed without error       00%      5269         -
# 3  Vendor (0xff)       Completed without error       00%      5257         -
# 4  Vendor (0xff)       Completed without error       00%      5245         -
# 5  Vendor (0xff)       Completed without error       00%      5232         -
# 6  Vendor (0xff)       Completed without error       00%      5220         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

For the second part related to network interfaces, if I don't configure into GUI the NIC's and upgrade, I will get the dhclient warnings you saw into first screenshot. The solution for me was to configure the 2 cards. It would be nice if a fix is provided for this issue, even if I consider it trivial.

I'm glad that I'm error free on my box, no more issues. :D

 

[cyberjock]

181 Non4k_Aligned_Access 0x0022 100 100 001 Old_age Always - 792 583 208

That sucks. That may not be something that can be fixed, but having 4k aligned writes can increase performance by alot depending on the SSD. I'd look into how you could fix the alignment of your writes on the SSD as this can cause premature aging.

How long has that SSD been installed? What is the server used for? SMART says it has 98% life remaining, so I'm curious. :)

 

[Daisuke]

> I'd look into how you could fix the alignment of your writes on the SSD as this can cause premature aging.
Thanks!

> How long has that SSD been installed? What is the server used for? SMART says it has 98% life remaining, so I'm curious.
I store media and development files, on it. I would say that the reads/writes I do are pretty aggressive. Check my build, all info is there related to purchase dates. :)