I've searched. I've tried. I need help. CIF share is SLOW and something will stop

hidperf · Oct 19, 2012

paleoN said:
No and turn off compression, if enabled, during the test. The dd commands you should be using are the ones in the [thread=981]performance sticky[/thread].

Thanks for the link. I swear I searched but I didn't come across that page. I'll gather some info when the SMART test is finished running. I have compression off right now.

paleoN said:
How about the manpage? Don't forget to specify the proper device.

Reconfigure it at least. ZFS can make better decisions when it knows how many disks it's dealing with. If the RAID card has a write cache with a BBU and you want to take advantage of it that's fine. If it doesn't then there's no reason I can think of to use it.

I used the commands from that page but couldn't get anything to work with the RAID card installed. I had also read I needed to use different commands for the LSI based card but couldn't get any results with that command set either. So I've since removed the card and now everything is working, command line wise, as it should.

Thank you for the help.

matram · Oct 20, 2012

Some benchmark results

I tried some different configurations with CIFS with limited success, but here are the results if anyone are interested. Numbers are MB/s reading a 1 GB file from the NAS.

FreeNAS with ZFS, 6 GB
... default CIFS config: 28.0

FreeNAS with UFS, 1 GB of RAM
... fresh install with default config: 23.8
... with DOS support off: 25.1
... plus Large file support: 26.3
... with 2GB RAM, plus autotune: 27,4

As comparison
FreeNAS (ZFS, 6 GB) NFS share: 45.3

And CIFS performance on other NAS systems:
Nextenta: 35.2
OpenFiler: 48.5

Raw performance of the disk array is 175 MB/s.

/Mats

hidperf · Oct 20, 2012

Here's what I got for results:

Code:

smartctl -l selftest /dev/ada0
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Short offline       Completed without error       00%       762         -
# 3  Extended offline    Completed without error       00%       756         -

Code:

smartctl -l selftest /dev/ada1
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       765         -
# 2  Extended offline    Completed without error       00%       756         -

Code:

smartctl -l selftest /dev/ada2
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

Code:

smartctl -l selftest /dev/ada3
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       765         -
# 2  Extended offline    Completed without error       00%       756         -

Code:

 smartctl -l selftest /dev/ada4
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

Code:

smartctl -l selftest /dev/ada5
smartctl 5.42 2011-10-20 r3458 [FreeBSD 8.2-RELEASE-p9 amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

Here's the results I get from the code in the stickied performance page.

Read

Code:

dd if=/dev/zero of=/mnt/freenas/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 304.359955 secs (352786826 bytes/sec)

Write

Code:

 dd if=/mnt/freenas/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 281.560926 secs (381353279 bytes/sec)

Seems awful slow compared to others results on that page.

matram · Oct 20, 2012

hidperf said:
Write

Code:
dd if=/mnt/freenas/tmp.dat of=/dev/null bs=2048k count=50k 51200+0 records in 51200+0 records out 107374182400 bytes transferred in 281.560926 secs (381353279 bytes/sec)

Seems awful slow compared to others results on that page.

You got 363 MB/s READ performance which I would say is very good :)

The sustained transfer rate from a single WD RED disk is 150 MB/s. The reason you get higher is the striping over 4+2 disks in your RAID6 array.

/Mats

paleoN · Oct 20, 2012

The smartctl -l isn't worth much. I should have specified what I was looking for:

Code:

smartctl -q noserial -a /dev/adaX

For each drive if you would.

hidperf said:
Seems awful slow compared to others results on that page.

Your results are on the lower, faster end. The people significantly faster had more spindles, 10K rpm disks or compression on which gives false results.

hidperf · Oct 20, 2012

matram said:
You got 363 MB/s READ performance which I would say is very good :)

The sustained transfer rate from a single WD RED disk is 150 MB/s. The reason you get higher is the striping over 4+2 disks in your RAID6 array.

/Mats

You know, after I wrote that I went and checked out the WD specs and wondered if that was the case. But I'm not sure what I'm looking for so I was hoping someone could fill me in.

paleoN said:
The smartctl -l isn't worth much. I should have specified what I was looking for:
Code:
smartctl -q noserial -a /dev/adaX
For each drive if you would.

Your results are on the lower, faster end. The people significantly faster had more spindles, 10K rpm disks or compression on which gives false results.

Here you go. What exactly are you looking for or what should I be looking for?

Code:

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10EFRX-68JCSN0
Firmware Version: 01.01A01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sat Oct 20 14:30:48 2012 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12660) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 145) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   152   136   021    Pre-fail  Always       -       3358
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       81
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       81
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       78
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   107   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Short offline       Completed without error       00%       762         -
# 3  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10EFRX-68JCSN0
Firmware Version: 01.01A01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sat Oct 20 14:37:38 2012 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (13500) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 154) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   155   137   021    Pre-fail  Always       -       3233
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       81
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       81
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       78
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   109   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       765         -
# 2  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (13200) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 151) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   155   135   021    Pre-fail  Always       -       3241
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       81
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       81
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       78
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   107   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (12960) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 148) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   152   135   021    Pre-fail  Always       -       3391
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       84
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       84
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       81
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   115   107   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       765         -
# 2  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (13140) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 150) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   155   136   021    Pre-fail  Always       -       3225
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       84
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       84
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       81
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   114   107   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (13320) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 152) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x30bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   157   135   021    Pre-fail  Always       -       3133
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       82
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       777
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       82
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       79
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2
194 Temperature_Celsius     0x0022   114   108   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   147   000    Old_age   Always       -       300
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%       766         -
# 2  Extended offline    Completed without error       00%       756         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

matram · Oct 20, 2012

Your SMART test show no errors on any drive. So I should put that aside for now.

Also note that the "dd" is run with a long block length 2 MB. I believe the maximum for CIFS is 60 kB. The file length is also extreme at 100+ GB. If you shorten the file and reduce blocklengths measured performance will go down.

If you will use freenas over 1GB network you will anyhow not be able to use more than at best 100 MB/s.

What i find interesting is that FreeNAS (for me) is not able to deliver more than 15% of the underlying performance through CIFS.

/Mats

paleoN · Oct 20, 2012

hidperf said:
Here you go. What exactly are you looking for or what should I be looking for?

Some of the major ones to look out for are:

Code:

  5 Reallocated_Sector_Ct

197 Current_Pending_Sector

198 Offline_Uncorrectable

199 UDMA_CRC_Error_Count

194 Temperature_Celsius

You would also like to see sane values in:

Code:

193 Load_Cycle_Count

196 Reallocated_Event_Count

200 Multi_Zone_Error_Rate

Despite Matram's expert analysis those attributes, particularly the first group, can show signs of a hard drive going south some time before it starts failing SMART tests.

All the drives look fine except for the last one, ada5 I guess.

Code:

199 UDMA_CRC_Error_Count    0x0032   200   147   000    Old_age   Always       -       300

A couple over the lifetime of the drive isn't a big deal, but this is far too many particularly for a drive this new. These indicate a hardware-level problem between the host controller and the controller chip on the drive. Some possible causes: faulty/loose cables, faulty SATA ports or dusty ones, faulty controller or even faulty PSU. Not to mention a defective drive. Swapping the cable is the easiest and one of the likeliest things to do which I suggest.

matram said:
Also note that the "dd" is run with a long block length 2 MB. I believe the maximum for CIFS is 60 kB. The file length is also extreme at 100+ GB. If you shorten the file and reduce blocklengths measured performance will go down.

Did you read the first post of the [thread=981]performance sticky[/thread]? The reasons for the values are explained quite well I thought. You are correct that the tests do not correlate at all with CIFS. This was just to get a baseline for the raw zpool performance. Which is quite good.

Once that one drive is taken care of we can move on to actual FreeNAS/CIFS tunings.

hidperf · Oct 20, 2012

paleoN said:
Some of the major ones to look out for are:
Code:
5 Reallocated_Sector_Ct 197 Current_Pending_Sector 198 Offline_Uncorrectable 199 UDMA_CRC_Error_Count 194 Temperature_Celsius

You would also like to see sane values in:
Code:
193 Load_Cycle_Count 196 Reallocated_Event_Count 200 Multi_Zone_Error_Rate

Despite Matram's expert analysis those attributes, particularly the first group, can show signs of a hard drive going south some time before it starts failing SMART tests.

All the drives look fine except for the last one, ada5 I guess.
Code:
199 UDMA_CRC_Error_Count 0x0032 200 147 000 Old_age Always - 300
A couple over the lifetime of the drive isn't a big deal, but this is far too many particularly for a drive this new. These indicate a hardware-level problem between the host controller and the controller chip on the drive. Some possible causes: faulty/loose cables, faulty SATA ports or dusty ones, faulty controller or even faulty PSU. Not to mention a defective drive. Swapping the cable is the easiest and one of the likeliest things to do which I suggest.

Did you read the first post of the [thread=981]performance sticky[/thread]? The reasons for the values are explained quite well I thought. You are correct that the tests do not correlate at all with CIFS. This was just to get a baseline for the raw zpool performance. Which is quite good.

Once that one drive is taken care of we can move on to actual FreeNAS/CIFS tunings.

Thanks for the insight. I swapped the SATA cable, I'm assuming I run the SMART test on that drive again and see if the results change? Or do I just run the performance test?

paleoN · Oct 20, 2012

hidperf said:
I swapped the SATA cable, I'm assuming I run the SMART test on that drive again and see if the results change? Or do I just run the performance test?

Get an updated smartctl -a of the drive and note the current count for 199 UDMA_CRC_Error_Count. Then do a bunch of file I/O and check to see if it's still increasing. The SMART test won't work as it's internal to the drive.

hidperf · Oct 20, 2012

I transferred a 7.7gig file between the two machines about a dozen times each way and ran the smartctl read and write test a couple times. The counter didn't change from the 300 it was before I started moving data.

Looks like transfer speeds are up, just not steady. This is good news right?

matram · Oct 21, 2012

Status?

hidperf said:
I put together a FreeNAS 8.2 box to use as a back up for my Windows machines, an SSH server and a storage device for my movies, photos and music. I've been messing with it for a few weeks now trying to get the transfer speeds up from the Windows box to the NAS but I can't seem to make any headway. At first I was getting a burst of 30MB/s speeds followed by 5MB/s, which seems to be a common thing from my Google searches. I found a few post and tried a few things and they only made things worse. Now I get 94MB/s burst initially and then it dies and won't complete the transfer.

hidperf said:
I'm getting 5MB/s and it "wavy". It's not a steady stream of data transfer, it's more like a large oscillation from 5MB/s to 0.

hidperf said:
BTW, when I copy a 7.76GB ISO from the FreeNAS box to the windows box I'm getting around 70MB/s. If that helps at all.

matram said:
You got 363 MB/s READ performance which I would say is very good :)

I would be helpful if you tried to clarify what the current status actually is with regard to your CIFS performance? Do you actually have a problem?

hidperf · Oct 21, 2012

hidperf said:
I transferred a 7.7gig file between the two machines about a dozen times each way and ran the smartctl read and write test a couple times. The counter didn't change from the 300 it was before I started moving data.

Looks like transfer speeds are up, just not steady. This is good news right?
View attachment 1343

matram said:
I would be helpful if you tried to clarify what the current status actually is with regard to your CIFS performance? Do you actually have a problem?

Current status was posted.

paleoN · Oct 21, 2012

hidperf said:
Looks like transfer speeds are up, just not steady. This is good news right?

Yes, it's good news.

The values of any and all Tunables/Sysctls you have. Also, your smb.conf file:

Code:

cat /etc/local/smb.conf

hidperf · Oct 21, 2012

paleoN said:
Yes, it's good news.

The values of any and all Tunables/Sysctls you have. Also, your smb.conf file:
Code:
cat /etc/local/smb.conf

I have nothing in the Tunables or Sysctls. I never made any adjustments and I didn't enable "Autotune".

smb.conf is :

Code:

[global]
    encrypt passwords = yes
    dns proxy = no
    strict locking = no
    read raw = yes
    write raw = yes
    oplocks = yes
    max xmit = 65535
    deadtime = 15
    display charset = LOCALE
    max log size = 10
    syslog only = yes
    syslog = yes
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    smb passwd file = /var/etc/private/smbpasswd
    private dir = /var/etc/private
    getwd cache = yes
    guest account = nobody
    map to guest = Bad Password
    obey pam restrictions = Yes
    # NOTE: read smb.conf.
    directory name cache size = 0
    netbios name = freenas
    workgroup = WORKGROUP
    server string = FreeNAS Server
    use sendfile = yes
    store dos attributes = yes
    local master = yes
    time server = yes
    security = user
    create mask = 0666
    directory mask = 0777
    client ntlmv2 auth = yes
    dos charset = CP437
    unix charset = UTF-8
    log level = 1
    aio read size = 4096
    aio write size = 4096


[windows_share]
    path = /mnt/freenas
    printable = no
    veto files = /.snap/.windows/.zfs/
    writeable = yes
    browseable = yes
    inherit owner = no
    inherit permissions = no
    vfs objects =  zfsacl
    guest ok = yes
    inherit acls = Yes
    map archive = No
    map readonly = no
    nfs4:mode = special
    nfs4:acedup = merge
    nfs4:chown = yes

paleoN · Oct 21, 2012

Time to set some, the comments are optional. See the documentation for Sysctls & Tunables and in case something goes horribly wrong.

Set the following Tuneable:

Code:

vfs.zfs.txg.timeout 	5 	5 as of ZFS v28

Set the following Sysctls:

Code:

kern.ipc.maxsockbuf		16777216	16MB
net.inet.tcp.sendbuf_max	16777216	16MB
net.inet.tcp.recvbuf_max	16777216	16MB

net.inet.tcp.sendspace		65536 	doubles memory *per socket*	
net.inet.tcp.recvspace		131072 	doubles memory *per socket*

The first three Autotune will set to 2MB, so at the least set them to 2MB.

For CIFS disable sendfile and bump AIO read/write to 16384. Under Auxiliary parameters add:

Code:

socket options = TCP_NODELAY
min receivefile size = 16384
write cache size = 131072

In my testing the SO_RCVBUF & SO_SNDBUF for socket options had no apparent affect. It either doesn't do anything on FreeBSD or the network Sysctls somehow take care of this or perhaps AIO?

Hopefully this post will stick around.

hidperf · Oct 22, 2012

paleoN said:
Time to set some, the comments are optional. See the documentation for Sysctls & Tunables and in case something goes horribly wrong.

Set the following Tuneable:
Code:
vfs.zfs.txg.timeout 5 5 as of ZFS v28

Set the following Sysctls:
Code:
kern.ipc.maxsockbuf 16777216 16MB net.inet.tcp.sendbuf_max 16777216 16MB net.inet.tcp.recvbuf_max 16777216 16MB net.inet.tcp.sendspace 65536 doubles memory *per socket* net.inet.tcp.recvspace 131072 doubles memory *per socket*
The first three Autotune will set to 2MB, so at the least set them to 2MB.

For CIFS disable sendfile and bump AIO read/write to 16384. Under Auxiliary parameters add:
Code:
socket options = TCP_NODELAY min receivefile size = 16384 write cache size = 131072
In my testing the SO_RCVBUF & SO_SNDBUF for socket options had no apparent affect. It either doesn't do anything on FreeBSD or the network Sysctls somehow take care of this or perhaps AIO?

Hopefully this post will stick around.

I did some research on the changes recommended so I had some idea what we we are doing. Thanks for the help.

Here's the first write I did after the changes and a reboot. Looks much smoother and faster.

And here's the read.

paleoN · Oct 23, 2012

hidperf said:
I did some research on the changes recommended so I had some idea what we we are doing.

OK. What change(s) smoothed out your writes? What does 196 Reallocated_Event_Count usually mean? The research is a good thing by the way.

hidperf said:
Here's the first write I did after the changes and a reboot. Looks much smoother and faster.

You can likely tune this a bit further, but it's not worth it with 8.3 nearly ready for release. With all the ZFS changes and improvements you would need to tune it all over again.

matram · Oct 23, 2012

More benchmarking

Hi Everyone,

A bit more input from my side. I did try out the following (changed) settings in smb.conf, all without any major change in performance:
SO_SNDBUF=8192 SO_RCVBUF=8192
IPTOS_LOWDELAY
TCP_NODELAY
read_raw=no write_raw=no

Trying out varying block size in dd command agains SMB disk, file size about 1.3GB, all without any major change
bs=512, count=2560k
bs=2k, count=640k
bs=8k, count=160k
bs=32k, count=40k

Not what I expected. Measuring block size effect on raw performance (dd on freenas server direct to/from disk, 1.3GB file, read performance shown)
8k - 64k about 1.4 GB/s
4k 1.1 GB/s
2k 850 MB/s
1k 570 MB/s
512 349 MB/s
The raid array has a 256GB SSD cache! Highest write performance is about 260 MB/s.

The only clear effect I could find was the use of SMB client.

Debian client on same ESXi server as FreeNAS (virtual network only) using dd from mounted SMB disk: 27 MB/s read performance
Same debian cient using smbclient get from FreeNAS: 80 MB/s
Win XP client on same ESXi server as FreeNAS using Intel NAS Performance Toolkit: 27 MB/s
Debian client on other (old) ESXI server (physical 1 Gb network) using dd: 18 MB/s
OSX client on physical 1 Gb network using dd: 37 MB/s

Still trying to understand why

/Mats

hidperf · Oct 23, 2012

paleoN said:
OK. What change(s) smoothed out your writes? What does 196 Reallocated_Event_Count usually mean? The research is a good thing by the way.

From what I've gathered:
vfs.zfs.txg.timeout decrease from 30 is supposed to shorten the stalls during I/O on ZFS.
kern.ipc.maxsockbuf, net.inet.tcp.sendspace and net.inet.tcp.recvspace increase the TCP window size, although I don't understand why the last two were set differently.
net.inet.tcp.sendbuf_max and net.inet.tcp.recvbuf_max seem to limit the buffer sizes but increase them well above stock settings.
socket options = TCP_NODELAY forces the socket to send packets in the buffer.

I'm going with those, but I really need more studying. CompSci is not my strong point.

One thing I read several times is net.inet.tcp.rfc1323 should be enabled to take advantage of TCP window scaling.

Reallocated_Event_Count is the number of times the drive had to reallocate sectors due to I/O errors.

How'd I do?

Important Announcement for the TrueNAS Community.

I've searched. I've tried. I need help. CIF share is SLOW and something will stop

Dabbler

Dabbler

Dabbler

Dabbler

Wizard

Dabbler

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Dabbler

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Dabbler

Similar threads