45x 4TB FreeNAS Testing Results

Status
Not open for further replies.
S

scottsee

Cadet
Joined
Aug 5, 2016
Messages
8
Hi There

I'm hoping someone could chime in on performance/observations of a FreeNAS box I performed load testing on this morning. the code outputs are a bit long and are not time synced. So I'm going to break this into multiple posts.

I'm not sure how to interpret these results.

arcstat show a 15-30% miss rate
gstat bounce around 10% ish
zpool iostat show peaks and valleys.




Processor Intel(R) Core(TM) i3-3240 CPU @ 3.40GHz, 3400 Mhz, 2 Core(s), 4 Logical Processor(s)
Installed Physical Memory (RAM) 32.01 GB
Drives 45 x 4TB WD RED SATA drives

Arcstat.py
Code:
 
   time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
08:36:30   236    64     27    64   27     0    0    52   24    18G   18G 
08:36:32   260    55     21    55   21     0    0    46   20    18G   18G 
08:36:33   243    53     21    53   21     0    0    43   19    18G   18G 
08:36:34   310    68     21    68   21     0    0    48   18    18G   18G 
08:36:35   235    63     26    63   26     0    0    53   25    18G   18G 
08:36:36   222    61     27    61   27     0    0    41   22    18G   18G 
08:36:38   251    66     26    66   26     0    0    54   24    18G   18G 
08:36:39   208    51     24    51   24     0    0    41   22    18G   18G 
08:36:40   280    57     20    57   20     0    0    47   18    18G   18G 
08:36:41   249    71     28    71   28     0    0    51   25    18G   18G 
08:36:42   259    64     24    64   24     0    0    54   22    18G   18G 
08:36:43   239    63     26    63   26     0    0    45   20    18G   18G 
08:36:45   271    67     24    67   24     0    0    45   20    18G   18G 
08:36:46   280    63     22    63   22     0    0    44   18    18G   18G 
08:36:47   939    64      6    64    6     0    0    49   26    18G   18G 
08:36:48  1.2K   528     42    81   10   447  100    64   16    18G   18G 
08:36:49  1.9K   725     38    39    3   686  100    45    8    18G   18G 
08:36:51  1.5K   519     35    57    5   462  100    37    8    18G   18G 
08:36:52  3.9K   195      5   195    5     0    0   195    5    18G   18G 
08:36:53   243    33     13    33   13     0    0    33   15    18G   18G 
08:36:54   303    35     11    35   11     0    0    35   13    18G   18G 
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
08:36:55   266    33     12    33   12     0    0    33   13    18G   18G 
08:36:57   407    38      9    38    9     0    0    32   15    18G   18G 
08:36:58   301    33     10    33   10     0    0    32   11    18G   18G 
08:36:59   241    33     13    33   13     0    0    33   15    18G   18G 
08:37:00   248    28     11    28   11     0    0    28   12    18G   18G 
08:37:01   258    33     12    33   12     0    0    33   14    18G   18G 
08:37:03   268    35     13    35   13     0    0    35   15    18G   18G 
08:37:04   281    39     13    39   13     0    0    39   15    18G   18G 
08:37:05   352    63     17    63   17     0    0    49   20    18G   18G 
08:37:06   266    43     16    43   16     0    0    43   17    18G   18G 
08:37:07   257    26     10    26   10     0    0    26   12    18G   18G 
08:37:09   219    33     15    33   15     0    0    33   17    18G   18G 
08:37:10   233    27     11    27   11     0    0    27   12    18G   18G 
08:37:11   269    32     11    32   11     0    0    32   13    18G   18G 
08:37:12   275    41     14    41   14     0    0    41   17    18G   18G 
08:37:13   238    32     13    32   13     0    0    32   14    18G   18G 
08:37:15   219    27     12    27   12     0    0    27   14    18G   18G 
08:37:16   250    31     12    31   12     0    0    31   14    18G   18G 
08:37:17   260    27     10    27   10     0    0    27   11    18G   18G 
08:37:18   257    33     12    33   12     0    0    33   15    18G   18G 
08:37:19   246    26     10    26   10     0    0    26   11    18G   18G 
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
08:37:20   269    40     14    40   14     0    0    40   16    18G   18G 
08:37:22   198    25     12    25   12     0    0    25   14    18G   18G 
08:37:23   310    37     11    37   11     0    0    34   13    18G   18G 
08:37:24   252    33     13    33   13     0    0    31   13    18G   18G 
08:37:25   285    40     14    40   14     0    0    40   16    18G   18G 
08:37:26   413    38      9    38    9     0    0    38   16    18G   18G 
08:37:28   294    40     13    40   13     0    0    40   16    18G   18G 
08:37:29   239    30     12    30   12     0    0    30   14    18G   18G 
08:37:30   271    27      9    27    9     0    0    27   11    18G   18G 
08:37:31   221    32     14    32   14     0    0    32   16    18G   18G 
08:37:32   214    25     11    25   11     0    0    25   13    18G   18G 
08:37:34   262    31     11    31   11     0    0    31   14    18G   18G 
08:37:35   273    33     12    33   12     0    0    33   13    18G   18G 
08:37:36   257    27     10    27   10     0    0    27   12    18G   18G 
08:37:37   254    35     13    35   13     0    0    35   15    18G   18G 
08:37:38   306    36     11    36   11     0    0    36   14    18G   18G 
08:37:39   258    40     15    40   15     0    0    40   17    18G   18G 
08:37:41   241    30     12    30   12     0    0    30   13    18G   18G 
08:37:42   241    30     12    30   12     0    0    30   13    18G   18G 
08:37:43   277    33     11    33   11     0    0    33   13    18G   18G 
08:37:44   293    48     16    48   16     0    0    46   18    18G   18G 
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
08:37:45   257    27     10    27   10     0    0    26   12    18G   18G 
08:37:46   240    39     16    39   16     0    0    37   17    18G   18G 
08:37:48   270    30     11    30   11     0    0    29   12    18G   18G 
08:37:49   261    27     10    27   10     0    0    26   12    18G   18G 
08:37:50   255    40     15    40   15     0    0    39   17    18G   18G 
08:37:51   226    30     13    30   13     0    0    28   14    18G   18G 
08:37:52   963    43      4    43    4     0    0    31   12    18G   18G 
08:37:54   248    47     18    47   18     0    0    27   13    18G   18G 
08:37:55   262    45     17    45   17     0    0    35   14    18G   18G 
08:37:56   279    47     16    47   16     0    0    32   14    18G   18G 
08:37:57   442    77     17    77   17     0    0    60   22    18G   18G 
08:37:58   235    59     25    59   25     0    0    49   23    18G   18G 
08:37:59   237    69     29    69   29     0    0    47   24    18G   18G 
08:38:01   243    59     24    59   24     0    0    49   22    18G   18G 
08:38:02   271    58     21    58   21     0    0    48   19    18G   18G 
08:38:03   242    66     27    66   27     0    0    56   25    18G   18G 
08:38:04   267    64     23    64   23     0    0    45   20    18G   18G 
08:38:05   248    65     26    65   26     0    0    55   24    18G   18G 
08:38:07   242    64     26    64   26     0    0    44   22    18G   18G 
08:38:08   349   103     29   103   29     0    0    78   31    18G   18G 
08:38:09   300    71     23    71   23     0    0    51   20    18G   18G 
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c 
08:38:10   245    55     22    55   22     0    0    45   20    18G   18G 
08:38:11   212    56     26    56   26     0    0    46   24    18G   18G 
08:38:12   245    52     21    52   21     0    0    42   18    18G   18G 
08:38:14   236    53     22    53   22     0    0    43   20    18G   18G 
08:38:15   276    70     25    70   25     0    0    49   21    18G   18G 
08:38:16   266    71     26    71   26     0    0    51   22    18G   18G 
08:38:17   242    68     28    68   28     0    0    57   27    18G   18G 
08:38:18   222    51     22    51   22     0    0    41   20    18G   18G 
08:38:20   183    45     24    45   24     0    0    34   24    18G   18G 
08:38:21   279    71     25    71   25     0    0    60   24    18G   18G 
08:38:22   233    54     23    54   23     0    0    44   20    18G   18G 
08:38:23   321    90     28    90   28     0    0    75   26    18G   18G 
08:38:24   251    60     23    60   23     0    0    48   21    18G   18G 
08:38:26   266    64     24    64   24     0    0    45   20    18G   18G 
08:38:27   437    68     15    68   15     0    0    51   21    18G   18G 
08:38:28   301    74     24    74   24     0    0    63   23    18G   18G 
08:38:29   262    64     24    64   24     0    0    53   22    18G   18G


zpool iostat
Code:
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
Production  36.9T   126T     16  4.72K  67.7K   405M
Production  36.9T   126T     13  4.30K  55.7K   354M
Production  36.9T   126T     10  4.81K  43.8K   397M
Production  36.9T   126T      8  4.55K  35.8K   386M
Production  36.9T   126T     16  4.36K  67.7K   340M
Production  36.9T   126T     10  4.55K  43.3K   379M
Production  36.9T   126T     27  4.87K   110K   415M
Production  36.9T   126T     12  4.31K  51.8K   371M
Production  36.9T   126T      9  4.31K  39.5K   346M
Production  36.9T   126T      7  4.05K  31.9K   387M
Production  36.9T   126T     16  4.80K  67.5K   397M
Production  36.9T   126T     12  4.54K  51.2K   375M
Production  36.9T   126T     14  4.95K  59.7K   401M
Production  36.9T   126T     10  4.30K  43.9K   367M
Production  36.9T   126T     10  4.38K  43.3K   360M
Production  36.9T   126T     12  4.73K  50.4K   394M
Production  36.9T   126T     10  4.54K  43.2K   368M
Production  36.9T   126T     10  4.75K  43.6K   368M
Production  36.9T   126T      8  4.47K  35.9K   387M
Production  37.0T   126T     10  4.37K  43.9K   389M
Production  37.0T   126T     14  4.58K  59.8K   397M
Production  37.0T   126T     12  4.59K  51.9K   387M
Production  37.0T   126T     10  4.30K  43.9K   356M
Production  37.0T   126T     10  4.34K  43.6K   357M
Production  37.0T   126T     12  4.67K  51.5K   389M
Production  37.0T   126T     10  4.80K  43.8K   389M
Production  37.0T   126T     10  4.71K  43.7K   388M
Production  37.0T   126T      6  4.40K  27.9K   372M
Production  37.0T   126T     12  4.45K  51.6K   358M
Production  37.0T   126T      8  4.55K  35.5K   387M
Production  37.0T   126T     15  4.00K  71.9K   365M
Production  37.0T   126T    404  4.49K  48.2M   380M
Production  37.0T   126T    561  3.95K  69.5M   365M
Production  37.0T   126T    600  4.37K  74.8M   361M
Production  37.0T   126T     82  4.36K  6.06M   348M
Production  37.0T   126T     11  4.76K  47.9K   398M
Production  37.0T   126T      0  4.53K      0   387M
Production  37.0T   126T      0  4.81K      0   374M
Production  37.0T   126T      0  4.76K      0   391M
Production  37.0T   126T      0  4.62K      0   353M
Production  37.0T   126T      0  4.74K  3.99K   377M
Production  37.0T   126T      0  4.77K      0   416M
Production  37.0T   126T      0  4.30K      0   354M
Production  37.0T   126T      0  4.72K      0   387M
Production  37.0T   126T      0  4.61K      0   363M
Production  37.0T   126T      0  4.69K      0   383M
Production  37.0T   126T      0  4.60K      0   382M
Production  37.0T   126T      0  4.63K      0   388M
Production  37.0T   126T     13  4.37K   192K   376M
Production  37.0T   126T      0  4.90K      0   397M
Production  37.0T   126T      0  4.50K      0   377M
Production  37.0T   126T      0  4.22K      0   352M
Production  37.0T   126T      0  4.84K      0   405M
Production  37.0T   126T      0  4.66K      0   397M


gstat
Code:
T: 1.002s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    1    309      0      0    0.0    299  11824    1.0   20.7| da0
    1    318      0      0    0.0    308  11888    1.0   21.5| da1
    1    314      0      0    0.0    304  11860    1.0   21.2| da2
    1    359      0      0    0.0    349  12008    0.9   20.7| da3
    1    367      0      0    0.0    357  12020    0.9   21.5| da4
    1    370      0      0    0.0    360  12012    0.9   21.9| da5
    1    363      0      0    0.0    353  11988    0.9   21.6| da6
    1    360      0      0    0.0    350  11976    0.9   21.3| da7
    1    385      2      8    0.1    375  12403    1.0   25.9| da8
    1    388      0      0    0.0    380  12479    1.0   25.6| da9
    2    391      0      0    0.0    383  12479    1.0   25.7| da10
    1    390      0      0    0.0    382  12463    1.0   25.5| da11
    1    384      0      0    0.0    376  12407    0.9   23.3| da12
    1    411      0      0    0.0    404  13361    0.9   25.7| da13
    1    411      0      0    0.0    404  13329    0.9   25.7| da14
    1    408      0      0    0.0    401  13297    0.9   25.5| da15
    2    404      0      0    0.0    397  13269    0.8   23.2| da16
    1    401      0      0    0.0    394  13357    0.8   24.5| da17
    2    357      1      4    0.1    350  13409    1.0   25.3| da18
    2    354      0      0    0.0    348  13517    1.0   24.2| da19
    2    349      0      0    0.0    343  13417    0.9   22.6| da20
    2    347      0      0    0.0    341  13457    0.9   22.6| da21
    2    354      0      0    0.0    348  13537    1.0   22.7| da22
    2    307      0      0    0.0    301  13377    1.1   21.4| da23
    2    308      2      8    0.1    300  13481    1.2   23.8| da24
    2    309      2      8    0.7    301  13393    1.2   23.5| da25
    1    313      0      0    0.0    307  13517    1.1   23.7| da26
    2    305      0      0    0.0    299  13389    1.2   23.6| da27
    2    286      0      0    0.0    279  13317    1.3   26.0| da28
    2    288      1      4    0.1    280  13221    1.4   26.1| da29
    2    288      0      0    0.0    281  13297    1.4   26.5| da30
    1    284      0      0    0.0    277  13066    1.4   25.7| da31
    2    285      0      0    0.0    278  13265    1.2   23.0| da32
    1    300      0      0    0.0    292  13213    1.2   22.5| da33
    2    298      0      0    0.0    290  13150    1.2   23.3| da34
    2    297      0      0    0.0    289  13166    1.2   23.5| da35
    1    304      0      0    0.0    296  13229    1.1   22.1| da36
    1    303      0      0    0.0    295  13229    1.1   21.7| da37
    2    324      0      0    0.0    314  12571    1.1   22.5| da38
    2    332      0      0    0.0    322  12555    1.0   22.1| da39
    2    333      0      0    0.0    323  12567    1.2   25.1| da40
    2    327      1      4    1.4    316  12563    1.2   25.2| da41
    2    323      0      0    0.0    313  12527    1.1   22.7| da42
    1    311      0      0    0.0    301  11864    1.0   21.5| da43
    1    309      1      4    1.6    298  11848    0.7   14.7| da44
    0      0      0      0    0.0      0      0    0.0    0.0| da45
    0      6      6     13    4.1      0      0    0.0    2.5| ada0
    0      9      9     44    0.1      0      0    0.0    0.1| ada1
 
Last edited:
S

scottsee

Cadet
Joined
Aug 5, 2016
Messages
8
arc_summary.py
Code:
System Memory:

        1.02%   323.59  MiB Active,     23.25%  7.23    GiB Inact
        74.85%  23.29   GiB Wired,      0.18%   57.93   MiB Cache
        0.70%   223.12  MiB Free,       0.00%   0       Bytes Gap

        Real Installed:                         32.00   GiB
        Real Available:                 99.83%  31.94   GiB
        Real Managed:                   97.39%  31.11   GiB

        Logical Total:                          32.00   GiB
        Logical Used:                   76.54%  24.49   GiB
        Logical Free:                   23.46%  7.51    GiB

Kernel Memory:                                  557.70  MiB
        Data:                           95.11%  530.44  MiB
        Text:                           4.89%   27.26   MiB

Kernel Memory Map:                              31.11   GiB
        Size:                           61.87%  19.25   GiB
        Free:                           38.13%  11.86   GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Storage pool Version:                   5000
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                64.61m
        Mutex Misses:                           9.28k
        Evict Skips:                            9.28k

ARC Size:                               60.38%  18.18   GiB
        Target Size: (Adaptive)         60.38%  18.18   GiB
        Min Size (Hard Limit):          12.50%  3.76    GiB
        Max Size (High Water):          8:1     30.11   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       95.09%  17.29   GiB
        Frequently Used Cache Size:     4.91%   914.20  MiB

ARC Hash Breakdown:
        Elements Max:                           6.11m
        Elements Current:               10.96%  669.83k
        Collisions:                             29.71m
        Chain Max:                              10
        Chains:                                 48.25k
                                                                Page:  2
------------------------------------------------------------------------

ARC Total accesses:                                     225.89m
        Cache Hit Ratio:                77.41%  174.85m
        Cache Miss Ratio:               22.59%  51.03m
        Actual Hit Ratio:               77.18%  174.34m

        Data Demand Efficiency:         90.42%  102.43m
        Data Prefetch Efficiency:       6.27%   75.87k

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           19.64%  34.34m
          Most Frequently Used:         80.06%  139.99m
          Most Recently Used Ghost:     0.70%   1.22m
          Most Frequently Used Ghost:   1.04%   1.82m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  52.97%  92.62m
          Prefetch Data:                0.00%   4.75k
          Demand Metadata:              46.51%  81.32m
          Prefetch Metadata:            0.52%   913.58k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  19.22%  9.81m
          Prefetch Data:                0.14%   71.12k
          Demand Metadata:              69.85%  35.65m
          Prefetch Metadata:            10.79%  5.50m
                                                                Page:  3
------------------------------------------------------------------------

                                                                Page:  4
------------------------------------------------------------------------

DMU Prefetch Efficiency:                        490.81m
        Hit Ratio:                      11.26%  55.24m
        Miss Ratio:                     88.74%  435.56m

                                                                Page:  5
------------------------------------------------------------------------

                                                                Page:  6
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           2380
        vm.kmem_size                            33404542976
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        1319413950874
        vfs.zfs.vol.unmap_enabled               1
        vfs.zfs.vol.mode                        2
        vfs.zfs.sync_pass_rewrite               2
        vfs.zfs.sync_pass_dont_compress         5
        vfs.zfs.sync_pass_deferred_free         2
        vfs.zfs.zio.exclude_metadata            0
        vfs.zfs.zio.use_uma                     1
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     5000
        vfs.zfs.version.acl                     1
        vfs.zfs.version.ioctl                   6
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
        vfs.zfs.min_auto_ashift                 9
        vfs.zfs.max_auto_ashift                 13
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.trim_max_active            64
        vfs.zfs.vdev.trim_min_active            1
        vfs.zfs.vdev.scrub_max_active           2
        vfs.zfs.vdev.scrub_min_active           1
        vfs.zfs.vdev.async_write_max_active     10
        vfs.zfs.vdev.async_write_min_active     1
        vfs.zfs.vdev.async_read_max_active      3
        vfs.zfs.vdev.async_read_min_active      1
        vfs.zfs.vdev.sync_write_max_active      10
        vfs.zfs.vdev.sync_write_min_active      10
        vfs.zfs.vdev.sync_read_max_active       10
        vfs.zfs.vdev.sync_read_min_active       10
        vfs.zfs.vdev.max_active                 1000
        vfs.zfs.vdev.async_write_active_max_dirty_percent60
        vfs.zfs.vdev.async_write_active_min_dirty_percent30
        vfs.zfs.vdev.mirror.non_rotating_seek_inc1
        vfs.zfs.vdev.mirror.non_rotating_inc    0
        vfs.zfs.vdev.mirror.rotating_seek_offset1048576
        vfs.zfs.vdev.mirror.rotating_seek_inc   5
        vfs.zfs.vdev.mirror.rotating_inc        0
        vfs.zfs.vdev.trim_on_init               1
        vfs.zfs.vdev.larger_ashift_minimal      0
        vfs.zfs.vdev.bio_delete_disable         0
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.metaslabs_per_vdev         200
        vfs.zfs.vdev.trim_max_pending           10000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.trim.enabled                    1
        vfs.zfs.trim.max_interval               1
        vfs.zfs.trim.timeout                    30
        vfs.zfs.trim.txg_delay                  32
        vfs.zfs.space_map_blksz                 4096
        vfs.zfs.spa_slop_shift                  5
        vfs.zfs.spa_asize_inflation             24
        vfs.zfs.deadman_enabled                 1
        vfs.zfs.deadman_checktime_ms            5000
        vfs.zfs.deadman_synctime_ms             1000000
        vfs.zfs.debug_flags                     0
        vfs.zfs.recover                         0
        vfs.zfs.spa_load_verify_data            1
        vfs.zfs.spa_load_verify_metadata        1
        vfs.zfs.spa_load_verify_maxinflight     10000
        vfs.zfs.ccw_retry_interval              300
        vfs.zfs.check_hostid                    1
        vfs.zfs.mg_fragmentation_threshold      85
        vfs.zfs.mg_noalloc_threshold            0
        vfs.zfs.condense_pct                    200
        vfs.zfs.metaslab.bias_enabled           1
        vfs.zfs.metaslab.lba_weighting_enabled  1
        vfs.zfs.metaslab.fragmentation_factor_enabled1
        vfs.zfs.metaslab.preload_enabled        1
        vfs.zfs.metaslab.preload_limit          3
        vfs.zfs.metaslab.unload_delay           8
        vfs.zfs.metaslab.load_pct               50
        vfs.zfs.metaslab.min_alloc_size         33554432
        vfs.zfs.metaslab.df_free_pct            4
        vfs.zfs.metaslab.df_alloc_threshold     131072
        vfs.zfs.metaslab.debug_unload           0
        vfs.zfs.metaslab.debug_load             0
        vfs.zfs.metaslab.fragmentation_threshold70
        vfs.zfs.metaslab.gang_bang              16777217
        vfs.zfs.free_bpobj_enabled              1
        vfs.zfs.free_max_blocks                 18446744073709551615
        vfs.zfs.no_scrub_prefetch               0
        vfs.zfs.no_scrub_io                     0
        vfs.zfs.resilver_min_time_ms            3000
        vfs.zfs.free_min_time_ms                1000
        vfs.zfs.scan_min_time_ms                1000
        vfs.zfs.scan_idle                       50
        vfs.zfs.scrub_delay                     4
        vfs.zfs.resilver_delay                  2
        vfs.zfs.top_maxinflight                 32
        vfs.zfs.delay_scale                     500000
        vfs.zfs.delay_min_dirty_percent         60
        vfs.zfs.dirty_data_sync                 67108864
        vfs.zfs.dirty_data_max_percent          10
        vfs.zfs.dirty_data_max_max              4294967296
        vfs.zfs.dirty_data_max                  3430032588
        vfs.zfs.max_recordsize                  1048576
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.max_distance             8388608
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.nopwrite_enabled                1
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            4836541440
        vfs.zfs.mfu_ghost_metadata_lsize        1283012608
        vfs.zfs.mfu_ghost_size                  6119554048
        vfs.zfs.mfu_data_lsize                  2311680
        vfs.zfs.mfu_metadata_lsize              886784
        vfs.zfs.mfu_size                        167542272
        vfs.zfs.mru_ghost_data_lsize            1310720
        vfs.zfs.mru_ghost_metadata_lsize        769158144
        vfs.zfs.mru_ghost_size                  770468864
        vfs.zfs.mru_data_lsize                  17503227904
        vfs.zfs.mru_metadata_lsize              1007967744
        vfs.zfs.mru_size                        18749691392
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       587776
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  8082700288
        vfs.zfs.arc_free_target                 56584
        vfs.zfs.arc_shrink_shift                7
        vfs.zfs.arc_average_blocksize           8192
        vfs.zfs.arc_min                         4041350144
        vfs.zfs.arc_max                         32330801152
                                                                Page:  7
------------------------------------------------------------------------
 
Mlovelace

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
45 4TB drives on a system with only 32GB of RAM?!?!? :eek: Your system is under-spec'd for that quantity of storage. What is the zpool layout?
 
S

scottsee

Cadet
Joined
Aug 5, 2016
Messages
8
Noticed this odd behavior on disk. 98% of the log shows the solid performance. don't know if this is just a hiccup.

Code:
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
Production                              39.6T   124T      6    815  34.4K  95.8M
  raidz2                                4.40T  13.7T      0    213      0  24.9M
    gptid/cceafaf7-3411-11e6-8364-a0369f82da38      -      -      0    232      0  9.80M
    gptid/ce18e36f-3411-11e6-8364-a0369f82da38      -      -      0    291      0  12.3M
    gptid/cf494223-3411-11e6-8364-a0369f82da38      -      -      0    285      0  12.1M
    gptid/d0758376-3411-11e6-8364-a0369f82da38      -      -      0    245      0  8.93M
    gptid/d1a96db2-3411-11e6-8364-a0369f82da38      -      -      0    264      0  10.7M
  raidz2                                4.40T  13.7T      0    354      0  41.3M
    gptid/d2d2523b-3411-11e6-8364-a0369f82da38      -      -      0    309      0  10.6M
    gptid/3dc2b539-55cd-11e6-a3de-a0369f82da38      -      -      0    356      0  13.2M
    gptid/d5323ca2-3411-11e6-8364-a0369f82da38      -      -      0    291      0  10.3M
    gptid/d65e21cf-3411-11e6-8364-a0369f82da38      -      -      0    186      0  7.80M
    gptid/d7943993-3411-11e6-8364-a0369f82da38      -      -      0    309      0  10.8M
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/d8c35872-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/d9f20c3f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/db270b03-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dc5adc18-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dd812c8c-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/deb83c65-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dfdec8a6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e10c7d1e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e2443126-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e378776d-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/e4aecd4e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e5e10597-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e70eb118-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e83f75ba-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e970d70a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0  11.5K      0
    gptid/eaa7b674-3411-11e6-8364-a0369f82da38      -      -      0      0  3.82K      0
    gptid/ebdacfbf-3411-11e6-8364-a0369f82da38      -      -      0      0  3.82K      0
    gptid/ed0c5f7a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/ee3d3f75-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/ef6ecd4f-3411-11e6-8364-a0369f82da38      -      -      0      0  3.82K      0
  raidz2                                4.40T  13.7T      2      0  11.5K      0
    gptid/f0accf0b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f1e496a5-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f31c7af6-3411-11e6-8364-a0369f82da38      -      -      2      0  11.5K      0
    gptid/f444c43c-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f5774397-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0  3.82K      0
    gptid/f6c96a7b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f7f540d6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f92a550f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fa654cda-3411-11e6-8364-a0369f82da38      -      -      0      0  3.82K      0
    gptid/fb94ad0e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      1    247  7.64K  29.6M
    gptid/fcf0a3b4-3411-11e6-8364-a0369f82da38      -      -      0    225  3.82K  9.02M
    gptid/fe2678c4-3411-11e6-8364-a0369f82da38      -      -      0    235      0  9.82M
    gptid/ff5f5215-3411-11e6-8364-a0369f82da38      -      -      0    238      0  9.38M
    gptid/008dcc92-3412-11e6-8364-a0369f82da38      -      -      0    203      0  7.82M
    gptid/01c4fd44-3412-11e6-8364-a0369f82da38      -      -      0    252  3.82K  10.2M
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                            1.83G  5.36G     18      0   422K      0
  mirror                                1.83G  5.36G     18      0   422K      0
    gptid/778d2de7-101d-11e5-9e35-002590a63c37      -      -      5      0  12.4K      0
    ada1p2                                  -      -     12      0   410K      0
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
Production                              39.6T   124T      6    615  35.7K  60.8M
  raidz2                                4.40T  13.7T      0    322      0  36.0M
    gptid/cceafaf7-3411-11e6-8364-a0369f82da38      -      -      0    175      0  6.37M
    gptid/ce18e36f-3411-11e6-8364-a0369f82da38      -      -      0     38      0   706K
    gptid/cf494223-3411-11e6-8364-a0369f82da38      -      -      0     56      0  1.93M
    gptid/d0758376-3411-11e6-8364-a0369f82da38      -      -      0    222      0  9.39M
    gptid/d1a96db2-3411-11e6-8364-a0369f82da38      -      -      0     16      0   337K
  raidz2                                4.40T  13.7T      3    204  15.9K  16.0M
    gptid/d2d2523b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/3dc2b539-55cd-11e6-a3de-a0369f82da38      -      -      0      0  3.96K      0
    gptid/d5323ca2-3411-11e6-8364-a0369f82da38      -      -      2     48  11.9K  1.15M
    gptid/d65e21cf-3411-11e6-8364-a0369f82da38      -      -      0    183      0  5.12M
    gptid/d7943993-3411-11e6-8364-a0369f82da38      -      -      0      1      0  87.2K
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/d8c35872-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/d9f20c3f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/db270b03-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dc5adc18-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dd812c8c-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/deb83c65-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dfdec8a6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e10c7d1e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e2443126-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e378776d-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/e4aecd4e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e5e10597-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e70eb118-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e83f75ba-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e970d70a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0  11.9K      0
    gptid/eaa7b674-3411-11e6-8364-a0369f82da38      -      -      0      0  3.97K      0
    gptid/ebdacfbf-3411-11e6-8364-a0369f82da38      -      -      0      0  3.97K      0
    gptid/ed0c5f7a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/ee3d3f75-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/ef6ecd4f-3411-11e6-8364-a0369f82da38      -      -      0      0  3.97K      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/f0accf0b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f1e496a5-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f31c7af6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f444c43c-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f5774397-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0  3.97K      0
    gptid/f6c96a7b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f7f540d6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f92a550f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fa654cda-3411-11e6-8364-a0369f82da38      -      -      0      0  3.97K      0
    gptid/fb94ad0e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0     89  3.97K  8.91M
    gptid/fcf0a3b4-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fe2678c4-3411-11e6-8364-a0369f82da38      -      -      0     36      0   929K
    gptid/ff5f5215-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/008dcc92-3412-11e6-8364-a0369f82da38      -      -      0     57      0  2.43M
    gptid/01c4fd44-3412-11e6-8364-a0369f82da38      -      -      0      0  3.97K      0
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                            1.83G  5.36G     17      0   438K      0
  mirror                                1.83G  5.36G     17      0   438K      0
    gptid/778d2de7-101d-11e5-9e35-002590a63c37      -      -      4      0  11.9K      0
    ada1p2                                  -      -     12      0   426K      0
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
Production                              39.6T   124T     17  5.32K  79.5K   583M
  raidz2                                4.40T  13.7T      0     79  3.79K  8.92M
    gptid/cceafaf7-3411-11e6-8364-a0369f82da38      -      -      0    207      0  10.6M
    gptid/ce18e36f-3411-11e6-8364-a0369f82da38      -      -      0    296      0  16.5M
    gptid/cf494223-3411-11e6-8364-a0369f82da38      -      -      0    253      0  11.0M
    gptid/d0758376-3411-11e6-8364-a0369f82da38      -      -      0    130      0  6.24M
    gptid/d1a96db2-3411-11e6-8364-a0369f82da38      -      -      0    311  3.79K  13.3M
  raidz2                                4.40T  13.7T      0    225  3.79K  20.2M
    gptid/d2d2523b-3411-11e6-8364-a0369f82da38      -      -      0    368      0  13.0M
    gptid/3dc2b539-55cd-11e6-a3de-a0369f82da38      -      -      0    376      0  13.5M
    gptid/d5323ca2-3411-11e6-8364-a0369f82da38      -      -      0    327  3.79K  12.6M
    gptid/d65e21cf-3411-11e6-8364-a0369f82da38      -      -      0    203      0  7.11M
    gptid/d7943993-3411-11e6-8364-a0369f82da38      -      -      0    354      0  12.9M
  raidz2                                4.40T  13.7T      0    980      0   108M
    gptid/d8c35872-3411-11e6-8364-a0369f82da38      -      -      0    568      0  38.8M
    gptid/d9f20c3f-3411-11e6-8364-a0369f82da38      -      -      0    496      0  37.6M
    gptid/db270b03-3411-11e6-8364-a0369f82da38      -      -      0    519      0  39.0M
    gptid/dc5adc18-3411-11e6-8364-a0369f82da38      -      -      0    482      0  37.2M
    gptid/dd812c8c-3411-11e6-8364-a0369f82da38      -      -      0    507      0  38.0M
  raidz2                                4.40T  13.7T      0  1.06K      0   120M
    gptid/deb83c65-3411-11e6-8364-a0369f82da38      -      -      0    402      0  40.7M
    gptid/dfdec8a6-3411-11e6-8364-a0369f82da38      -      -      0    400      0  40.7M
    gptid/e10c7d1e-3411-11e6-8364-a0369f82da38      -      -      0    411      0  40.7M
    gptid/e2443126-3411-11e6-8364-a0369f82da38      -      -      0    410      0  40.7M
    gptid/e378776d-3411-11e6-8364-a0369f82da38      -      -      0    402      0  40.7M
  raidz2                                4.40T  13.7T      5  1.11K  22.7K   121M
    gptid/e4aecd4e-3411-11e6-8364-a0369f82da38      -      -      1    415  7.58K  40.9M
    gptid/e5e10597-3411-11e6-8364-a0369f82da38      -      -      0    418      0  40.9M
    gptid/e70eb118-3411-11e6-8364-a0369f82da38      -      -      1    410  7.58K  41.0M
    gptid/e83f75ba-3411-11e6-8364-a0369f82da38      -      -      0    465  3.79K  40.9M
    gptid/e970d70a-3411-11e6-8364-a0369f82da38      -      -      0    415  3.79K  40.9M
  raidz2                                4.40T  13.7T      2    401  18.9K  38.4M
    gptid/eaa7b674-3411-11e6-8364-a0369f82da38      -      -      1    408  7.58K  20.5M
    gptid/ebdacfbf-3411-11e6-8364-a0369f82da38      -      -      0    378  3.79K  16.2M
    gptid/ed0c5f7a-3411-11e6-8364-a0369f82da38      -      -      0    365  3.79K  15.7M
    gptid/ee3d3f75-3411-11e6-8364-a0369f82da38      -      -      0    407      0  20.4M
    gptid/ef6ecd4f-3411-11e6-8364-a0369f82da38      -      -      0    374  3.79K  16.1M
  raidz2                                4.40T  13.7T      0    302  3.79K  28.5M
    gptid/f0accf0b-3411-11e6-8364-a0369f82da38      -      -      0    338      0  16.2M
    gptid/f1e496a5-3411-11e6-8364-a0369f82da38      -      -      0    361      0  17.2M
    gptid/f31c7af6-3411-11e6-8364-a0369f82da38      -      -      0    302  3.79K  13.6M
    gptid/f444c43c-3411-11e6-8364-a0369f82da38      -      -      0    388      0  18.8M
    gptid/f5774397-3411-11e6-8364-a0369f82da38      -      -      0    300      0  13.6M
  raidz2                                4.40T  13.7T      2  1.02K  11.4K   119M
    gptid/f6c96a7b-3411-11e6-8364-a0369f82da38      -      -      1    403  7.58K  40.2M
    gptid/f7f540d6-3411-11e6-8364-a0369f82da38      -      -      0    391      0  40.3M
    gptid/f92a550f-3411-11e6-8364-a0369f82da38      -      -      0    395      0  40.3M
    gptid/fa654cda-3411-11e6-8364-a0369f82da38      -      -      0    390      0  40.3M
    gptid/fb94ad0e-3411-11e6-8364-a0369f82da38      -      -      0    459  3.79K  40.2M
  raidz2                                4.40T  13.7T      3    195  15.2K  19.4M
    gptid/fcf0a3b4-3411-11e6-8364-a0369f82da38      -      -      1    353  7.58K  12.6M
    gptid/fe2678c4-3411-11e6-8364-a0369f82da38      -      -      0    257      0  8.59M
    gptid/ff5f5215-3411-11e6-8364-a0369f82da38      -      -      0    340      0  13.4M
    gptid/008dcc92-3412-11e6-8364-a0369f82da38      -      -      0    326      0  12.2M
    gptid/01c4fd44-3412-11e6-8364-a0369f82da38      -      -      1    285  7.58K  9.77M
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                            1.83G  5.36G     17      0   417K      0
  mirror                                1.83G  5.36G     17      0   417K      0
    gptid/778d2de7-101d-11e5-9e35-002590a63c37      -      -      4      0  11.4K      0
    ada1p2                                  -      -     12      0   406K      0
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
Production                              39.6T   124T     17  1.19K  75.7K   124M
  raidz2                                4.40T  13.7T      2    159  11.4K  14.5M
    gptid/cceafaf7-3411-11e6-8364-a0369f82da38      -      -      0    254  3.78K  13.5M
    gptid/ce18e36f-3411-11e6-8364-a0369f82da38      -      -      0    301      0  19.7M
    gptid/cf494223-3411-11e6-8364-a0369f82da38      -      -      0    278      0  19.6M
    gptid/d0758376-3411-11e6-8364-a0369f82da38      -      -      1    214  7.57K  14.2M
    gptid/d1a96db2-3411-11e6-8364-a0369f82da38      -      -      0    295      0  23.7M
  raidz2                                4.40T  13.7T      0    213  3.78K  26.3M
    gptid/d2d2523b-3411-11e6-8364-a0369f82da38      -      -      0    259      0  11.7M
    gptid/3dc2b539-55cd-11e6-a3de-a0369f82da38      -      -      0    318  3.78K  14.5M
    gptid/d5323ca2-3411-11e6-8364-a0369f82da38      -      -      0    253      0  9.52M
    gptid/d65e21cf-3411-11e6-8364-a0369f82da38      -      -      0    193      0  8.63M
    gptid/d7943993-3411-11e6-8364-a0369f82da38      -      -      0    315      0  14.3M
  raidz2                                4.40T  13.7T      0     99      0  11.3M
    gptid/d8c35872-3411-11e6-8364-a0369f82da38      -      -      0     47      0  1.70M
    gptid/d9f20c3f-3411-11e6-8364-a0369f82da38      -      -      0     68      0  2.88M
    gptid/db270b03-3411-11e6-8364-a0369f82da38      -      -      0     35      0  1.46M
    gptid/dc5adc18-3411-11e6-8364-a0369f82da38      -      -      0     79      0  3.32M
    gptid/dd812c8c-3411-11e6-8364-a0369f82da38      -      -      0     56      0  2.38M
  raidz2                                4.40T  13.7T      1      0  7.56K      0
    gptid/deb83c65-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dfdec8a6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e10c7d1e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e2443126-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e378776d-3411-11e6-8364-a0369f82da38      -      -      1      0  7.56K      0
  raidz2                                4.40T  13.7T      0      0  3.78K      0
    gptid/e4aecd4e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e5e10597-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e70eb118-3411-11e6-8364-a0369f82da38      -      -      0      0  3.78K      0
    gptid/e83f75ba-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e970d70a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      3    293  22.7K  22.4M
    gptid/eaa7b674-3411-11e6-8364-a0369f82da38      -      -      1    330  7.56K  10.8M
    gptid/ebdacfbf-3411-11e6-8364-a0369f82da38      -      -      0    356  3.78K  15.0M
    gptid/ed0c5f7a-3411-11e6-8364-a0369f82da38      -      -      0    405      0  17.1M
    gptid/ee3d3f75-3411-11e6-8364-a0369f82da38      -      -      0    364      0  12.2M
    gptid/ef6ecd4f-3411-11e6-8364-a0369f82da38      -      -      2    321  11.3K  13.5M
  raidz2                                4.40T  13.7T      1    197  7.56K  18.5M
    gptid/f0accf0b-3411-11e6-8364-a0369f82da38      -      -      0    257      0  10.9M
    gptid/f1e496a5-3411-11e6-8364-a0369f82da38      -      -      0    293      0  9.29M
    gptid/f31c7af6-3411-11e6-8364-a0369f82da38      -      -      1    361  7.56K  13.0M
    gptid/f444c43c-3411-11e6-8364-a0369f82da38      -      -      0    345      0  10.6M
    gptid/f5774397-3411-11e6-8364-a0369f82da38      -      -      0    381      0  14.1M
  raidz2                                4.40T  13.7T      0      0  3.78K      0
    gptid/f6c96a7b-3411-11e6-8364-a0369f82da38      -      -      0      0  3.78K      0
    gptid/f7f540d6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f92a550f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fa654cda-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fb94ad0e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      3    249  15.1K  31.2M
    gptid/fcf0a3b4-3411-11e6-8364-a0369f82da38      -      -      0    356      0  15.0M
    gptid/fe2678c4-3411-11e6-8364-a0369f82da38      -      -      0    270      0  11.4M
    gptid/ff5f5215-3411-11e6-8364-a0369f82da38      -      -      0    324      0  13.7M
    gptid/008dcc92-3412-11e6-8364-a0369f82da38      -      -      0    267      0  11.3M
    gptid/01c4fd44-3412-11e6-8364-a0369f82da38      -      -      3    309  15.1K  13.0M
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                            1.83G  5.36G     13      0   389K      0
  mirror                                1.83G  5.36G     13      0   389K      0
    gptid/778d2de7-101d-11e5-9e35-002590a63c37      -      -      7      0  15.6K      0
    ada1p2                                  -      -      5      0   374K      0
--------------------------------------  -----  -----  -----  -----  -----  -----

                                           capacity     operations    bandwidth
pool                                    alloc   free   read  write   read  write
--------------------------------------  -----  -----  -----  -----  -----  -----
Production                              39.6T   124T      4  1.86K  19.1K   237M
  raidz2                                4.40T  13.7T      0    705      0  87.0M
    gptid/cceafaf7-3411-11e6-8364-a0369f82da38      -      -      0    199      0  16.5M
    gptid/ce18e36f-3411-11e6-8364-a0369f82da38      -      -      0     59      0  4.26M
    gptid/cf494223-3411-11e6-8364-a0369f82da38      -      -      0    117      0  9.94M
    gptid/d0758376-3411-11e6-8364-a0369f82da38      -      -      0    233      0  16.7M
    gptid/d1a96db2-3411-11e6-8364-a0369f82da38      -      -      0     40      0  3.40M
  raidz2                                4.40T  13.7T      0    169      0  20.6M
    gptid/d2d2523b-3411-11e6-8364-a0369f82da38      -      -      0    284      0  12.9M
    gptid/3dc2b539-55cd-11e6-a3de-a0369f82da38      -      -      0    288      0  11.8M
    gptid/d5323ca2-3411-11e6-8364-a0369f82da38      -      -      0    235      0  11.2M
    gptid/d65e21cf-3411-11e6-8364-a0369f82da38      -      -      0    153      0  6.95M
    gptid/d7943993-3411-11e6-8364-a0369f82da38      -      -      0    300      0  13.2M
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/d8c35872-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/d9f20c3f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/db270b03-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dc5adc18-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dd812c8c-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/deb83c65-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/dfdec8a6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e10c7d1e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e2443126-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e378776d-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0      0      0      0
    gptid/e4aecd4e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e5e10597-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e70eb118-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e83f75ba-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/e970d70a-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      0    474      0  59.3M
    gptid/eaa7b674-3411-11e6-8364-a0369f82da38      -      -      0    233      0  9.86M
    gptid/ebdacfbf-3411-11e6-8364-a0369f82da38      -      -      0    236      0  9.97M
    gptid/ed0c5f7a-3411-11e6-8364-a0369f82da38      -      -      0    196      0  8.28M
    gptid/ee3d3f75-3411-11e6-8364-a0369f82da38      -      -      0    202      0  8.51M
    gptid/ef6ecd4f-3411-11e6-8364-a0369f82da38      -      -      0    257      0  10.9M
  raidz2                                4.40T  13.7T      0    346  3.83K  43.3M
    gptid/f0accf0b-3411-11e6-8364-a0369f82da38      -      -      0    223      0  9.43M
    gptid/f1e496a5-3411-11e6-8364-a0369f82da38      -      -      0    269      0  11.4M
    gptid/f31c7af6-3411-11e6-8364-a0369f82da38      -      -      0    272  3.83K  11.5M
    gptid/f444c43c-3411-11e6-8364-a0369f82da38      -      -      0    277      0  11.7M
    gptid/f5774397-3411-11e6-8364-a0369f82da38      -      -      0    221      0  9.31M
  raidz2                                4.40T  13.7T      0      0  3.83K      0
    gptid/f6c96a7b-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f7f540d6-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/f92a550f-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
    gptid/fa654cda-3411-11e6-8364-a0369f82da38      -      -      0      0  3.83K      0
    gptid/fb94ad0e-3411-11e6-8364-a0369f82da38      -      -      0      0      0      0
  raidz2                                4.40T  13.7T      2    211  11.5K  26.5M
    gptid/fcf0a3b4-3411-11e6-8364-a0369f82da38      -      -      0    311  3.83K  13.1M
    gptid/fe2678c4-3411-11e6-8364-a0369f82da38      -      -      0    211      0  8.93M
    gptid/ff5f5215-3411-11e6-8364-a0369f82da38      -      -      0    277      0  11.5M
    gptid/008dcc92-3412-11e6-8364-a0369f82da38      -      -      0    260      0  11.0M
    gptid/01c4fd44-3412-11e6-8364-a0369f82da38      -      -      1    281  7.66K  11.9M
--------------------------------------  -----  -----  -----  -----  -----  -----
freenas-boot                            1.83G  5.36G     43      0   525K      0
  mirror                                1.83G  5.36G     43      0   525K      0
    gptid/778d2de7-101d-11e5-9e35-002590a63c37      -      -     14      0  35.9K      0
    ada1p2                                  -      -     28      0   489K      0
--------------------------------------  -----  -----  -----  -----  -----  -----

--------------------------------------  -----  -----  -----  -----  -----  -----
 
S

scottsee

Cadet
Joined
Aug 5, 2016
Messages
8
Mlovelace said:
45 4TB drives on a system with only 32GB of RAM?!?!? :eek: Your system is under-spec'd for that quantity of storage. What is the zpool layout?
Click to expand...

Yes it is, I know that too. I'm trying to see if hardware wise it can handle what I'm throwing at it. the MB is maxed out at 32GB. It's $2K for the new guts to put dual E52620's and 128GB.

My Production pool is 9 x raidz2 with 5 disks each.
 
styno

styno

Patron
Joined
Apr 11, 2016
Messages
466
How did you setup/ran these performance tests? It seems that your disks are not stressed at all...
 
S

scottsee

Cadet
Joined
Aug 5, 2016
Messages
8
styno said:
How did you setup/ran these performance tests? It seems that your disks are not stressed at all...
Click to expand...

That's my point exactly. The write speed never increases as my work load to this device increases. I ran 3 tests, all showed the same results. Tests were permed from our enterprise filers to the FreeNAS device. I'm looking what would cause what I'm seeing here.

Filer 1 sent 100,000 files (860GB)
Filer 1 and 2 both sending 100,000 files (860GB)
Filer 1, 2 and 3, all sending 100,000 files (860GB)

This was performed over 40gb network.
 
Dice

Dice

Wizard
Joined
Dec 11, 2015
Messages
1,410
If this is for <anything else than just pure shits n giggles> then disregard the below:

I would not bother doing any tests and even considering applying data in this described case.
Since you appear to be aware of the extent of undersized system - you should definitely get proper hardware before wasting more time.
 
Status
Not open for further replies.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "45x 4TB FreeNAS Testing Results"

Similar threads

computerconnections
Memory Errors
2
Replies
22
Views
14K
Radu
Radu
G
  • Locked
System has started crashing repeatedly. Pls help with some troubleshooting.
Replies
6
Views
3K
cyberjock
C
D
  • Locked
CCB errors with USB boot
Replies
2
Views
4K
digity
D
F
  • Locked
CAM status: SCSI status error - what does it means?
Replies
7
Views
19K
cyberjock
C
D
  • Locked
Replacing Failed Disk Issue
Replies
8
Views
3K
Robert Trevellyan
R
Top