Recommendations to improve an odd server?

turboaaa

Dabbler
Joined
Dec 31, 2017
Messages
36
First, let me say that I was able to correct most of my problems by just searching this forum. This has been a great resource, I will not argue with anyone telling me that I have done something stupid. I am aware that the hardware installed is not ideal, and if this is the cause of my performance hit then I must accept the consequences of my choices. This server is used as a backup and performance does not need to be the best or even that good.

The "problem" : I am hitting ~6.4Gbs over NFS from esxi, ideally I would want to max out my 10Gbs uplink. This could just be a case of me not properly understanding the limitations of NFS and the hardware I am using. I can live with this, but want to see what recommendations are made for improvement.

I started with 3 x raidz2 vdevs, but am now running 7 x 3 way mirrors. Oddly I did not see any improvements when making this change, I assume this is because of another limiting factor on my system. The drives are a mix of 7200RPM and 5400RPM models, so I expect performance to match the 5400RPM disks. I honestly do not have the experience to say if this is the limiting factor. This was done with the expectation of replacing the 1TB disks with 2TB as they begin to fail. Eventually we will probably just want the extra capacity. (2.5" disks).
  • This server is providing NFS storage to an esxi cluster.
  • I am currently testing performance with sync disabled as the SSDs that came with the server, though have decent latency for their age, have throughput limiting me to ~160MB/s. I plan on replacing with Optane PCie cards in the near future.
  • The switch is a Unifi x16 10Gb.
  • Load is ~150 Windows 10 VMs. Each VM has 11GB of RAM and files are stored on another server. Just OS and apps are on the VMs.
Server :

OS Version:
FreeNAS-11.2-U6
(Build Date: Sep 17, 2019 0:16)
Processor:
Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz (20 cores)
Memory:
384 GiB

Controller : IT Mode LSI SAS 9207-8i SATA/SAS 6Gb/s PCI-E 3.0 Host Bus Adapter LSI00301
NIC : 10Gb network card is the one sold on Amazon as the FreeNAS Mini-XL upgrade. It shows up as a Chelsio.
Disks : Mix of ST2000LM015-2E8174 and ST91000640SS
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
I am hitting ~6.4Gbs over NFS from esxi

Is this in reads, writes, or total bandwidth? That's a pretty good number if it's purely ingesting writes from an ESXi cluster; mind you, being async is really what's allowing this to happen. Are the VMs actively running from the NFS export on FreeNAS, or are they being backed up to it?

Eventually we will probably just want the extra capacity. (2.5" disks).

Badger Sense activated. Desire for inexpensive capacity in 2.5" format leads to ...

ST2000LM015

... SMR disks. If the write workload is sufficient to overwhelm the non-shingled "cache portion" of the drives, or there isn't enough time for the drives to destage internally, you'll hit extremely low performance levels. SSH into your host and check the results of gstat -dp to see if your drives have a particularly high amount of latency on writes, or if they're doing writes/deletes/etc when you aren't pushing a large amount of activity to the pool.
 

turboaaa

Dabbler
Joined
Dec 31, 2017
Messages
36
This would be writes, reads are coming from the ARC. I forgot to mention that the ~6.4Gbps is not always sustained, though it is relatively consistent from within one of the VMs running disk benchmarks. I am in the middle of cloning 90 images and below is a snapshot of the gstat output as well as the interface stats.

You are very correct that those cheap SATA disks are a concern. If this was more of a production server I would not consider it. I even failed to check that they were shingled (egg on my face). If that is the case I will probably just use 1TB 7200 RPM disks, and add on an external tray of drives if needed in the future.

1571855361296.png


Code:
dT: 1.003s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da0
    6    171      1     64   13.7    169  19649   13.1      0      0    0.0   44.2| da1
    8    178      1     16   10.4    177  20626   11.6      0      0    0.0   43.2| da2
   52    175      2     84   22.4    172  20175   12.7      0      0    0.0   44.4| da3
    5    188      2     64   22.3    185  19980    3.5      0      0    0.0   23.0| da4
    3    166      0      0    0.0    165  18683   11.5      0      0    0.0   47.3| da5
    8    168      1    108   13.7    166  19054   12.1      0      0    0.0   48.9| da6
    3    192      4     68   10.6    187  22780   11.9      0      0    0.0   45.2| da7
    1    218      1      8   25.8    216  24643    4.6      0      0    0.0   28.8| da8
    8    180      0      0    0.0    179  21200   27.3      0      0    0.0   45.6| da9
    8    226      2     76   19.8    223  26286    6.8      0      0    0.0   29.2| da10
    8    226      1     72   34.7    224  25361    5.6      0      0    0.0   28.1| da11
    6    198      0      0    0.0    197  23394   13.6      0      0    0.0   46.4| da12
    5    195      0      0    0.0    193  22481   12.2      0      0    0.0   51.4| da13
    5    203      0      0    0.0    201  23306   12.0      0      0    0.0   50.5| da14
    2    196      4    323   14.1    190  22118   13.1      0      0    0.0   50.5| da15
    2    208      0      0    0.0    206  24375   12.4      0      0    0.0   51.2| da16
   23    194      0      0    0.0    192  22134   14.1      0      0    0.0   51.6| da17
    8    202      0      0    0.0    200  23610   12.9      0      0    0.0   50.3| da18
    4    202      1     72   21.0    199  23546   13.2      0      0    0.0   50.7| da19
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da20
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da21
    0    337      2     52    8.3    333  34942    1.9      0      0    0.0   28.2| da22
    0    389      0      0    0.0    323  34942    1.9     64  12345   64.8   60.3| da23


EDIT :

Better gstat snapshot. da22/23 are the 5400rpm drives.

Code:
dT: 1.052s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da0
    0    576      8    209  249.0    569  71173  113.6      0      0    0.0   99.6| da1
    9    522      9    624  164.6    513  63482  114.8      0      0    0.0   99.9| da2
    0    540     13    848  119.7    527  65513  114.0      0      0    0.0   99.6| da3
   14    570      0      0    0.0    570  66354   67.5      0      0    0.0   89.0| da4
    0    577      3    175  882.4    574  71447   98.1      0      0    0.0   99.3| da5
    0    565      5    327  149.7    560  69640  106.9      0      0    0.0   98.5| da6
    0    581      7     99   35.3    574  71260  108.1      0      0    0.0   99.4| da7
    4    546      0      0    0.0    546  64205   87.7      0      0    0.0  100.0| da8
   18    547      5     57   19.9    542  67263  107.5      0      0    0.0  100.0| da9
    0    397     26    928   38.7    371  37298   19.0      0      0    0.0   80.3| da10
    8    315      6    183  123.4    309  38253  192.1      0      0    0.0   96.7| da11
    0    240      8    677  171.1    232  28732  238.4      0      0    0.0  108.5| da12
    0    261      5    346  225.9    257  32448  268.6      0      0    0.0  109.5| da13
    9    288      2     95   98.4    286  36153  233.2      0      0    0.0  111.1| da14
   16    277      5    373  210.2    272  33745  221.8      0      0    0.0  109.2| da15
   31    242      4    278  206.1    239  29995  261.9      0      0    0.0  106.6| da16
    0    288      2    194  168.7    286  35849  231.5      0      0    0.0  105.3| da17
   37    246      4    285  231.6    242  30189  252.6      0      0    0.0  100.1| da18
   28    328      1     53  261.7    327  40539  213.7      0      0    0.0  100.1| da19
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da20
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da21
    0    429      9    377   32.5    420  41976   22.9      0      0    0.0   50.3| da22
    0    540      5     76   65.9    474  48484   30.9     61   3982   64.4   59.5| da23
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
da3 and da17 have significantly higher queue depths going on than others. Are they 2TB drives by any chance?

According to this, the drive is uses PMR, not SMR. Let me know what I am reading wrong because it wouldn't be the first time.

https://www.seagate.com/www-content...op-fam/barracuda_25/en-us/docs/100807728a.pdf

You're just missing a newer revision of the document; although even then that's not a guarantee because Seagate seems to "conveniently forget" to put this as a bullet point on their drive details.

https://www.seagate.com/www-content...op-fam/barracuda_25/en-us/docs/100807728g.pdf

Under 1.0 Introduction:

Shingled magnetic recording with perpendicular magnetic recording heads/media.

They'll call it "perpendicular recording" because of the position of the heads, but the track layout is shingled.
 

turboaaa

Dabbler
Joined
Dec 31, 2017
Messages
36
Marketing materials are always fun. Thanks for pointing that out.

da22 and 23 are the 2TB drives. I grabbed another quick snapshot showing more activity.

Code:
dT: 1.052s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da0
    0    576      8    209  249.0    569  71173  113.6      0      0    0.0   99.6| da1
    9    522      9    624  164.6    513  63482  114.8      0      0    0.0   99.9| da2
    0    540     13    848  119.7    527  65513  114.0      0      0    0.0   99.6| da3
   14    570      0      0    0.0    570  66354   67.5      0      0    0.0   89.0| da4
    0    577      3    175  882.4    574  71447   98.1      0      0    0.0   99.3| da5
    0    565      5    327  149.7    560  69640  106.9      0      0    0.0   98.5| da6
    0    581      7     99   35.3    574  71260  108.1      0      0    0.0   99.4| da7
    4    546      0      0    0.0    546  64205   87.7      0      0    0.0  100.0| da8
   18    547      5     57   19.9    542  67263  107.5      0      0    0.0  100.0| da9
    0    397     26    928   38.7    371  37298   19.0      0      0    0.0   80.3| da10
    8    315      6    183  123.4    309  38253  192.1      0      0    0.0   96.7| da11
    0    240      8    677  171.1    232  28732  238.4      0      0    0.0  108.5| da12
    0    261      5    346  225.9    257  32448  268.6      0      0    0.0  109.5| da13
    9    288      2     95   98.4    286  36153  233.2      0      0    0.0  111.1| da14
   16    277      5    373  210.2    272  33745  221.8      0      0    0.0  109.2| da15
   31    242      4    278  206.1    239  29995  261.9      0      0    0.0  106.6| da16
    0    288      2    194  168.7    286  35849  231.5      0      0    0.0  105.3| da17
   37    246      4    285  231.6    242  30189  252.6      0      0    0.0  100.1| da18
   28    328      1     53  261.7    327  40539  213.7      0      0    0.0  100.1| da19
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da20
    0      0      0      0    0.0      0      0    0.0      0      0    0.0    0.0| da21
    0    429      9    377   32.5    420  41976   22.9      0      0    0.0   50.3| da22
    0    540      5     76   65.9    474  48484   30.9     61   3982   64.4   59.5| da23
 

turboaaa

Dabbler
Joined
Dec 31, 2017
Messages
36
lol, now that I actually have all 90 VMs running things start to hurt. I can still use this as intended, but the performance is not great (not super surprised). The only other thing I can think of is to move these to two way mirrors instead of three, and increase the number of vdevs. Definitely see why NetApp is worth the money.

Speaking of which. I was looking at some of the ixsystems all flash servers to replace the NetApp appliance that powers our VMs. Any recommendations on that front? I plan on keeping NetApp for other purposes, but I feel we could save some money going with another storage solution. Not one I build, but one built and supported to provide the write performance to keep my users happy.
 
Last edited:

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
All-flash is definitely the way to go for any kind of serious sustained performance, and 90 VMs sounds like you're in that domain. While I've never used an official iXsystems TrueNAS box myself, I have to imagine it's got similar options for performance and tuning as FreeNAS (with a few more tweaks under the hood, since it's consistent hardware.)

Feel free to reach out to the iX team, or possibly find one of them lurking around as forum members. I know I see @dlavigne with the official blue iXsystems tag popping up periodically, perhaps we can summon him to reach out and engage with you if you're interested. :)
 
Top