Added SSD's assuming performance? None?

[BSTAMPER]

So I have a DL380P G9 and was using what it came with (free) on the HP storage controller. Ran out of space and figured if I'm gonna need more I might as well buy SSD. Bought some 1TB SSD's and threw them on a 9200-8i. Everything I do on this is VMFS. I somewhat assumed I'd see a huge performance increase over the spindles. I installed Cisco ISE and on install it does a read/write check/test. Read on the spindles was 382MB/second and write was like 91 MB/second. So I deployed it onto the SSD lun and did the same test. Expected some major write gains. Ended up with 403 MB/second read and 94MB/second write. Am I missing something? Should I be seeing way better than this? I mean my original problem is solved in the fact I needed more storage but should I expect more out of the 9200-8i with SSDs?

 

[Patrick M. Hausen]

What precisely is your zpool topology?

 

[BSTAMPER]

Appreciate your reply!! Below is the output of zpool list and zpool status. IN short I have 3. One is the SAS drives connected to the HP controller (again don't care much about these). Then 2 pools with SSD's. One with 4x 1TB SSDs and one with 120GB SSD's. All Raidz. Today I rebuilt it all. Started from scratch, and built it to this point. I'm still seeing the same results.

Code:

root@truenas[~]# zpool list
NAME          SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
SAS-Pool-01  3.25T   225M  3.25T        -         -     0%     0%  1.00x    ONLINE  /mnt
SSD-Pool-01  3.62T  31.3G  3.59T        -         -     0%     0%  1.00x    ONLINE  /mnt
SSD-Pool-02   350G  49.4M   350G        -         -     0%     0%  1.00x    ONLINE  /mnt
boot-pool     238G  1.16G   237G        -         -     0%     0%  1.00x    ONLINE  -
root@truenas[~]# zpool status
  pool: SAS-Pool-01
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        SAS-Pool-01                                     ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/485eddb5-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/48a5374b-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/48fd2661-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/49264c53-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/49126860-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/4941e38d-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/497f6960-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/495f6149-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0

errors: No known data errors

  pool: SSD-Pool-01
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        SSD-Pool-01                                     ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/778bb4eb-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/77e86d2b-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/77e997a5-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/783d76a9-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0

errors: No known data errors

  pool: SSD-Pool-02
 state: ONLINE
config:

        NAME                                            STATE     READ WRITE CKSUM
        SSD-Pool-02                                     ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/685668b4-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/6870c0b6-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0
            gptid/687e47cc-b828-11eb-a123-2c768a55f8b4  ONLINE       0     0     0

errors: No known data errors

  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          da15p2    ONLINE       0     0     0

errors: No known data errors

 

[BSTAMPER]

In addition to keep my existing VMs/Data I setup starwinds free on my windows box and shared out an iscsi target from an SSD on it. I'm seeing expected speeds on this same ISE install 380Mb/s write speeds over the 90 I'm getting from the Freenas. This proves the ESX host/Network/iScsi isn't the issue. The weird thing is both the SAS and SSD appear to cap out on write speed about the same 90-100Mb/s on this install. I've beat my head against this so long I'm not sure what the deal is.

 

[Patrick M. Hausen]

RAIDZ is generally not recommended for iSCSI storage. ome good documents can be found here:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.truenas.com

Resource - Why iSCSI often requires more resources for the same result

iSCSI is a SAN protocol. NFS, CIFS, etc., are NAS protocols. For a NAS protocol, the client sends a command to the filer, such as "open this file", or "read ten blocks", or "remove this file." On the filer, the local NAS protocol daemon...

www.truenas.com

 

[BSTAMPER]

Again, thanks much for your response. I tried this a couple ways. Not sure if they are good tests or not. I took two of the SSD's and put them in a mirror. No change. I then took and just added one to the extent with no pool. Again, no change in speed? WOuld those be valid tests to say something is wrong here? My next test is going to be getting freenas setup on another machine I have to validate its not something with the hardware but it sure does seem like I should get more than this.

 

[BSTAMPER]

So it appears this had nothing to do with the Raidz but everything to do with the network. After much testing I found that the network cards were pegging out at 1G/s. I put in MPIO and still wasn't getting great throughput although better. Ended up getting an x520-2 10G card and upgrading to that. Now my throughput to the ssd's is hitting 400-600MB/s vs the 100MB/s from before. This is looking much better now. Even the SAS is looking at 300-400. So all around everything got better!

 

[jgreco]

The X520-{DA,SR}2 is an older card (12 years old, Q3 2009) but is a real workhorse. It is probably not going to be the absolute fastest, but it definitely has the best general compatibility of anything I know of. I do recommend it, but it is limited to PCIe 2.0, and may require some extra tuning to get the maximum possible out of it. The Chelsio T520-CR is probably the best FreeNAS-compatible card out there.

 

[BSTAMPER]

Yea, I know its a pretty old card. I actually have it running in my desktop as well. Using iperf and can easily saturate the 10G from another machine on 10G. In this case I think the 400-600MB/s read/write is the max these SSD's will do so I'm not to concerned with it. My whole issue was with a VM that I'm running that at boot checks the I/O and it needs to be above 200 or it throws a warning and in many cases has issues installing.