Help Choosing HBA

[clifford64]

I have tried searching, but I am having a hard time finding the information that will be helpful to me.

Background: I currently have a TrueNAS 13 system running 12 HDDs as a 6x2 mirror. I am currently upgrading hard drives, but running WD Reds. I am getting pretty lack luster performance in terms of drive latency performing day to day operations. My setup is a 9200-8i in IT mode that connects to a PCIE SAS expander that is an IBM 46M0997 which then connects to the 12 drives. I am guessing that SAS expander is possibly the bottleneck and might be having some compatibility issues. I was pretty new to TrueNAS when I set up the system a couple years ago, I didn't realize that it probably would have been better to go with a second HBA instead of this SAS expander.

Actual questions:

1. Are there any good SAS 3 HBAs for under $80 a piece? I may be searching for the wrong items, but I am having a hard time finding SAS 3008 cards for used prices. I read an article on Serve the Home that recommended SAS 3008 and new chips over the 2008 and said they were cheap now.
Yes there are a few out there, I just wasn't looking hard enough. The majority appear to be over $100 as they are still coming down in price.

2. Does anyone else think this SAS expander I currently have is probably a bad idea as well? I didn't find very many PCIE expanders and most were backplane only expanders and my server currently doesn't have a backplane.
There doesn't appear to be anything wrong with this setup.

3. Is it better to just purchase another 9200-8i in IT mode and remove my SAS expander and use that?
There doesn't appear to be anything wrong with this setup.

4. When I redo my setup, I am planning on doing a 2x6 raidz2. Do you think there might be any issues with having vdevs spawn multiple HBAs and I should just get 1 single card that is 16 port?

5. I have very little knowledge of HBAs and iops performance, would going with another 9200 card risk iops issues with 12 drives like that? I am assuming not because these cards in theory should be able to support many times the amount of drives I have, but the Serve the Home article said to go with the newer controllers because of iops. I am guessing this is because SSDs are more common in pools now and they would be able to max out the iops on a controller depending on how many you have.

 

[ChrisRJ]

Can you be more specific about what performance you see and how it differs from the expected results?

Ad point 3, this would certainly increase the bandwidth per drive. Since such an HBA should be pretty cheap, why not just try it?

 

[clifford64]

Can you be more specific about what performance you see and how it differs from the expected results?

Ad point 3, this would certainly increase the bandwidth per drive. Since such an HBA should be pretty cheap, why not just try it?

Random reads and writes seems to be very high in latency. To the point where I am getting CAM errors within some of the VMs I am running. I only have 12 VMs running on two different ESXi hosts connected to truenas through iSCSI.

I probably will go with another 9200, just figured I would get some thoughts from others first before I just jump right into it again. Without having extensive experience with storage administration, I wanted to make sure my thinking is correct and if anyone else sees any obvious problems with my thinking.

 

[ChrisRJ]

How are there various system load parameters? Are there patterns to see? 12 VMs is not very specific in terms of the load they cause, so can you be more specific here?

 

[HoneyBadger]

Random reads and writes seems to be very high in latency. To the point where I am getting CAM errors within some of the VMs I am running. I only have 12 VMs running on two different ESXi hosts connected to truenas through iSCSI.

Can you confirm your system specs? Your signature is at odds with itself, as you've listed your motherboard as an X8SIL (which is the Xeon 5500/5600 generation) but your CPU is shown as an E3-1245v2 (Ivy Bridge) - that physically wouldn't fit in the socket. Can you let me know if maybe an update is in order?

VM hosting is one of the toughest workloads for any storage array. Mirrors are the right choice here certainly, but iSCSI also wants a lot of RAM - and in your case, I don't see an SLOG device in your signature either. Are you running your zvols with sync=standard?

Sight unseen, my guesses start with checking the physical layer.

Network cables are securely attached and not kinked or bent? Replug them if you have the redundancy or can arrange for an interruption in service. Use another client to check with iperf (I believe it can be loaded onto ESXi but it's not present normally) to ensure that network bandwidth is good.

If you have a drive that's failing or experiencing periodic errors, that could be the cause. If you run gstat -dp on your TrueNAS machine, do you notice one drive out of the others that's showing up with higher response times?

 

[clifford64]

How are there various system load parameters? Are there patterns to see? 12 VMs is not very specific in terms of the load they cause, so can you be more specific here?

Not many patterns. Seems to be kinda random. I believe a lot of performance will be resolved when I redo my system. Main storage will be put into the HDD pool with 2 raidz2s with 6 drives each and this will be NAS only with SMB shares and no iscsi. I will then be doing an SSD pool in a mirror for iscsi and VMs. Probably hosted with the internal SATA controller. I think we are getting a little off topic here and I would like to circle back to the more specific HBA questions I had specifically.

Can you confirm your system specs? Your signature is at odds with itself, as you've listed your motherboard as an X8SIL (which is the Xeon 5500/5600 generation) but your CPU is shown as an E3-1245v2 (Ivy Bridge) - that physically wouldn't fit in the socket. Can you let me know if maybe an update is in order?

VM hosting is one of the toughest workloads for any storage array. Mirrors are the right choice here certainly, but iSCSI also wants a lot of RAM - and in your case, I don't see an SLOG device in your signature either. Are you running your zvols with sync=standard?

Sight unseen, my guesses start with checking the physical layer.

Network cables are securely attached and not kinked or bent? Replug them if you have the redundancy or can arrange for an interruption in service. Use another client to check with iperf (I believe it can be loaded onto ESXi but it's not present normally) to ensure that network bandwidth is good.

If you have a drive that's failing or experiencing periodic errors, that could be the cause. If you run gstat -dp on your TrueNAS machine, do you notice one drive out of the others that's showing up with higher response times?

I need to update my signature. I have an Intel S1200BTL. I believe a lot of performance will be resolved when I redo my system. Main storage will be put into the HDD pool with 2 raidz2s with 6 drives each and this will be NAS only with SMB shares and no iscsi. I will then be doing an SSD pool in a mirror for iscsi and VMs. Probably hosted with the internal SATA controller. But we are getting a little off topic. I know I have other items with my system that needs to be addressed, but I'd like to circle back more to the HBA aspect of it with my current setup and the specific questions I mentioned above.

 

[HoneyBadger]

But we are getting a little off topic. I know I have other items with my system that needs to be addressed, but I'd like to circle back more to the HBA aspect of it with my current setup and the specific questions I mentioned above.

Then my short answer is "barring a hardware failure, your existing SAS2008 HBA and expander should not be a bottleneck" - even if your HBA is in one of the x4 slots on your S1200BTL, that's still PCIe 2.0 x4 = 2GB/s of bandwidth.

I would look at testing the pool performance locally (using something like fio) and then if that checks out, test the network bandwidth via iperf to rule that out.

But even if everything is functioning as expected, a 2x10GbE iSCSI MPIO link is more than capable of completely overwhelming the write speed of 6 HDD mirror vdevs.

 

[ChrisRJ]

I think we are getting a little off topic here and I would like to circle back to the more specific HBA questions I had specifically.

Quite honestly, this collides with your statements about not knowing too much about storage. You assume that the issue is with your SAS expander, which means that it could be anywhere. I don't want to attack you, and you are not the first to make that kind of statement. But please think about it.

Would you say the same thing to to an IT professional that you had hired for USD 200 / hour? Probably you would accept his judgement that your initial assessment of the situation may(!) have been wrong and that additional information and testing is needed before a recommendation can be made. And that is exactly what @HoneyBadger and I have told you.

 

[clifford64]

Quite honestly, this collides with your statements about not knowing too much about storage. You assume that the issue is with your SAS expander, which means that it could be anywhere. I don't want to attack you, and you are not the first to make that kind of statement. But please think about it.

Would you say the same thing to to an IT professional that you had hired for USD 200 / hour? Probably you would accept his judgement that your initial assessment of the situation may(!) have been wrong and that additional information and testing is needed before a recommendation can be made. And that is exactly what @HoneyBadger and I have told you.

I am truly not trying to be rude, but I am just looking for answers to my specific questions. My knowledge is lacking when it comes to HBAs and those are the specific questions I have about them and how they are utilized in my setup.

I have gone over performance of my system in different threads and have asked for help on it before. I am quite aware that if I want amazing performance out my system, that I would need to do significant upgrades. Those upgrades cost money and I really don't have much for this project. This thread is trying to answer some questions I have about the HBA setup I have in my system, and in general, as now would be the time to make any adjustments if it has any impact on my current performance as I am going to be changing the main pool I have to 2x6 Raidz2s at file level nas storage and then going with an SSD pool for iSCSI VMs. I was trying to provide a little background information without trying to type out my entire NAS' life story as I sometimes tend to do.

Then my short answer is "barring a hardware failure, your existing SAS2008 HBA and expander should not be a bottleneck" - even if your HBA is in one of the x4 slots on your S1200BTL, that's still PCIe 2.0 x4 = 2GB/s of bandwidth.

I would look at testing the pool performance locally (using something like fio) and then if that checks out, test the network bandwidth via iperf to rule that out.

But even if everything is functioning as expected, a 2x10GbE iSCSI MPIO link is more than capable of completely overwhelming the write speed of 6 HDD mirror vdevs.

Looking at the SAS expander (not really sure how to describe this), is this is normal setup? I have seen SAS expanders in terms of backplanes, but not actual PCIE cards before. Do you think I should swap it out for another HBA? Or would that make no difference at all and be a waste of money?

Looking at a new SSD pool, would it be better to utilize the built in SATA controller? Or connect them to the HBA?

 

[ChrisRJ]

All I was trying to do is convey my opinion that the "specific question" may actually be the wrong one to ask. I did this because the wording suggested to me(!) that there was some premature focus on this topic, rather than first understanding the bigger picture. If proper analysis, rather than guesswork, has been the foundation for this direction, all is good. The only thing I would recommend then, is to avoid words like "think", "believe", "assume" in the future. Those all strongly imply that something is not really clear.

Good luck!

 

[HoneyBadger]

Looking at the SAS expander (not really sure how to describe this), is this is normal setup? I have seen SAS expanders in terms of backplanes, but not actual PCIE cards before. Do you think I should swap it out for another HBA? Or would that make no difference at all and be a waste of money?

PCI-card style expanders are common for tower systems or direct-attach backplanes - they start to become bottlenecks when you're looking at a couple dozen SSDs, but for HDDs you're usually well below the theoretical limits of eight SAS2 lanes. I would save the money here.

Looking at a new SSD pool, would it be better to utilize the built in SATA controller? Or connect them to the HBA?

TRIM on SATA SSDs behind SAS controllers has been challenging especially on earlier LSI HBAs. With a small number of them, just hook them directly to your motherboard SATA controller.

I would like to highlight the question again about sync writes though:

VM hosting is one of the toughest workloads for any storage array. Mirrors are the right choice here certainly, but iSCSI also wants a lot of RAM - and in your case, I don't see an SLOG device in your signature either. Are you running your zvols with sync=standard?

When running remote iSCSI workloads with sync=standard you're opening yourself up to a non-zero chance of data loss, since your remote writes are being told they are completed when they are accepted into a ZFS transaction group, as opposed to sync=always where it will not return complete until the write is on non-volatile storage. Turning this on for an HDD pool with no SLOG is "punishing" to say the least, but you may see less impact on an all-SSD pool.

 

[nabsltd]

Are there any good SAS 3 HBAs for under $80 a piece?

Although $83 is more than $80, these work well with TrueNAS. You can cross-flash them to LSI 9300-8i IT mode.

 

[Redcoat]

Look at The Art of Server store on eBay (vendor with very good rep here) - examples in the $70's right now.

 

[clifford64]

PCI-card style expanders are common for tower systems or direct-attach backplanes - they start to become bottlenecks when you're looking at a couple dozen SSDs, but for HDDs you're usually well below the theoretical limits of eight SAS2 lanes. I would save the money here.



TRIM on SATA SSDs behind SAS controllers has been challenging especially on earlier LSI HBAs. With a small number of them, just hook them directly to your motherboard SATA controller.

I would like to highlight the question again about sync writes though:



When running remote iSCSI workloads with sync=standard you're opening yourself up to a non-zero chance of data loss, since your remote writes are being told they are completed when they are accepted into a ZFS transaction group, as opposed to sync=always where it will not return complete until the write is on non-volatile storage. Turning this on for an HDD pool with no SLOG is "punishing" to say the least, but you may see less impact on an all-SSD pool.

Thank you for the answers. Yes, my zvol is set sync=standard. I don't think I described the performance of it in a very good manner. Again, the thread wasn't really meant to focus on the overall, but more hardware that I currently have and it's potential impact.

Day to day performance over iSCSI I would say is actually pretty good. Most of my VMs are not actually doing anything on a regular basis. It is only if one of them overlaps with another and multiple VMs start getting loaded down. Doing something in one VM on a daily basis and I don't have any problems at all. I have received the CAM errors in one of my VMs, but I have not noticed any issues with it's actual performance. When putting a load on multiple VMs at once is when I start to notice some performance, but obviously this is going to be expected with the number of vdevs and random iops that are occurring from multiple sources at once. In all actuality, I think I have reached the peak performance that I can with actual hard drives. I just wanted to make sure that the HBA and SAS expander setup I have aren't holding back anything else that might be left on the table. If I want more performance, then I would definitely have to run things with SSDs which is kind of my next step moving forward with the VM pool and mass storage on the HDD pool.

I will keep the SAS expander and 9200-8i for now. New SSD pool will connect directly to SATA Controller.

 

[HoneyBadger]

Again, the thread wasn't really meant to focus on the overall, but more hardware that I currently have and it's potential impact.

I understand, but I feel I'd be remiss if I didn't point out the potential entry into the Danger Zone that you've (inadvertently?) made.

It is only if one of them overlaps with another and multiple VMs start getting loaded down.

I believe this is actually a case of your network being too fast for your disks, and there being no more gradual bottleneck in the middle in the form of an SLOG device. There's a quote in my signature regarding "watering a potted plant with a firehose" that applies with multiple 10GbE interfaces being pointed at disks, and 20Gbps (2GB/s) is far more than your spinning disks can provide.

Check the spoiler tag below - try this dtrace script out and see if you're constantly bumping off the limit of your maximum dirty data (~3200MB)

Show : dirty.d

Paste this code into a file in vi/nano and launch via dtrace -s dirty.d YourPoolNameGoesHere

Code:

txg-syncing
{
        this->dp = (dsl_pool_t *)arg0;
}

txg-syncing
/this->dp->dp_spa->spa_name == $$1/
{
        printf("%4dMB of %4dMB used", this->dp->dp_dirty_total / 1024 / 1024,
            `zfs_dirty_data_max / 1024 / 1024);
}

In all actuality, I think I have reached the peak performance that I can with actual hard drives.

More RAM would help greatly, but you've unfortunately reached a platform limit with the S1200BTL. I've seen 12 spinning disks put up good performance for a larger VM workload, but it was also sporting 144GB of RAM and 400GB of L2ARC, so the HDD vdevs only had to handle whatever was outside the hottest half-terabyte, and the write workload.

 

[clifford64]

Check the spoiler tag below - try this dtrace script out and see if you're constantly bumping off the limit of your maximum dirty data (~3200MB)

I ran the code you specified and it was only coming back with anywhere from 1-60MB. I have no idea what this code does or what you are talking about, but I am assuming that because it is nowhere near that limit, it probably doesn't have a big negative performance impact on the pool.

Anyways, I went with the recommendations here and kept the system the same and just dropped in the SSDs to create an iSCSI SSD pool for the VMs. I am still experimenting with this new SSD pool, so I am doing my best to work out the bugs. Will probably open a new thread if I can't figure it out.

Truth be told, I think I was expecting SSD-like performance with so many hard drives and the the way they have been configured. I should have obviously known that this would not be the case and I think that is what got us a little off track. I have just been so used to using SSDs in other computers that I have forgotten the standard performance of hard drives.

Anyways, I appreciate the feedback and the assistance here.

 

[HoneyBadger]

I ran the code you specified and it was only coming back with anywhere from 1-60MB. I have no idea what this code does or what you are talking about, but I am assuming that because it is nowhere near that limit, it probably doesn't have a big negative performance impact on the pool.

Were you putting a heavy write workload on the system while this script was running though? What it does is simply output the current amount of "dirty" or "pending" data in RAM. If you go above 60% of your maximum, ZFS will apply artificial delays to incoming writes.

Anyways, I went with the recommendations here and kept the system the same and just dropped in the SSDs to create an iSCSI SSD pool for the VMs. I am still experimenting with this new SSD pool, so I am doing my best to work out the bugs. Will probably open a new thread if I can't figure it out.

Feel free to @ me if/when you do. Please take the warning about sync writes to heart though if you plan to run important workloads on this though.

Truth be told, I think I was expecting SSD-like performance with so many hard drives and the the way they have been configured. I should have obviously known that this would not be the case and I think that is what got us a little off track. I have just been so used to using SSDs in other computers that I have forgotten the standard performance of hard drives.

It really is a whole different world of performance when comparing HDD to SSD. Every time I boot up an old machine with a hard drive, I'm quite convinced that it's broken, until I remember that was what computing used to be.