Last Straw! ZFS/Disk Performance Issues

[cyberjock]

A couple things from supermicro's tech support:

The backplanes are built to SATA II specs.
They are thinking that the mobo SATA signals may not be strong enough for the added circuitry of the backplane.

Personally I find it hard to believe that it's built to SATA II specs. If it were they'd have included it in the manual. It's all about using big words and other technical stuff to make their product look good. Nobody would sell a SATA II or SATA III controller and call it "SATA". I call BS on Supermicro. It's totally possible that it may/should work at SATA II speeds, but if it doesn't specifically call it out I wouldn't believe it. SATA uses a current loop for the data signal. Not sure if I'd believe that the signal isn't strong enough because of the way current loops work. /sigh

The M1015 is used alot in this forum. There's definitely no downside to upgrading. It may be that the M1015 will work with the backplanes just because it can handle(and ignore) extra noise in the circuits(or whatever the problem is).

 

[TravisT]

I've done a couple other tests, and I've received my M1015 that I'll be trying in the box as soon as my cables arrive.

I have 4 1TB drives installed in the server now, and I wanted to see what kind of performance I could get by creating a RAID 10 volume with ZFS. I did, and my overall write performance was about 94 MB/s using the same tests I have been using. Does this indicate poor performance of my onboard SATA controller? I'm hoping so, and that the M1015 flashed to LSI IT firmware will speed things up considerably (and fix my drive bay problems).

 

[TravisT]

Update: 160Mb/s read in the same configuration. Can't wait for those cables to arrive to compare.

 

[TravisT]

So I received my forward breakout cables yesterday for my M1015 flashed to P14 "IT" firmware.

I've had a couple of strange mps error messages since installing the card in the new freenas box. Not sure what is causing the errors, but if I see them again, I'll investigate further.

Bare drive connected to the controller has a write speed of about 130MB/s. Drive installed in the drive bays again drops down to nearly nothing. I know that several options have been thrown out here, I'm having a hard time accepting the fact that all three drive bays are defective, or that essentially no one else has reported slow transfer rates of the 100+ reviews at newegg, among other retailers.

Is there any way to dive deeper and see what is actually going wrong here? Whether it's noise on the SATA bus, or something else, I really wish I could nail it down to something. I think by installing the new controller it rules out the motherboard/onboard SATA controller, and other than the hard drives themselves (which work fine when connected directly), it doesn't seem like there is anything else.

On another note, the drive performance seems a little low compared to some others. My CPU usage during the write testing is near 100% idle, and memory usage seems to be very low. What would my limiting factor be here? I plan to add RAM at some point after re-configuring my machines in the near future, but until then, it seems that I have more than enough.

I have a Seagate Barracuda 1TB 7200RPM drive installed in this server also, and the performance is pretty close to the WDs (although a bit lower), so I lean towards it not being a disk problem.

Any ideas appreciated!

 

[TravisT]

Sorry, quick update:

Code:

Sep  1 21:16:07 freenas kernel: mps0: IOC Fault 0x40002622, Resetting
Sep  1 21:17:06 freenas kernel: (da0:mps0:0:0:0): SCSI command timeout on device handle 0x000a SMID 657
Sep  1 21:17:06 freenas kernel: (da0:mps0:0:0:0): SCSI command timeout on device handle 0x000a SMID 658
Sep  1 21:17:07 freenas kernel: mps0: IOC Fault 0x40001500, Resetting

I noticed on the 8.3.0-Beta1 release notes it says the mps driver is only compatible with firmware version 13. I have 14 on my LSI/IBM M1015. Should I downgrade that, or is it an issue in the 8.2 Release-p1?

 

[TravisT]

It will take some more testing, but I think I found the problem with the drive bays.

I removed a drive from the drive caddy and inserted it directly into the backplane. My transfer rate for a read came in at 108MB/s. If that solves the problem, I will either have to modify the drive caddies, or deal with supermicro tech support... Either is better than having this problem unsolved.

Bottom line (if my further testing proves true):

Drives were not fully inserted into the backplane connectors, even though they were completely seated when installed into the drive caddies.

 

[cyberjock]

It will take some more testing, but I think I found the problem with the drive bays.

I removed a drive from the drive caddy and inserted it directly into the backplane. My transfer rate for a read came in at 108MB/s. If that solves the problem, I will either have to modify the drive caddies, or deal with supermicro tech support... Either is better than having this problem unsolved.

Bottom line (if my further testing proves true):

Drives were not fully inserted into the backplane connectors, even though they were completely seated when installed into the drive caddies.

Are you sure the caddies are being COMPLETELY inserted? From memory of using them I had an issue with a friend with the caddies. He thought it was pushed in all of the way, swung the arm up and clicked it closed, but it wasn't actually installed. Since the caddy wasn't fully inserted before he swung the arm up it was still out just enough to not work. Is this possibly the case with you?

Have you taken apart the backplane at all? It could be that the backplane isn't mounted fully inside and the backplane moves when you push the hard disk in, hence it doesn't work. But when you can push it in without the caddy holding you back from pushing the hard drive through the backplane you can obviously make a connection.

Just some ideas to think about before you try to RMA(if they will let you) and may be faster and cheaper!

 

[TravisT]

[root@freenas] ~# dd if=/dev/zero of=/dev/da1 bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 800.482245 secs (134136869 bytes/sec)

Although I don't have a solution as to how I should fix this, at least I know what the problem is now. The drive trays don't seem to be adjustable in any way, so I'll wait for an answer back from tech support to see if they can provide any useful information.

 

[TravisT]

Are you sure the caddies are being COMPLETELY inserted? From memory of using them I had an issue with a friend with the caddies. He thought it was pushed in all of the way, swung the arm up and clicked it closed, but it wasn't actually installed. Since the caddy wasn't fully inserted before he swung the arm up it was still out just enough to not work. Is this possibly the case with you?

Have you taken apart the backplane at all? It could be that the backplane isn't mounted fully inside and the backplane moves when you push the hard disk in, hence it doesn't work. But when you can push it in without the caddy holding you back from pushing the hard drive through the backplane you can obviously make a connection.

Just some ideas to think about before you try to RMA(if they will let you) and may be faster and cheaper!

I've had that happen in the past too, but not with these particular caddies. I did check that, and I looked at everything as well as I could without disassembling anything. The caddies are being inserted completely. They can't be pulled out, and typically when they are not inserted completely, they are able to pull out. They do make contact, but it must not be full contact when inserted. I haven't pushed around on the backplanes, but everything looks tight, and there doesn't seem to be a good way to bring the backplanes forward or the drives back so they mate better.

Before I do any major operations, I'll check to see what tech support says. A couple options I have are to get some undrilled rails from supermicro and drill the mounting holes for the hard drives 1-2mm back from where they are now, or to move the backplanes to the other side of the mounting tabs (bringing it closer to the drives). Not sure if either of these are viable options though.

 

[cyberjock]

Have you taken apart the backplane at all? It could be that the backplane isn't mounted fully inside and the backplane moves when you push the hard disk in, hence it doesn't work. But when you can push it in without the caddy holding you back from pushing the hard drive through the backplane you can obviously make a connection.

To clarify what I was trying to say there.. I mean that maybe some screws are lose or missing and allowing stuff that should be stationary to move enough to cause your problem. I never took mine apart, but before I installed it in a computer it looked like it was relatively easy to disassemble if needed. I'm just thinking that maybe a couple of loose/missing screws may be your only problem. In that case a trip to Ace Hardware or something may fix your problem. :)

 

[TravisT]

No, I understood what you were saying - just didn't convey my findings well. I haven't checked specifically to make sure all hardware was tight (backplane, etc.), but I did look at it thoroughly without taking it apart. It all seems tight and together - besides, I'm having this problem on 3 separate backplanes, so I think it's more than just a loose backplane. I plan to check that way before I try to re-invent the wheel though. I'm sure I can figure something out.