Suspect 1 of 10 Disks Slow In RAIDZ2

[MrVining]

I've been a member for years but only recently started using FreeNAS again. I plan to build a system for FreeNAS but for now it's running on a bit of a Frankensteined server.

(2) E5-2620 v2 CPUs, 32 GB Registered ECC RAM, a (2) 10GB NICs & (2) 1GB NICs, and luckily IPMI on the SuperMicro server board.

I added (10) 2TB HDDs and configured them as a RAIDZ2. I am getting extremely poor performance. Even when I go to the Shell and run zpool scrub tank, then zpool status tank a minute after... I'm seeing rock star performance upwards of 20-30 M/s...

There is little other traffic on this pool and it is the only one in the system. The CPU graph in the GUI is rarely able to be distinguished from the X access.

I suspect that ONE of the drives are to blame, but I'm not finding a way to test performance of ONE disk in the pool. I was trying to find a way of measuring latency of each disk while a scrub occurs, but coming up empty handed in my search results.

Anyone have an idea of how to do this?

 

[MrVining]

Oh, and I installed the OS on a 16GB Cruzer thumb drive I had laying around. No SAS HBAs on this sytem, but on the next I'll probably go with some form of LSI x.x.x -16i advice on that would be helpful, but I have yet to research that.

 

[wreedps]

Are there LEDs on the drives? I am able to tell that way sometimes which one is acting funny.

 

[Bidule0hm]

You can use diskinfo to test each drive: diskinfo -t adaX (replace X with the number of the drive)

But I have the feeling one drive is failing so maybe just seeing the SMART infos will show the culprit.

 

[Apollo]

Scrub is not a good indicator about drive thoughput. If you have many, many, many snapshots and no data modification across those snapshots, then you CPU and drives are not going to show high numbers.
I wrote a small program a few weeks ago that will create a file of a certain size, and while it runs, you can monitor individual disk performance using gstat or Freenas GUI report.

 

[MrVining]

It's a new install, there are no snap shots.

@Bidule0hm That command is epic. One of the drives was reading at 7M/s. Really all of them are total junk My plan was to replace them with 6T WD Purple drives or something like that as they die.

I shut down, connected an LSI 9211-4i and (3) 500G Samsung 850s and those babies read around 500M/s using that command. I thought I was out of the woods... I used those three drives to make a RAIDZ pool. Via NFS, that pool was just as bad as the 10 spindle RAIDZ2. There is one warning "Firmware version 5 does not match driver version 20 for /dev/mps0". Clearly the firmware is way out of date on that LSI card, but the drives were able to reed at 500M using that test so I'm not convinced that is the issue.

I'm considering doing some tests with Solaris as that's what I've used in the past. The whole point of me trying FreeNAS is that I don't want to learn... This is for a networking lab, very few VMs will be running off of it. I'm going to try iSCSI and see if that makes a difference. Side note. The (3) disk RAIDZ scrubbed at 50M/s, in Solaris it never falls below 1G/s that's a pretty big difference I'm thinking FreeNAS just does scrubs differently.

 

[jgreco]

You really don't want WD Purple drives. They're made for surveillance video use and handle errors differently.

 

[MrVining]

Can you suggest a different drive for somewhat heavy use.

Also I believe my issue with performance may be related to vSphere. I have more testing to do.

 

[Fuganater]

Use NAS grade or Enterprise grade drives. WD Red, Seagate NAS, WD SE, WD RE etc.

 

[jgreco]

You can't get 1G/s out of a 3 disk RAIDZ even on Solaris. The fastest non-SSD drives around top out shy of 250MBytes/sec and even assuming it was all large file and you could read perfectly sequential data you would be capped at 750MB/sec.

 

[jgreco]

Can you suggest a different drive for somewhat heavy use.

Also I believe my issue with performance may be related to vSphere. I have more testing to do.

Depends. Lots of small files or random block accesses tend to benefit from faster RPM/lower latency "enterprise" class drives (and lots of RAM and L2ARC) whereas the NAS class drives (WD Red) are good at storing large amounts of sequential data.

I'd note that SSD has been decimating the enterprise drive segment, so options there are not as robust as they once were.

 

[MrVining]

Yes you can get 1G Byte per sec pretty easily with 3-4 SSDs that read at 500 MB per sec. And my 5 spindle desktop class RAIDZ would scrub around 500 MB/s with VMs running on it (very light IO VMs such as Ubuntu Teamspeak server) in Solaris. Kind of a moot point, clearly I have another issue at play. I have just updated firmware on my HBA, we'll see how that goes.