Checking new HDD's in RAID

[tstorzuk]

Question for the community,

I'm building a new FreeNAS RAID with brand new HDD's. I would like to make sure that all of the drives are going to function properly before moving any data onto them.

Is there some type of program that you would suggest that I could run to perform a stress test on the RAID array before hand? Is there a test that can be run directly from FreeNAS? If so, what would the parameters be to get the best out of it?

I had previously asked this same question;
http://forums.freenas.org/showthread.php?420-RAID-stress-test

I want to check to see if any progress has been made and if this has been implemented in the GUI. If so, where could I find it?

Regards

 

[jgreco]

So you start with a SMART conveyance test. Then you move on to a SMART extended test.

Then move on to what we call burn-in, prior to making any filesystems or anything.

Read all the data off each disk with dd. Write zeros to each entire disk with dd. Re-read all that data off each disk with dd.

Do each set of tests in parallel and watch to see if any of the disks are unnaturally slower than the others, a warning flag.

Then you make your filesystem(s). Then you run iozone in a seek-heavy manner. Then you keep that running a few weeks (no, seriously, weeks is on the short end).

If your system is healthy at the end of that, you've probably done as much as you can to ensure that the hardware is good.

 

[tstorzuk]

jgreco,

Thank you for your post. Very informative. I am going through each part one by one. It's excellent information for a starting point, but from my first look I am already finding more and more questions about how to run these test.

SMART conveynace test. This can/should only be performed if not using a hardware RAID card (correct?). Otherwise the RAID card should be able to run this test (if so I will check my RAID card documentation)? Also, the drives to be tested have to be fully mounted? I think that's what the image in the user guide is showing me (pg. 74 of 258 in the Disks section of the Add S.M.A.R.T. Test window "http://www.freenas.org/images/resources/freenas8.3.1/freenas8.3.1_guide.pdf")(additional info on pg. 216 of 258). I will have to install my FreeNAS onto my new server before being able to look any further into how to make this test run.

SMART extended test. Same as above....?

Burn-in. a.k.a. Heat Soaking. Exercising the hardware to see if it will fail based on a bathtub curve of expected failure rates. So I'm guessing I should go to my drive manufacturer to find out the bathtub curve, and determine an acceptable test length to check the drives. However I didn't find any information on HOW to perform this test. But I will check with my drive manufacturer to see if they have something that they can supply (bootable disc or flash drive image), as well as a suggested time length to run it.

DD Read/Write tests. Excellent write up on Wikipedia on what this does "http://en.wikipedia.org/wiki/Dd_(Unix)". DD, known as Data Description, basically does raw read/writes to the HDD. So it is typically thought as a "Destroy Data" or "Delete Disk" tool. I'll have to read up on the options and switches available for this tool. And I understand why you said to do it right after running the burn-in test. I am pretty certain that I will need help with running this and getting the right parameters for it
(dd if=/dev/zero bs=1024 count=1000000 of=file_1GB
dd if=file_1GB of=/dev/null bs=64k).

Parallel testing. Would this require that the drives be in separate PC's or can this test be run on the same PC? I am going to be using 24 HDD's so being able to complete this test quickly, and all at once would be preferable. What type of test do you perform on your drives to accomplish this?

Make the filesystem. No problem there.

Run Iozone in a seek-heavy manner (http://www.iozone.org/). So, can I run this from within FreeNAS, or would I have to install it onto a stand-alone computer to run it on the networked device? I see that they have a linux, windows and apple versions. It makes no mention of FreeBSD.

Again, thank you so much for your post. Extremely informative. Great information. I'll be able to research more about the suggestions you gave to me.

 

[jgreco]

You probably shouldn't be using a RAID card with FreeNAS. At least, you shouldn't be using the RAID features and you should be able to see the disks directly, instead of as "virtual devices" or "disk volumes" or whatever terminology is in vogue these days. For example, the IBM ServeRAID M1015 is a great card, once you get rid of the RAID firmware and use it as a simple SAS/SATA HBA. Gads, I've gotta write a FAQ about this.

The conveyance test does whatever the drive manufacturer deems necessary for testing after transit of a drive. The extended test probably does a subset of the same thing, but it's a belt-and-suspenders way to just make sure everything is checked. Or if you're Santa Claus, checking it twice.

I was unclear on burn-in; burn-in for practical purposes is everything from the moment the SMART tests are done to the moment you put it in production and start copying data on. You want this period to exercise everything in the system that is likely to fail. Power cycles. Reboots. All the dd tests I suggested. The iozone testing. You keep exercising things to make things fail, until you are confident that the system is solid. You should beat the heck out of the system, because once you have data on it and there's a problem, you're in much deeper doo-doo. You can see if your desired redundancy level can successfully rebuild (because you don't want to find out the hard way that it can't). Good time to experiment.

A quick test, one of the most useful I've found, is the straight dd test. That's where you take something like

# for i in 0 1 2 3 4 5 6 7; do
> dd if=/dev/da${i} of=/dev/null bs=1048576 &
> done

which reads the contents of all the disks. With identical disks, these should all take roughly the same amount of time. You can do the same thing to write zeroes to all the disks. Again, with identical disks, apx. the same time. You can basically do as little or as much testing as you feel comfortable with. Me, I'm getting old, I've seen too much avoidable/stupid/arrrrgh, so I'll let a system run for a very long time if I have ANY doubt at all about it.

 

[tstorzuk]

jgreco,

Thanks for the information about the IBM ServeRAID M1015. However that will not work for my application. My server board only has 2 PCIe ports. My case is a Norco RPC-4224 (24 drives). So 16 drives would not suffice.

However if you have any information on how to bypass the RAID functionality on a 3ware 9550SX-12MI RAID card, I would be fully interested in finding out about that.

Thank you for clarifying what you meant about "burn-in" testing. I was under the impression that I was to physically stress the HDDs to make sure that they were going to function for a longer period of time.

As per the run Iozone in a seek-heavy manner. Should that be disregarded? You made no mention in your last post, or how to properly run it on the new server.

When running the dd test, is that done directly within FreeNAS? From the GUI or use the keyboard and input as root? Exactly how to run it properly?

Again, thank you for all of your help. It is greatly appreciated.

 

[cyberjock]

I have a 9550SE and the only way to "bypass"(I put this in quotes for a reason) the RAID controller features that I could find was to make each hard drive its own RAID array. This is very far from optimal, and if you intend to trust the server with your data I'd highly recommend you get a different controller. The configuration I just describe is the "worst case" scenario for using a RAID controller with ZFS. The drawback with this configuration is ZFS won't have direct disk access(which ZFS assume it does by design) and if your controller fails you won't be able to get back at your data unless you buy the exact same controller(and sometimes the exact same firmware version).

So my advice.. buyer beware... and go looking for another controller. I know, not a good choice considering that using another cards = more money spent. You didn't specify why the M1015 card won't work for your application but I could easily make the argument that your controller won't work for your application either. I'd never let a friend build a server with the 9550SE after seeing how it worked for me.

 

[Stephens]

You didn't specify why the M1015 card won't work for your application but I could easily make the argument that your controller won't work for your application either. I'd never let a friend build a server with the 9550SE after seeing how it worked for me.

His motherboard only has 2 PCIe slots. The M1015 is 8 ports per card, which would give him 16 total. His Norco RPC-4224 supports 24 drives and he'd like to use all of them. Personally, I'd change motherboards, but... *shrug*

 

[tstorzuk]

His motherboard only has 2 PCIe slots. The M1015 is 8 ports per card, which would give him 16 total. His Norco RPC-4224 supports 24 drives and he'd like to use all of them. Personally, I'd change motherboards, but... *shrug*

Stephens,

You are correct about the reason.

It's not just changing the motherboard. It's probably needing to change the CPU's and the RAM. So, it not just a simple modification or just a few extra dollars to get the M1015 cards for my 24 drives.

New server motherboard ~$300
New server CPU's ~$500-$1500
New RAM ~$300-$600
New controller cards ~$300-$600

So it would cost an extra ~$1400-$3000 to swap everything out so that I can run the M1015 controller cards. BTW, I'm not willing to buy used stuff from eBay to save a few bucks.

It will have to wait, the transfer over to the suggested hardware. I'll have to shop around to see if I could possibly use any of my existing hardware. But I would like to be able to start using this server sooner than later.

 

[tstorzuk]

Ok, so I started looking for a MOBO that will allow me to use all 24 HDDs through IBM ServeRAID M1015 controller cards (I would need a minimum of 3 (x8) PCI-E 2.0 interfaces onboard). I only found two that might work out of all the searching that I did. If you know of a better MOBO that will work for my needs (24 HDD's broken out in 6 sets of 4 using SFF-8087 connections).

I am not interested or able to use the onboard SATA connectors, as my case has backplanes with SFF-8087 connections. I'd rather not use a break out cable reversed.

Below is my search infomation, posted for your use as well as for me to easily find this information again wherever I am;


Supermicro Motherboards – Socket 1155 Intel Xeon E3-1200 v2 family with PCI-E 3.0 support. DO NOT choose a Proprietary form factor as it will not fit into your standard case.

X9SAE – (MBD-X9SAE-B)– 1 PCI-32bit, 2 (x4) PCI-E 2.0, 2 (1x) PCI-E 2.0, 1 (x16) PCI-e 3.0
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

X9SAE-V – 1 PCI-32 bit, 2 (x4) PCI-E 2.0, 2 (x1) PCI-E 2.0, 2 (x8) PCI-E 3.0 in x16
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

X9SCM-iiF – (MBD-X9SCM-iiF-B) - 2 (x8) PCI-E 3.0*** slots, 2 (x4) PCI-E 2.0 in x8 slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

THIS ONE IS SUPPOSED TO WORK FINE AS PER PREVIOUS POSTS
X9SCM-F – (MBD-X9SCM-F-B) - 2 (x8) PCI-E 3.0*** slots, 2 (x4) PCI-E 2.0 in x8 slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

CPU: Intel Xeon E3 1230 v2 (8M Cache, 3.3 GHz) ($229.99)

RAM: 4x8GB DDR3, 1600MHz, ECC, CL11, 1.5V, Unbuffered, DIMM, 240-pin ($266.99 for all)

IBM ServeRAID M1015 SAS/SATA Controller (46M0861) - 6 Gbps SAS 2.0 PCI Express 2.0 RAID controller ($115.00 each – 3 required per server)


X9SCM – (MBD-X9SCM-B) - 2 (x8) PCI-E 3.0*** slots, 2 (x4) PCI-E 2.0 in x8 slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

X9SCL+-F – (MBD-X9SCL+-F-B) - 2 (x8) PCI-E 3.0*** in x8 slots, 1 (x4) PCI-E 2.0 in x8 slot
6x SATA2 (3Gbps)

X9SCL-F – (MBD-X9SCL-F-B) - 2 (x8) PCI-E 3.0*** in x8 slots, 1 (x4) PCI-E 2.0 in x8 slot
6x SATA2 (3Gbps)

X9SCL – (MBD-X9SCL-B) - 2 (x8) PCI-E 3.0*** in x8 slots, 1 (x4) PCI-E 2.0 in x8 slot
6x SATA2 (3Gbps)

X9SCA-F – (MBD-X9SCA-F-B) - 1 (x16) PCI-E 3.0***, 2 (x4) PCI-E 2.0 (in x8), 3 PCI 32-bit slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

X9SCA – (MBD-X9SCA-B) - 1 (x16) PCI-E 3.0***, 2 (x4) PCI-E 2.0 (in x8), 3 PCI 32-bit slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

X9SCI-LN4F – (MBD-X9SCI-LN4F-B) - 1 (x16) PCI-E 3.0*** slot, 1 PCI 32-bit slot
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)

X9SCI-LN4 – (MBD-X9SCI-LN4-B) - 1 (x16) PCI-E 3.0*** slot, 1 PCI 32-bit slot
4 SATA2 (3Gbps), 2 SATA3 (6Gbps)


Tyan Motherboards – Socket 1155 Intel Xeon E3-1200 family

S5510 – (S5510GM3NR) - 1 PCI-E Gen.2 x4 slot, 2 PCI-E Gen.3 x8 slot (x8), 1 PCI-E Gen.2/Gen.3 x8 slot
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

S5512 – (S5512GM2NR) - 1 PCI-E Gen.2 x8 slot (x4), 2 PCI-E Gen.2 x1 slots, 1 PCI-E Gen.3 x16 slot (x8), 1 PCI-E x8 Gen.3 slot
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

S5512 – (S5512WGM2NR) - 2 PCI-E Gen.2 x1 slots, 1 PCI-E Gen.3 x16 slot (x8), 1 PCI-E x8 Gen.3 slot
8 SAS, 2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

S5512 – (S5512GM4NR) - 1 PCI-E Gen.2 x8 slot (x4), 2 PCI-E Gen.2 x1 slots, 1 PCI-E Gen.3 x16 slot (x8), 1 PCI-E x8 Gen.3 slot
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)

S5510-LE – (S5510G2NR-LE) - 2 PCI-E Gen.3 x8 slot (x8), 1 PCI-E Gen.3 x8 slot (x4)
6 SATA2 (3Gbps)

S5512-LE – (S5512G2NR-LE) - 2 PCI-E Gen.2 x1 slots, 1 PCI-E Gen.3 x16 slot (x8), 1 PCI-E x8 Gen.3 slot
6 SATA2 (3Gbps)

S5512-HE – (S5512G2NR-HE) - 1 PCI-E Gen.2 x8 slot (x4), 2 PCI-E Gen.2 x1 slots, 1 PCI-E Gen.3 x16 slot (x8), 1 PCI-E x8 Gen.3 slot
2 SATA3 (6Gbps), 4 SATA2 (3Gbps)


Intel Motherboards – Socket 1155 Intel Xeon E-1200 v2 family.

S1200BT – (S1200BTSR & S1200BTLR) - Up to 5 PCI-E 3.0

S1200KP – For embedded applications


IBM ServeRAID M1015 SAS/SATA Controller (46M0831) - 6 Gbps SAS 2.0 PCI Express 2.0 RAID controller

8 internal 6 Gbps SAS/SATA ports (2 x4 Internal SFF-8087 SAS or SATA connectors)
(x8) PCI-Express 2.0

Careful as to which type of bracket it comes with. Lots of them don’t come with any bracket. Otherwise, they typically come with a LOW PROFILE bracket. That’s great if you are putting it into a LOW PROFILE case. Otherwise it’s of no use. You will need a STANDARD PROFILE bracket if you are going to install it into any other type of case.

Amazon – (46M0831) ($217.00) – External Site ($145.00) – Full height bracket included

Amazon – (46M0861) Full height bracket included ($115.00) – External Site

Other part numbers (what is the difference between each part number???);
46M0862
68Y7366
SAS9220-8I
46C8933 – No bracket

Full Height Bracket – on Ebay ($10.00)
How to Cross Flash to IT mode

 

[tstorzuk]

X9SCM-F – (MBD-X9SCM-F-B) - 2 (x8) PCI-E 3.0*** slots, 2 (x4) PCI-E 2.0 in x8 slots
4 SATA2 (3Gbps), 2 SATA3 (6Gbps) ($168.99)

CPU: Intel Xeon E3 1230 v2 (8M Cache, 3.3 GHz) ($229.99-$228.99)

RAM: 4x8GB DDR3, 1600MHz, ECC, CL11, 1.5V, Unbuffered, DIMM, 240-pin ($266.99 for all)

IBM ServeRAID M1015 SAS/SATA Controller (46M0831) - 6 Gbps SAS 2.0 PCI Express 2.0 RAID controller - 8 internal 6 Gbps SAS/SATA ports (2 x4 Internal SFF-8087 SAS or SATA connectors)
(x8) PCI-Express 2.0 ($94.00 - $115.00 - $120.00)

I found a problem with using these combined parts in the Norco RPC-4224 case. The standard 1.64ft/500mm SFF-8087 cables are too short to reach from the backplane to the IBM ServeRAID M1015 SAS/SATA Controllers. You need longer cables. I'm going to try these LSI/3Ware CBL-SFF8087-06M 0.6m SFF-8087 to SFF-8087 Multi-lane Internal SATA Cable. They are 0.6m long (100mm longer than the other ones). Hopefully that is long enough, otherwise I'll have to resort to 36" lengths, which will leave a bunch of extra cable in the way inside the case. So I'll be waiting until ~Wednesday for the new cables to arrive.

 

[tstorzuk]

Here's some photos of the problem with the new IBM ServeRAID M1015 cards. The first photo, of the old RAID cards, shows the connection ports on the front of the board;


Here's a couple of photos of the server backplane, and how they pass through the fan bracket;



And lastly, here's a couple more photos of how far these cables reach. They do go as far as the farthest back connection, but they won't bend and fit into the port. Unfortunately these cards were designed with the ports at the farthest back part instead of at the front like the other cards;

 

[jgreco]

You need longer cables. I'm going to try these LSI/3Ware CBL-SFF8087-06M 0.6m SFF-8087 to SFF-8087 Multi-lane Internal SATA Cable. They are 0.6m long (100mm longer than the other ones). Hopefully that is long enough, otherwise I'll have to resort to 36" lengths, which will leave a bunch of extra cable in the way inside the case. So I'll be waiting until ~Wednesday for the new cables to arrive.

Yes, this is the sort of fun stuff you get when you build in-house. Probably explains why we have such a collection of parts and cables laying around.

Just in case it is helpful, be aware that LSI makes cables in 05M, 06M, 08M, and 10M variants (those are part suffixes not meters, the length is the number divided by 10). So there's a reasonable option for you in the 08M. Also some options for more specialized cables if you spend some time looking around. Since the system is just Molex iPass, one could also probably just order the Molex 79576 cables at any available length, and there are some interesting odd ones like 0.65m, 0.87m, and 0.90m.

 

[tstorzuk]

Yes, I have boxes of extra parts from the years of building my own. And I rely on forums for information about what works with what. Hence the post letting others know that the standard cable lengths won't work with the M1015 cards on any motherboard in a RPC-4224 case with the 120mm fan bracket. They might work with the standard 80mm fan bracket, but I haven't tested that out because I want the higher volume of air flow from my 120mm fans.

Had I know ahead of time, I would have gotten the cables with the rest of the other parts.

Another thing; the Supermicro MBD-X9SCM-F-B motherboard is not able to crossflash the M1015 cards. I have some Intel SE7520AF2 motherboards that were able to crossflash the M1015 cards though.

 

[Brian M]

While doing my own FreeNAS build I did some documentation of the setup that might help some newbies with this particular section of testing HDDs. Posting here because this thread is linked from the documentation.

FreeNAS HDD testing

The following commands assumes SATA drives - which are "ada#" devices in FreeNAS (the most likely type beginners are likely to use) - for other types refer to the smartmontools man pages/website.
It also assumes that this is a freshly setup FreeNAS system if you plan on following step by step.

Determine how many drives are connected by (in the left panel - Storage - View Disks)

to manually run the SMART tests open up Shell

The SMART "conveyance" test is manufacturer determined tests on the drive - to identify possible damage incurred during transportation of the device. This is a good first test to run as suggested by jgreco
so smartctl -t conveyance /dev/ada0
then after it completes (which will take a few minutes - check 10-20 minutes later) use the log command "smartctl -l xselftest /dev/ada0" again.

the SMART Short test checks electrical & mechanical performance - as well as small parts of the drives surface.
then smartctl -t short /dev/ada0 (and repeat for each drive you have 0 through XX)
to see the summarized results smartctl -l xselftest /dev/ada0 (and again repeat for each drive number)
or smartctl -a /dev/ada0 to see the full SMART results for that drive (even with the largest size shell, not all will be visible, so you can use "smartctl -a /dev/ada0 | more" to press the space bar to view each page of the full results.

The SMART Long test scans the entire disk surface - this will take a long time - estimate at least 1 minute per GB. Do this before you go to sleep, and it will probably be done by morning.
to start it: smartctl -t long /dev/ada0 (repeating for each disk)
and then the next morning the same log command: smartctl -l xselftest /dev/ada0


A second recommended way to check on drive performance and do some stress testing is to do a "dd" read of the each of the drives.
From the Shell you can run the following command - note this is to test 6 drives, adjust it if you have fewer or more drives.
# for i in 0 1 2 3 4 5; do (press return it will then start a new line where you continue)
> dd if=/dev/ada${i} of=/dev/null bs=1048576 & (press return again)
> done (after pressing return it will actually run the above commands)
this will take a long time (hundreds of minutes) and as long as you keep the Shell open, when it does complete it will report how long it took for each of the drives, if one is much longer than the others, it could have a performance issue (even if it passes SMART) (when I did this test on 6 drives, I left it overnight and saw the next morning they were all within 10% of each other)

After this I created the Volume, this has to be done before setting up the Automated SMART tests.
(easiest way is in "Storage" then "View Volumes" & add all the volumes - I used ZFS RAID Z2 myself on 6 disks refer to documentation for the best choice for you)

Then Add new SMART test
- can create a short run first - a good idea is to have a short run go every night in the middle of the night when it won't be noticed.
- a long run (which takes quite a while) is recommended once a week or so.

I had to fix permissions on the created filesystem before I could begin copying data to it, I'd missed a step when I first created the share - once data can be copied, then you can run utilities to stress test both read & write on the drives, like the mentioned iozone, or your drive performance test of choice (like HELIOS LanTest for Mac OS X, or AJA System Test - enable network volumes in its preferences)


If a drive seems to fail at any test, test it in another way (swap the drive with another one in the array, or connect it to another computer) to verify it isn't a bad cable or controller.

For anyone curious about my setup, I listened to the suggestions in the forums (thanks cyberjock & jgreco & more) and went with server motherboard, ECC ram, etc (I chose NAS style hard drives as well - a little more expensive than the cheapest typical drive, but also come with a much longer warranty). Total cost was still under a pre-built NAS solution and has way more future expansion possible.
Everything plugged in and worked with no special configuration necessary. After several days of testing - no drives had errors or problems, copied 6+ TB of data over several more days to the FreeNAS with no errors. I have some confidence at least that I don't seem to have any drives DOA or "infant mortality" type issues.

...Hopefully I didn't make any typo's in this post.

 

[anRossi]

I'd like to elaborate on Brian M's post with some of the testing I did as well.

After doing the SMART conveyance and short and long tests (yes, all three), I proceeded to do the dd read test, and then a dd write test. My drives were under 10% of each other in the dd tests.
Then I got fancy.

I see above jgreco talks about really stressing a drive to know if it's going to be stable. So I looked at the iozone tool he mentions.
N.B. I couldn't get my FreeBSD VM to work so I didn't compile a version for freebsd. Instead, I just booted a linux live distro and used the 64-bit linux build I could make. Hence, all my drive names and paths are in linux format. Please substitute with BSD style device names if you do get a BSD version of iozone.

The following are the commands I issued to my drives (which were formatted with ext4) in parallel. The point wasn't to benchmark the drives or the filesystem, but to really give these new drives a work out!

./iozone -I -f /mnt/sdb/testfile -i 0 -i 2 -i 5 -r 128k -s 512G -V 85
Explanation of the options:
-I forces it to write directly to the device immediately. This software seems a little outdated, so without this option all the performance data is completely wrong (I'm talking gigabyte/sec. speeds on mechanical disks). Using this option I got sensible values. (I still want to compare relative performance between drives)
-f specifies the file to write to. Without specifying, iozone will write to a temp file in the current working directory.
-i runs tests 0 (write a file, then rewrite it), 2 (random read of the file, then random write of the file), and then 5 (strided read).
-r set the data transfer size to 128k bytes (the stride was 17 * 128k, so it would skip that many 128k blocks before reading a 128k block again)
-s set the file size to 512G
-V is a decimal value to specify a pattern to write to disk and then verify on reading back. 85 is 0x55, which is an alternating 0 and 1 pattern (in binary).

This test took a while, and didn't stress the drives enough in my opinion, so I thought I'd turn up the heat.


./iozone -C -I -i 0 -i 8 -i 5 -t 2 -F /mnt/sdb/testfile.1 /mnt/sdb/testfile.2 -T -r 1m -s 420G -V 170
-T enables throughput mode. In this mode, multiple threads are each executing the test on their own file, to simulate the kind of environment you might have with a NAS. (I imagine).
-t 2 sets the thread count to 2.
-F specifies the files to be used by the threads in order.
-C tells it to print out the individual throughput per thread as well as the aggregate. This uncovered some interesting information later.
-s now specifies the size of each file. Be careful not to exceed the freespace on your filesystem. I was trying to get as close to full as possible (1TB drives)
-i now adds a new test that's only in throughput mode: 8 mixed reader and writer. This test has half of your threads reading and half writing. There's only 2 threads here, so one for each.
-r sets the block size to 1 megabyte. I didn't want to wait forever, I just wanted a random workload.
-V is still verification mode. 170 is the inverse of the pattern 85: 0xAA. Just want to make sure the data is good while under stress.

This took longer than the first test, but I still wanted to stress the system, so I doubled the number of threads


./iozone -C -I -i 0 -i 8 -i 5 -t 4 -F /mnt/sdb/testfile.1 /mnt/sdb/testfile.2 /mnt/sdb/testfile.3 /mnt/sdb/testfile.4 -T -r 512k -s 212G -V 85
-t4 = 4 threads.
-r 512k half megabyte blocks. I'm getting more patient.
-V 85, back to the previous pattern.


Now that I got patient, I figured, let's crank this up to 11.

./iozone -C -I -i 0 -i 8 -i 2 -t 8 -F /mnt/sdb/testfile.1 /mnt/sdb/testfile.2 /mnt/sdb/testfile.3 /mnt/sdb/testfile.4 /mnt/sdb/testfile.5 /mnt/sdb/testfile.6 /mnt/sdb/testfile.7 /mnt/sdb/testfile.8 -T -r 64k -s 108G -V 170

8 threads, 64k block size. Running in parallel on all 4 drives, this took almost an entire week; 6 days and 15 hours for the last drive to finish. (The fastest drive finished 9 hours earlier; maybe cause for concern.)
the -C option here showed that during the mixed workload (test 8) half of the threads had half the throughput as the other threads. I don't know if this is a drive feature or an OS feature, but it appears that something is favoring writes over reads (or vice versa, I don't have the data to tell).


At this point, I had over a week's worth of testing in. I feel fairly confident that my drives are stable enough to set up my NAS.

IOzone has tons of other options and you can really play with it. For the purposes of stressing a drive, I settled on the above tests to share. I ran other tests, but I don't feel that any were more useful than these.

 

[panz]

./iozone -C -I -i 0 -i 8 -i 2 -t 8 -F /mnt/sdb/testfile.1 /mnt/sdb/testfile.2 /mnt/sdb/testfile.3 /mnt/sdb/testfile.4 /mnt/sdb/testfile.5 /mnt/sdb/testfile.6 /mnt/sdb/testfile.7 /mnt/sdb/testfile.8 -T -r 64k -s 108G -V 170

8 threads, 64k block size. Running in parallel on all 4 drives

I didn't understand "running in parallel"... :(

 

[anRossi]

I didn't understand "running in parallel"... :(

I should have been more clear. In the posts by Brian M and jgreco, they have a loop to start a given command on all drives, so that they run the same thing at the same time.
If you just run a command on a drive and then wait until it finishes before running a command on the next drive, and so on, that would be considered running the commands "serially" or "sequentially". By running the same thing on multiple drives during the same time span, you can consider it "parallel", since the timelines for each drive are now parallel to each other. :)

To repeat the loop used in a generic format:

Code:

for i in 0 1 2 3 ... n; do
<command> /path/to/ada${i} &
done

definitions:
n is the total number of drives you want to test, minus 1.
<command> is a command you want to run on all drives.
/path/to/ada${i} is a sort of placeholder to demonstrate how to properly address a drive. "/path/to/" could be "/dev/" or a folder path, as is the case of my iozone testing.
The ampersand is the important part that makes the command run in the "Background" so that the loop can go on to the next drive. Without the ampersand, this will just run the command on each drive in a serial fashion.

For example, I ran the following command (edited for BSD correctness):

Code:

for i in 0 1 2 3; do
./iozone -C -I -i 0 -i 8 -i 5 -t 2 -F /mnt/ada${i}/testfile.1 /mnt/ada${i}/testfile.2 -T -r 1m -s 420G -V 170 &
done

This assumes that I have formatted my drives, and mounted them in folders named after the drive.
A command I may have run to set up my testing might look like this:

Code:

for i in 0 1 2 3; do
mkdir /mnt/ada${i}
mkfs.ext4 /dev/ada${i} < /bin/y  #(for the record, this is a linux command, since I haven't looked up how to format a filesystem in BSD. Just format a filesystem in this step.)
mount /dev/ada${i} /mnt/ada${i}
done

I hope this helps answer your question. :)


edit: it looks like the code tags add line numbers to my examples. Don't type the "1. " in front of the commands. That's not a part of the command or loop at all.

 

[panz]

For example, I ran the following command (edited for BSD correctness):

Code:

for i in 0 1 2 3 do;
./iozone -C -I -i 0 -i 8 -i 5 -t 2 -F /mnt/ada${i}/testfile.1 /mnt/ada${i}/testfile.2 -T -r 1m -s 420G -V 170 &
done

This assumes that I have formatted my drives, and mounted them in folders named after the drive.

for i in 0 1 2 3 do;

didn't work for me;

for i in 0 1 2 3; do

worked.

I see that you put the path like "/mnt/ada${i}/testfile.1 /mnt/ada${i}/testfile.2" in the middle of the command line. Is there a reason for that?

A command I may have run to set up my testing might look like this:

Code:

for i in 0 1 2 3 do;
mkdir /mnt/ada${i}
mkfs.ext4 /dev/ada${i} < /bin/y  #(for the record, this is a linux command, since I haven't looked up how to format a filesystem in BSD. Just format a filesystem in this step.)
mount /dev/ada${i} /mnt/ada${i}
done

Would it work too if I put /dev/ada${i} (= pointing to the device) instead of addressing the filesystem?

Thank you very much, I'm learning A LOT ;)

 

[anRossi]

I had typoed that semicolon. I have fixed it now. Thanks for catching it :)

I see that you put the path like "/mnt/ada${i}/testfile.1 /mnt/ada${i}/testfile.2" in the middle of the command line. Is there a reason for that?

Those file paths are values for the -F parameter. In throughput mode, ioZone reads/writes to more than one file and ioZone will simply use temp files in the directory where it is invoked from unless specified. My command specifies them since, most likely, ioZone is stored on a flash drive. An alternative would be to omit the -F parameter and its values and simply cd into the root of the filesystem and invoke iozone with a full path. e.g.
cd /mnt/ada${i}
/path/to/iozone <parameters>

Would it work too if I put /dev/ada${i} (= pointing to the device) instead of addressing the filesystem?

iozone is created as a benchmark for filesystems, not of drives, so unfortunately you can't give it just a device name. It can only run tests on a filesystem since it reads/writes to files it creates. You definitely have to address a filesystem. I recommend reading how to use the newfs BSD command to format a UFS filesystem. The defaults should be good enough for these purposes. I believe you can format an entire disk with a filesystem, without slicing or partitioning it. (At least I could on linux)

 

[panz]

The thing I'm not understanding is if I could run Iozone on a ZFS pool, like:

iozone -C -I -i 0 -i 8 -i 5 -t 1 -F /mnt/mypool/testfile.1 -T -r 1m -s 420G -V 170

where "mypool" is a RAIDZ2 array of six disks.