There should be a builds section...

[mlcdigital]

It would be cool to have a section here on peoples' builds.

I just put together an i5, 16TB, 20x3TB HDD server rack running FreeBSD.

I'm having kernel crashes I think due to I'm guessing the RAID controller.
I'm using an Areca 12xx configured as JBOD.

It ran stable for 12hrs...

I know there's rules to the amount of drives you should use for RAIDZs but I don't want to follow those rules and loose data.

Check out my read/write scores...
I did a new write to a /mnt/Data/ddtest2 to eliminate reading any data cached in RAM.

Code:

ZRAID3 42.9TB
Write:
dd if=/dev/zero of=/mnt/Data/ddtest bs=32k count=2000000
2000000+0 records in
2000000+0 records out
65536000000 bytes transferred in 99.579738 secs (658125852 bytes/sec)
627.637722015 MB/Sec

Read:
dd if=/mnt/Data/ddtest of=/dev/null bs=32k
2000000+0 records in
2000000+0 records out
65536000000 bytes transferred in 62.411415 secs (1050064317 bytes/sec)
1001.419369698 MB/Sec

ZRAID3 10 35.7TB
Write:
dd if=/dev/zero of=/mnt/Data/ddtest bs=32k count=2000000
2000000+0 records in
2000000+0 records out
65536000000 bytes transferred in 100.659199 secs (651068165 bytes/sec)
620.90698719 MB/Sec

Read:
dd if=/mnt/Data/ddtest of=/dev/null bs=32k 
2000000+0 records in
2000000+0 records out
65536000000 bytes transferred in 51.252652 secs (1278685054 bytes/sec)
1219.449094772 MB/Sec

Did I do this right?...
If I did...
Pretty sick numbers right?...

 

[paleoN]

I know there's rules to the amount of drives you should use for RAIDZs but I don't want to follow those rules and loose data.

What's the pool configuration:

Code:

zpool status -v

Did I do this right?...

You mean like this sticky - [thread=981]Notes on Performance, Benchmarks and Cache.[/thread] Only if you wanted yours to be different from everyone else.

 

[mlcdigital]

What's the pool configuration:

Code:

zpool status -v

You mean like this sticky - [thread=981]Notes on Performance, Benchmarks and Cache.[/thread] Only if you wanted yours to be different from everyone else.

I created 2 different pools to test speed results.
RAIDZ3: was a 20 drive RAIDZ3 with the 4k block size.
RAIDZ3 10: was 2x10 drive RAIDZ3s in the same volume name.

A straight 20 drive RAIDZ3 has slightly faster write results but the RAIDZ3 10 (two RAIDZ3s striped) had significant read performance benefits.

I think I did it right but it was based on this thread which basically has you write a file larger then twice your RAM size then write to a second file so the first file isn't cached then has you read the first file...
http://it.toolbox.com/blogs/database-soup/testing-disk-speed-the-dd-test-31069

 

[mlcdigital]

How do you write enough data to take up 40TiB of space?....

Code:

dd if=/dev/zero of=/mnt/Data/ddtest bs=40960g count=2000000

Or...

Code:

dd if=/dev/zero of=/mnt/Data/ddtest bs=40t count=2000000

What is that count line for because I'm getting errors or erroneous output from the command.
The dd man pages says I can use T or G...?

 

[mlcdigital]

Then I read threads about proper ZRAID drive configurations...

http://forums.freenas.org/archive/index.php/t-158.html

and wonder...

Am I allowed to run a 20 drive ZRAID?...

 

[paleoN]

How do you write enough data to take up 40TiB of space?....

Are you just trying to fill it for some reason? What does bs stand for anyway? If bs=32k & count=1 what total size would that be? If count=33?

The dd man pages says I can use T or G...?

Where? I have trouble believing T would work for bs, but who knows.

Am I allowed to run a 20 drive ZRAID?...

Allowed, sure? Far too wide, yes based on everything I've read. I wouldn't want to try and resliver one of those disks. It would likely take forever if it finished at all.

Start a triple-parity RAIDZ (raidz3) configuration at 9 disks (6+3)
(N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 6

You could do a 7 x disk raidz3 also, but at that point I would just do a 6 x disk raidz2 with 3TB disks.

 

[mlcdigital]

Are you just trying to fill it for some reason? What does bs stand for anyway? If bs=32k & count=1 what total size would that be? If count=33?

Where? I have trouble believing T would work for bs, but who knows.

Allowed, sure? Far too wide, yes based on everything I've read. I wouldn't want to try and resliver one of those disks. It would likely take forever if it finished at all.
You could do a 7 x disk raidz3 also, but at that point I would just do a 6 x disk raidz2 with 3TB disks.

I want to dd a file big enough to fill most of the RAID so I can test performance and stability with the RAIDs duplication of data. I can just use the k or m value to create a 40TiB file but I was trying to use the g or t value.

Far too wide?... Resliver?...
I'm new to the FreeNAS/ZFS/ZRAID world, I understand RAIDs and their function, I understand creating volumes in the ZRAID, but reslivering is something I need to read up on.
I haven't actually lost a FreeNAS RAID before *knock on wood*.

 

[survive]

Hi mlcdigital,

"Too wide" means you have to many drives in the vdev. Ideally you want to stay around 10 drives max per vdev (depending on what type of raidz you use). For best performance you want the number of data drives to divide 128 cleanly. For example, a 6 drive raidz2 has 4 data drives & 2 parity (distributed of course, but for this example we are looking at one single write) so your 128k write gets split up into 4 32k writes to the data drives & 2 parity writes to the parity drives. Each raidz type has some specific combinations of drives that yield the best performance....I can dig around and find the combos if you need, or you can just keep in mind that you want the number of data disks to divide 128 cleanly.

With your "dd" tests the bs stands for "block size" and count is how many blocks to write so if you want to make a monster file you could do something like a million (count) 8m (block size) for an 8TB file (check my math). Assuming "dd" takes a "t" for block size you tried to write out 2000000 of them.

Basically look at the man page and figure out the syntax for writing a reasonably big block, then figure out how many of them you need to make the file size you want. Be sure you have compression turned off on the dataset if you use /dev/zero as your source!

-Will

 

[ProtoSD]

It would be cool to have a section here on peoples' builds.

There already is, it's called the Resources section:

http://forums.freenas.org/forumdisplay.php?24-Resources