SOLVED Multiple vdev with log and l2arc configuration

[KevinM]

Before I realized that FreeNAS cannot see zpools created on the commandline, I did this:

Code:

dmesg | grep da | grep MB | grep -v transfers | awk '{print $1 "\t" $2}'
ada0:	4096MB
ada1:	3072MB
ada2:	3072MB
ada3:	3072MB
ada4:	3072MB
ada5:	3072MB
ada6:	3072MB
ada7:	3072MB
ada8:	3072MB
ada9:	3072MB
ada10:	3072MB
ada11:	3072MB
ada12:	3072MB
ada13:	3072MB
ada14:	3072MB
da0:	64MB
da1:	64MB
da2:	256MB
da3:	256MB

zpool create nas raidz2 ada1 ada2 ada3 ada4 ada5 ada6 
zpool add nas raidz2 ada8 ada9 ada10 ada11 ada12 ada13
zpool add nas spare ada7 ada14
zpool add nas mirror da0 da1
zpool add nas cache da2 da3

freenas# zpool status
  pool: nas
 state: ONLINE
  scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	nas         ONLINE       0     0     0
	  raidz2-0  ONLINE       0     0     0
	    ada1    ONLINE       0     0     0
	    ada2    ONLINE       0     0     0
	    ada3    ONLINE       0     0     0
	    ada4    ONLINE       0     0     0
	    ada5    ONLINE       0     0     0
	    ada6    ONLINE       0     0     0
	  raidz2-1  ONLINE       0     0     0
	    ada8    ONLINE       0     0     0
	    ada9    ONLINE       0     0     0
	    ada10   ONLINE       0     0     0
	    ada11   ONLINE       0     0     0
	    ada12   ONLINE       0     0     0
	    ada13   ONLINE       0     0     0
	logs
	  mirror-2  ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	cache
	  da2       ONLINE       0     0     0
	  da3       ONLINE       0     0     0
	spares
	  ada7      AVAIL   
	  ada14     AVAIL

I've been trying to duplicate this layout in the GUI without success:

Code:

May  5 21:06:18 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Your disk size must be higher than 2GB]

I've tried making the vdevs and extra disks all at once, making the vdevs first and adding then the cache and l2arc, etc. The system does not want to let me add the log or cache disks. What am I doing wrong?

 

[KevinM]

Recreated the cache and l2arc disks as greater than 2 GB. I was thrown for a bit because zfs itself doesn't seem to care.

 

[titan_rw]

You should be able to use a pool created on the command line. Once it's created, just "zpool export" it, then auto import from the gui.

Being that you're using the entire disk, and not partitions as the gui does, you'll probably run into problems with replacing disks using the gui. You'd have to do any disk replacements from the command line. I think this is the case anyway. Maybe the gui handles pools that have used the entire disk, I'm not sure now.

I'm not sure if the gui will let you create mirrored log. When you check multiple 'log' devices under volume manager I would assume it would stripe the logs into the pool instead of mirroring them.

Also, for your spares, keep in mind they're 'warm' spares. As of the current version of freenas, you will have to initiate the use of any spares in the pool. This will not happen automatically.

If it were me, I wouldn't bother with spares. I'd simply integrate them into the disk vdevs. Either increasing storage, or increasing raidZ level to increase redundancy. In your case that would be 7 drive z2 vdevs (more storage), OR 7 drive z3 vdevs (more redundancy).

Having 'cold' spares in a drawer somewhere might not be a bad idea. Then if a disk does fail, you have one on hand right away for replacement.

Also, this:

zpool add nas mirror da0 da1

I don't think will add mirrored log devices. I think it will stripe in another mirrored vdev into the main storage pool. You'd need "zpool add nas log mirror da0 da1".

I've never done a pool config that complicated. I'd have to play with the gui in a vm or something to see how to make such a pool. I'd probably start by making the single z2 vdev. Then extend the pool with the second z2 vdev. Then I'd probably use the command line for adding cache and log's. The cache devices you could probably add with the gui. But you'll probably need the command line for the mirrored logs.

- - - Updated - - -

I didn't even notice the MB suffix. Yea, freenas needs bigger disks, even for testing. I assume this is a VM for testing, and not for production use.

You can use sparse files for testing zpools with. So the size doesn't really matter until you start writing data to them.

 

[KevinM]

I was able to create the ZLOG mirror in the GUI once I realized I had to make the disks bigger. I'm also going through the freenas 8.3.1 guide and in the Volume Manager section 6.3.3.2 ZFS Extra it says that when two or more log devices are checked, FreeNAS will mirror them. This was because losing the ZIL device on older pools would basically kill the pool. With current software you would lose pending writes, but I would consider that an acceptable risk with mirrored drives.

Also, for your spares, keep in mind they're 'warm' spares. As of the current version of freenas, you will have to initiate the use of any spares in the pool. This will not happen automatically.

If it were me, I wouldn't bother with spares. I'd simply integrate them into the disk vdevs. Either increasing storage, or increasing raidZ level to increase redundancy. In your case that would be 7 drive z2 vdevs (more storage), OR 7 drive z3 vdevs (more redundancy).

Having 'cold' spares in a drawer somewhere might not be a bad idea. Then if a disk does fail, you have one on hand right away for replacement.

Interesting. I'm testing in a VM environment but will be getting a new box shortly. One of my biggest questions is how to carve up the storage. From what I've been reading for RAIDZ2 best practice is 2n+2, which is a bit awkward for 36 drives. Most of the discussion I've seen recommends limiting the size to around 8 drives plus whatever for parity. I am assuming that 2 18-drive RAIDZ2 vdevs would be a bad idea. If 2n+2 is not critical perhaps 3 10+2 RAIDZ2 volumes, or 4 7x2 also in RAIDZ2? This would be a lot simpler if the new box had 40 drives...

Point taken about the hot spares. I might pick up a few extra drives just to throw in a drawer.

 

[cyberjock]

If you have a backup strategy using ZFS replication nightly or rsync or something, then I might try 3 sets of RAIDZ3 of 12 disks, or if you are really crazy, 2 sets of RAIDZ3 with 18 disks.

But, if you aren't doing backups, anything with more than a RAIDZ2 of 8 drives is asking for data loss of the permanent type. :P

Depending on the workload, the 2n+2 isn't always needed. For serving 1GB+ file sizes you won't really care about the "performance penalty" with not having the "correct" number of drives.

 

[titan_rw]

Cyberjock's 3 vdevs of 12 disks in z3 would be best for capacity. That's pretty wide though, especially considering there'd be 3 of them. Capacity would be 27 disks.

You could also do 4 vdevs of 9 disks. Either z2 or z3 depending on your comfort level. Capacity is either 28 disks (z2) or 24 disks (z3). As per the above option, z2 here would be a bit 'wide' here, especially with 4 of them striped together.

With cold spare(s) though, and monitoring of the drives, I'd probably do the 4x 9disk z2 setup, but without backups, I'd only do it for media, or other stuff that could be replaced if lost. For that matter, if it was just for media, cyberjocks 3x 12disk z3 is good too. Keep in mind resilver times will probably be extended by the 12 disk vdevs.

The more 'conventional' setup might be 6 vdevs of 6 drives in z2. Capacity is 24 disks. This uses the 'optimal' number of disks per vdev, and should also be faster due to the fact there's more vdevs to stripe data over. But I'm with cyberjock on the 'optimal' thing. Usually not an issue. zfs is pretty efficient at non-optimal writes, etc. Unless you're doing insane amounts of io, I doubt you'd notice any difference.

If you're looking to expand when needed by adding drives, the 6 vdevs of 6 drives in z2 would be best. You only need to add groups of 6 disks instead of 9, or 12.


I was looking at the various 24 drive 4u chassis. With 24, I'd probably do 3 vdevs of 8 drives in z2. With 36, it definitely does get more complicated.

 

[KevinM]

The issue with backups is that I won't be getting the funds for a second box to rsync to until next fiscal, which for us is July. Until that happens it is going to be a challenge to back this thing up. Right now the goal will be to minimize risk as much as possible.

The setup I have right now in VM is 3 11-drive RAIDZ3 vdevs with 3 spares. Would the consensus be that 6 6-drive RAIDZ2 vdevs be safer? What is the safest configuration leaving at least 24 usable disks? I am assuming that the safest configuration period will be some sort of mirror arrangement leaving 18 usable disks.

Regarding workload, I'm not really sure what it's going to be. It's going to store some Tivoli backups and provide some test/dev iSCSI or NFS LUNs to our VMWare environment. We are also planning on long-term storage for legal documents that we are required to keep for a period of time (I am going to try to hold off on that until we get a second box). There will be the usual assortment of CIFS shares. In short, I'm not really going to know about load until we get the box built and in production.

 

[cyberjock]

If you aren't getting funds for a backup system until July I wouldn't be trying to build the system until July. Backups are no joke and can't be overstated, especially in a business environment. Also, there are potential nasty gremlins waiting for you when you setup your backup schedule and find yourself needing to backup 20+TB of data the first time the task runs. I had problems when I did that myself. It was much smoother to setup backups to run every 12 hours and as you do the initial server fill the backups can keep up. Just think about trying to backup 20+TB across a Gb LAN port. Even if you assume a full 100% utilization you are talking quite a few days. Then things get ugly if you are trying to do a backup nightly and the previous night is still running. Whoops!

There are ways that you can do everything right and still have complete loss of all data on a zpool. For example, bad RAM, corruption of the zpool resulting in the inability to mount(often due to a kernel panic or loss of power to the server), etc. Both of those things happen to forum members pretty regularly(and they seem to always not have backups).

So if you aren't willing to accept the fact that you can do everything right and still lose all data with no warning then you shouldn't be doing anything until you have backups ready to be worked at the same time.

Unfortunately, this post is in memorial to all the forum users that have not listened and made backups a priority. Plenty of blood, sweat, and tears have resulted.

 

[titan_rw]

What is the safest configuration leaving at least 24 usable disks? I am assuming that the safest configuration period will be some sort of mirror arrangement leaving 18 usable disks.

Do you mean splitting 36 disks into mirrored sets or 18 usable? That would be horrible for 'safety'. Each mirrored set is a single parity vdev. Any bad sector during a resilver puts the data integrity of the entire pool in jeopardy. The only thing this would potentially be better for is speed. I imagine that's not the primary concern.

As far as best config for 24 usable drives, there'd be the two that were mentioned:

4x vdevs of 9 disks in z3

or

6x vdevs of 6 disks in z2.

Each config has the same amount of space used for redundancy (33%). Either 3/9, or 2/6.

The z3 setup does give you triple redundancy in each vdev, but the vdevs are proportionally larger. Funny how the math works. One is 4x6, the other is 6x4.

I'm not a statistician, so I can't really comment on which one would be better from a % possibility of failure. I would bet that on average people would recommend the z2 setup, as the vdevs are smaller. It would probably perform better for iscsi loads as well.

Interesting box you're looking to setup. Let us know how it turns out.

 

[cyberjock]

What brand/model disks were you planning to use? I might be able to tell you statistically what is the safest storage... :P

I have a background in mathematics...

 

[KevinM]

What brand/model disks were you planning to use? I might be able to tell you statistically what is the safest storage... :P

I have a background in mathematics...

36 x SEA 3TB CONST ES.2 3.5 SAS 7.2K Mfg#: ST33000650SS

All bought at the same time for extra reliability.