SOLVED Root on zfs is still missing

Status
Not open for further replies.

hai

Dabbler
Joined
Apr 4, 2014
Messages
17
Hi list,

I gave 9.3 a try hoping that it is finally capable of being installed on zfs. Well, that is true and it works but not as expected.
On FreeBSD I can create a zpool during installation and place the system on a raidz for example. Not so with FreeNAS. Are there any plans to have that or am I missing something?

Problem is that you have a rock solid NAS with all the beautiful zfs features and then you have to rely on a (potentially faulty) single USB stick. Plus the hassle with booting from USB. Don't know how good that works on other servers than Dell (2950, R710) but it was not usable that way, because you have to make it a fake USB harddrive by BIOS and than it all gets messed up. Only solution: Using a real USB HD.

Apart from that I like that it has Grub now and no more UFS ;-)

Cheers,

Hai
 

hai

Dabbler
Joined
Apr 4, 2014
Messages
17
Ah, thanks. That looks a bit better: "if a reliable boot disk is required, use two identical devices and select them both during the installation. Doing so will create a mirrored boot device." But that is still just a mirror and not raidz or raidz-2. And you might waste (two) HD slots. CF card or SSD would mean buying extra hardware.

Separation is OK but why not this way. During installation you can choose to create a zpool (which is on the same physical harddrives as the data zpool which will be created later on). Or the pool lives on a number of USB sticks or SSD or whatever. Then you'd have all the benefits of zfs for the OS itself and separation.
 
S

sef

Guest
Correct, you cannot (at this time, and I don't particularly have any desire to add it) create a raidz for installation, nor can you install onto an existing pool.
 

hai

Dabbler
Joined
Apr 4, 2014
Messages
17
Ok, I see. So, what would you suggest to prevent the following scenario?

You have a FreeNAS server running in a data center which is miles away. The OS gets messed up (preferably in the middle of a rainy night) because an external USB device or the USB bus or a cable or whatever has (hardware) issues. A reboot leads to an unusable state and a number of VM, important data, you name it won't be available unless you get there and change a bit of hardware.

My own answer:
I think anything with USB is out of question here. That leaves internal devices which are mirrored. Disadvantages: Wasting disk space and slots. Buying internal CFs instead doesn't help as at least a R710 can't deal with two CFs. Mhm, maybe an SSD and a CF if possible.

Don't get me wrong FreeNAS is great but I've been always concerned about the availability of the system due to hardware failures. I know I could use TrueNAS instead with its HA feature. But that means buying an extra server.

hai
 
J

jkh

Guest
This is an easy problem to solve. Stick a couple of SATA DOMs inside the case of that server (if it's in a datacenter, it's racked and it has SATA ports directly on the motherboard) and mirror them. This is what we do. It is rare that SATA doms fail, but even if one does, the mirror will save you from an outage. We have hundreds of systems in datacenters with this configuration and they work just fine, so we know for a fact that this is a good option. This is a solved problem!
 

hai

Dabbler
Joined
Apr 4, 2014
Messages
17
Thanks, that is a good idea. And on top of it you'll have a zfs mirror with 9.3. I am happy now ;-)

Cheers,

hai
 
Joined
Nov 11, 2014
Messages
1,174
This is an easy problem to solve. Stick a couple of SATA DOMs inside the case of that server (if it's in a datacenter, it's racked and it has SATA ports directly on the motherboard) and mirror them. This is what we do. It is rare that SATA doms fail, but even if one does, the mirror will save you from an outage. We have hundreds of systems in datacenters with this configuration and they work just fine, so we know for a fact that this is a good option. This is a solved problem!

This is exactly what I was thinking to do , just instead of DOM put 2 old intel ssd (intel x25m) and install on both for mirrored boot drive for reduddancy, but it got me thinking: in bios I will have to choose 1st boot drive , then since this is software raid if the first drive is corrupted still starts to boot how it will switch to the other one instead, since bios will always push 1st priority drive even it's a bad one . Do I got that wrong ?
 
S

sef

Guest
You should be able to specify a first and second boot device.

The place you will have a problem is if the boot blocks are corrupt, but don't result in an error; the BIOS may not go to the next one. The other problem is if the grub blocks (second step of the boot process) are bad or corrupt, in which case I doubt it'll boot. But you will still be able to boot and have your server running by selecting the second one at that point.

The advantage of the mirrored boot is that your filesystem -- the OS, and configuration database -- are in less risk of data loss -- ZFS itself should protect against corruption (not in preventing it, but in detecting it).
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Well, if you have a BIOS that lets you choose 2 hard drive boot devices, you choose the SSDs as the first two. Note that if the primary disk is detected but is failed the system may still try to boot from it, requiring human intervention to change the boot device to come back online.

Not sure what the problem is aside from a "worst" case of you having to change the boot device in the BIOS before the system comes back online. :P
 
Joined
Nov 11, 2014
Messages
1,174
I have Supermirco X9SCM-F as recommended, and I can have 1st and 2nd boot options to be those 2 ssds , but like you both said we are talking when the drive is not completely gone and trying to boot partialy, so bios will not switch to 2nd boot option and will keep booting from the bad one. I am just trying to make this as reliable as possible. Do you think will be better option if hardware mirror is created with these 2 ssds with the motherboard raid, so it will always try to boot from the that device till one of the mirrors is still good ? Is this will be more reliable option or no ?



P.S. I always save the config file if any changes are made.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'd do ZFS mirrors before hardware RAID. I have the exact same motherboard. :)
 
Joined
Nov 11, 2014
Messages
1,174
Well I got mine base on your recommendation.:):):)
Supermicro SC836 TQ chasis, 32GB RAM and 2 LSI9211-IT
By the way what is you storage raid configuration I still debate how to configure and how many drives to get to make around 24TB-32TB storage without using stripped mirrors ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I'm running 10x6TB is RAIDZ2. The choice of pool and such is very much personal so doing what I'm doing may not be a good idea. Mine is for home storage of movies, pictures, etc. I wouldn't even consider this design if I were going to do anything that involved high I/O such as virtual machines for ESXi.
 
Joined
Nov 11, 2014
Messages
1,174
Funny, this is my exact purpose of storage. No high i/o but decent speed when transferring large files is what I need. Is in it 10 drives too much for single dev ? I am not going to refer you to your own guide , which I also read carefully :) but I was thinking for 8drives in Raidz2 ?

Is it 6 drives in RaidZ2 slower that 6 drives in 2 RaidZ striped ? Which one is more resource demanding ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
10 is the largest RAIDZ2 you should ever do.
 
Joined
Nov 11, 2014
Messages
1,174
10 is the largest RAIDZ2 you should ever do.

I almost forgot to ask you , you have the same X9SCM-F What is the speed you get over 1G network ? I tried Z2 with 6 drives , tried Stripped z1 with 6 drives ,tried 3 mirrors almost any combination and I can always write to the nas with 108mb -110MB/s but When I read from it 87MB/s-97MB/s ?!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I've got 10Gb, and I do 200-400MB/sec depending on various things. Over 1Gb I could pretty much hit at least 110MB/sec both ways on large files.
 
Joined
Nov 11, 2014
Messages
1,174
I planing to put a 10Gb card in in near future , but I should be still able to saturate my 1G both ways , right ?
Since you have x9scm-f, Do you use 82579LM or 82574L adapter ? Also in BIOS only thing I change was enable AES-NI and everything else is default, if i disable power states , turn of this "efficient mode" and have it run at 3.4 GHz all the time will it help ?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I leave things at defaults, and I've used both adapters.
 
Status
Not open for further replies.
Top