Installing / running from USB drive?

[David Dyer-Bennet]

The server I'm configuring is for amateur video (production) storage; we want maximum space, optimizing for that rather than performance (current projects will reside on the workstations primarily, this is more archival). We want some redundancy to protect us from disk failures. We will of course back up to other systems, servers, cloud, offsite, or whatever seems good at the time (the obvious first destination is a second server just like the first one, which we'll need to do when we get the first server working). We'll be using ZFS, since I'm a big fan (have been running a Solaris-based ZFS server at home since 2006).

The motherboard (Asus M5A78L-M/USB3) has 6 SATA controllers, and our case conveniently mounts 6 drives. We've populated it with 6 x 4TB drives, and I'm currently planning to make a single RAIDZ vdev from them when I get that far, giving us roughly 20TB usable. (Might possibly wimp out and go RAIDZ2, that's obviously safer, and I'm sure a 4TB disk does take quite a while to resilver. But that's not the issue I want to discuss right now.) (So, note that I'm a complete and total noob to FreeNAS and also FreeBSD, but fairly familiar with ZFS administration on Solaris.)

Goal: use all the SATA ports in the box for data disks.

I've downloaded FreeNAS-9.2.1.3-RELEASE-x64.img.xz, uncompressed it with 7zip on a Windows box, and then copied the .img file to the USB drive with win32DiskImager as per the instructions in the manual. It's a 16GB thumb drive, that should be safely more than needed.

I can then insert that USB drive in the server box and boot it. It finds the USB drive, and boots from it, and I see FreeBSD stuff happening, see it find the disk drives, etc. 2.4 says "You then boot into
that device to load the FreeNAS® operating system", but that's not what happens when I try.

First I see a Boot selection with a very short timeout. I think it's maybe partitions on the flash drive? F1, F2, and F5 are the clearly visible options as it goes by. Then there are a bunch of boot log lines, and another short menu with a short timeout that seems to be about booting multiuser vs single user and a couple of other choices. Then it gets to mountroot, waits a while, and gives me a "mountroot>" prompt with some choices. The "?" command says it shows the possible boot devices, and what it gives me is the six disk drives, ada5-ada0; but no hint of the USB key as an option.

In the user manual 1.3.3 seems to say pretty distinctly that what I think I'm trying to do, have my system boot off a USB device, is the preferred mode of operation. But it seems likely I'm not doing it quite right.

How am I supposed to do this? Manual reference is fine, if it's already there and I've just missed it somehow. (I don't have either an optical drive in the server, or a spare SATA port to connect one to even temporarily.)

 

[cyberjock]

So your mountroot prompt is because during bootup something goes horribly wrong and the server doesn't know how to continue. So it drops to the prompt so you can troubleshoot. I can tell you right now that because you are asking about it, you won't be able to use it.

I can tell from that model number your motherboard is an AMD. Many AMD boards aren't compatible with FreeBSD. So you may not be able to fix this without getting other hardware.

What I'd do is try updating your BIOS, make sure all of your BIOS settings are set properly, and make sure your USB stick is plugged into a USB2 port instead of USB3. Other than that, I don't have any recommendations except buy a real man's hardware.. Intel. Ok, I'm joking about it being a "real man's" hardware, but Intel typically works better.

 

[gpsguy]

The motherboard works with FreeNAS. joeschmuck, one of the moderators here, uses one.

How much RAM did you put in it? Hopefully you went with ECC.

In addition to what cyberjock suggested, I'd try booting your flash drive in another system. Just to see if it boots. If it doesn't, I'd reinstall the OS again. Users typically have better luck, getting the ISO version, booting from a CD and installing the OS onto the flash drive. You can even do this in another machine.

btw, your 16Gb flash drive is overkill. We recommend a minimum of 4Gb, but anything larger than that, is just unusable space.

 

[David Dyer-Bennet]

Other than that, I don't have any recommendations except buy a real man's hardware.. Intel. Ok, I'm joking about it being a "real man's" hardware, but Intel typically works better.

Intel typically doesn't support ECC, except on motherboards costing twice this or more, requiring processors costing twice this; hence all my servers being AMD-based for some time.

And I see gpsguy has confirmed this MB does work with FreeBSD.

 

[David Dyer-Bennet]

The motherboard works with FreeNAS. joeschmuck, one of the moderators here, uses one.

Thanks for the confirmation. I did check around a bit before buying this.

What does "install the OS again" mean, in this context? Re-write the .img file onto the flash drive? Or something else? I can't shake the feeling that there may be a step I'm omitting, a step everybody takes so much for granted that nobody ever mentions it (even in the manual).

Is the .img file I downloaded and put onto the flash drive going to be my longterm running image, or is it something I use to perform an installation to something (such as another flash drive)?

Yeah, I know the 16GB is bigger than needed, but at this point we have things like that lying around; might as well use them.

I can, I guess, install from CD onto flash on some other computer. Or...maybe my server box would be able to boot from an external USB optical drive? I can try that.

 

[gpsguy]

Yes please reinstall FreeNAS using a CDROM, it usually works on the first try. A lot of users have problems trying the route you took. You should be able to use your USB ODD.

If it won't boot - see if it will boot in another machine. By the way, plug the flash drive into a USB 2.0 port on your server.


Sent from my phone

 

[SmallGuy]

You have also to checksum the image file you have downloaded and compare the result with the checksums available on the download page.

 

[David Dyer-Bennet]

I assume that's some sort of checksum under the Download button on the Download FreeNAS Release page? But it's the wrong size for an MD5, which is the standard checksum to use for that. It's not labeled in any way, and I can't find anything else on that page saying what sort of checksum it is. Does anybody know?

In 30 some years of transferring files over the Arpanet/Internet, I've never actually had FTP deliver a broken copy when it claimed to complete successfully, so I'm kind of out of the habit of doing such checks. Also the .img.xz file uncompressed without error, which is pretty much impossible if the download was corrupted. Still, nothing wrong with making absolutely sure when I'm having trouble, I'll grant you.

 

[mirdragon]

I had similar issues on my gigabyte board when installing - had to go into bios and enable some legacy boot options as the newer boards are UEFI and this kept kicking in on mine and caused the usb stick not to boot properly.

 

[SmallGuy]

http://doc.freenas.org/index.php/Getting_FreeNAS
http://download.freenas.org/

 

[David Dyer-Bennet]

Okay, sha256. Yep, the .img.xz file checksum matches. And since I'm preparing to try a CD-based install, the .iso file...also checks out. Thanks, I can stop worrying about that (wasn't much, but).

 

[gpsguy]

One other thing ... read this thread, especially message #8 about an annoyance that joeschmuck ran into with the motherboard.

 

[David Dyer-Bennet]

One other thing ... read this thread, especially message #8 about an annoyance that joeschmuck ran into with the motherboard.

Re-boot behavior is certainly strange; not sure what I can do yet, one thing at a time.

People recommend turning off or not using USB3 -- but there's no control over that in the BIOS. Which means the rear USB ports are unusable. And that is an issue with the BIOS mine came with, moving the USB drive to a front USB port which is USB 2 resolved the boot issue.

Also, where in the BIOS is anything about enabling ECC? I'm temporarily working with non-ECC ram because we ordered the wrong ram the first time around (must have gotten crossed up chasing all the valid lists), but in the long run must have ECC. Haven't yet found any ECC settings, I thought I'd looked at everything, but I don't actually remember seeing clocking and such, so there must be more to find somewhere.

(The transcoding issues are side issues at worst for us; I said video, but specifically video production, backing the workstations where current projects are stored. We can't afford 10GB ethernet, so working on remote disk for even very amateurish post-production is just not on.)

I'm about to check if I have the latest BIOS, and record what version I have in any case.

Am I right in thinking that the default setup has taken 2GB from each drive for swap space? Haven't looked closely yet, but it kind of sounds like that. I guess that's tolerable.

Where is the config information stored? I expected it to be on the USB key, but looking at some of the WebGUI settings it kind of looks like it goes in one of the disk pools?

I may *also* try an install from Cd, just for drill; I've got the CD and the usb cd drive up there waiting for me.

 

[cyberjock]

Actually, things aren't all kosher with your board and ECC RAM IMO.

ECC can be:

1. Installed in a motherboard, but not actually *used*(that is.. it acts like non-ECC RAM)
2. Installed in a motherboard, and actually used as ECC RAM.

Looking at your board, notice it says non-ECC RAM...

That to me means you're in category #1. The fact that you said you saw no ECC options in the BIOS made me check this. Because normally no ECC options in the BIOS is a red flag that you don't actually support ECC functions.(now you see why I hate the hell out of AMD)

Now, once you get ECC RAM you can do a dmidecode command that will simulate an ECC failure. But, i'm kind of expecting it to fail based on what I just read.

 

[gpsguy]

Straight from Asus' website ...

4 x DIMM, Max. 32GB, DDR3 2000(O.C.)/1866(O.C.)/1800(O.C.)/1600(O.C.)/1333/1066 MHz ECC, Non-ECC, Un-buffered Memory

It supports ECC RAM and Crucial lists ECC RAM as an option for the board. joeschmuck has one is using ECC RAM with his.

My guess, is that the OP either needs to upgrade the BIOS, or perhaps the ECC options aren't displayed unless he has ECC RAM in the box.

 

[gpsguy]

Yes, by default, it creates a 2Gb swap partition on each drive. While you can change the size, it's not recommended to set it to 0. With today's drive sizes, 2Gb shouldn't be an issue. You should have sufficient RAM, such that the system never has to use swap. If it starts using swap, having it, will probably save your pool.'

The configuration information is stored on the 4th partition of the boot device. You can back it up (or restore it) using the webGUI. Or, create a cronjob to back it up automatically. If you want to do the latter, search the forum for an example.

 

[David Dyer-Bennet]

Actually, things aren't all kosher with your board and ECC RAM IMO.

I think they are. You missed a bit on the ASUS site -- this board supports ECC or non-ECC memory, as I read it. Ah, and I see gpsguy already pointed that out too.

And I did eventually find ECC config in the BIOS; it's hidden under "chipset", which is not a place I looked early on. It does seem to default to disabling ECC; or maybe that's because I'm currently running non-ECC ram. (I screwed up -- we ordered the wrong RAM initially. I'm now running on borrowed non-ECC ram just to make sure everything else works, and to start my learning curve with FreeNAS. We'll be ordering the right RAM probably yet this week, I don't want to go into production without ECC. Maybe I'm overly paranoid, but with the amounts of RAM in things today, and the stated probabilities of errors, seems like non-ECC is a suicidal choice for critical data.

 

[cyberjock]

Yeah.. apparently I was blind. I swear I looked at that thing 3 times.

In any case, the dmidecode command will let you do a test and that will be definitive. You should be able to sleep at night regardless. ;)

 

[David Dyer-Bennet]

Thanks very much to everybody who has contributed here! I've spent the day messing with the server, doing some benchmarking, etc., and things seem to be working very well. Performance isn't bad at all, even with a sub-optimal arrangement like a 6-drive RAIDZ array.

Only one remaining high-priority problem -- which I'll search through Mr. Google and the forum for, and start a new thread for if I don't find anything.