How to build FreeNAS 9.10?

Status
Not open for further replies.

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Finally getting around to trying out the FreeNAS build stuff again.

Looking through some of the source code, an interesting option shows up:


So, if we set the "CHECKOUT_SHALLOW" environment variable before "make checkout", then the checkout should be a LOT faster. eg:

# setenv CHECKOUT_SHALLOW 1
# make bootstrap-pkgs && make checkout

Tried it out here, and yep, that seems to work ok. Instead of taking ~20 minutes, the checkout finished in under 5.

No idea if the build will go ok... that's still running. :)
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Finally getting around to trying out the FreeNAS build stuff again.

Looking through some of the source code, an interesting option shows up:


So, if we set the "CHECKOUT_SHALLOW" environment variable before "make checkout", then the checkout should be a LOT faster. eg:

# setenv CHECKOUT_SHALLOW 1
# make bootstrap-pkgs && make checkout

Tried it out here, and yep, that seems to work ok. Instead of taking ~20 minutes, the checkout finished in under 5.

No idea if the build will go ok... that's still running. :)
Interesting... will try this next time I build.

@JustinClift, I saw your pull request re: FreeBSD versions. I can corroborate your statements:
  • FreeNAS 9.10 can be built with FreeBSD 10.3-RELEASE or later.
  • FreeNAS 10 can only be built with FreeBSD 11.0-RELEASE or later.
I've been building both on FreeBSD 11 just about every day for the last week or so.

I believe that the 12GB RAM requirement specified in the github docs is too low. But what do I know? :rolleyes:
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Cool. :)

Yeah, I'm not sure what to do with the RAM requirement. Tried building in a VM with 8GB RAM and 2GB swap (forgot to change the swap!), which failed. Now trying it with 8GB RAM and 22GB swap. Not expecting a speedy result, but as long as it works I'll be ok with it. ;)

I could probably try with a 8GB RAM + 8GB swap VM afterwards, to see if 16GB total succeeds. If even that fails I'm of a mind to bump up that RAM requirement statement a fair bunch. :)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Cool. :)

Yeah, I'm not sure what to do with the RAM requirement. Tried building in a VM with 8GB RAM and 2GB swap (forgot to change the swap!), which failed. Now trying it with 8GB RAM and 22GB swap. Not expecting a speedy result, but as long as it works I'll be ok with it. ;)

I could probably try with a 8GB RAM + 8GB swap VM afterwards, to see if 16GB total succeeds. If even that fails I'm of a mind to bump up that RAM requirement statement a fair bunch. :)
FWIW, I've tweaked my FreeBSD 11.0 VM with these settings:
  • 2 vCPU
  • 12GB RAM
  • 16GB swap
...for a total of 28GB of memory, which works fine. I'm curious to see how 16GB will work for you. The catch is that an initial build - starting from a 'clean slate' - requires far more resources than later, incremental builds. :D
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Just checked the progress of the build job (8GB RAM + 22GB swap). It completed fine... total time taken (single threaded using POUDRIERE_JOBS 1) was 14:53:08. eg nearly 15 hours. Ouch. ;)
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Just checked the progress of the build job (8GB RAM + 22GB swap). It completed fine... total time taken (single threaded using POUDRIERE_JOBS 1) was 14:53:08. eg nearly 15 hours. Ouch. ;)
Yikes!
The thing to do is: iteratively reconfigure FreeBSD with less and less total memory until a 'new' build fails. That would establish the lower bound memory requirement for a build system.

Or just say that a reasonable build system needs >= 28 GB memory and be done with it! :rolleyes:
 

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
...total time taken (single threaded using POUDRIERE_JOBS 1) was 14:53:08. eg nearly 15 hours. Ouch. ;)
That's in line with my single-threaded build (running in a VM on a Zotac EI750, which has a Core i7-4770R in it).
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
The thing to do is: iteratively reconfigure FreeBSD with less and less total memory until a 'new' build fails. That would establish the lower bound memory requirement for a build system.

Had an idea for a different approach, as it's in line with something else I've been meaning to try. ;)

Kind of thinking that if some kind of (low resource usage) host monitoring agent runs in the VM as the build is happening, it should be possible to get a decent idea of the maximum mem/swap/cpu (etc) used during the build process.

eg cacti, zabbix, statd or whatever

I have an outstanding task to become familiar with Prometheus anyway, so I'm setting up a server with that first (as simply as possible), to send metrics to. Then I'll run the build again and we can find out if these kind of metrics do turn out useful for establishing guidelines.
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
As a data point, running the build in the same VM (4 cpu, 8GB RAM + 22GB swap) without setting POUDRIERE_JOBS 1 also worked. Total time for the "make release" step was around 9 hours. The swap usage didn't seem to be overboard, I only saw it get to ~4GB of swap at one point (manually checking occasionally).

The monitoring of the system stats using Prometheus was a failure though. It can measure the host cpu, and load metrics, but I haven't yet figured out how to retrieve swap usage metrics from the agent. It may be the "node exporter" agent doesn't capture that yet, as it seems to be very Linux focused. *BSD support doesn't seem well rounded (so far).
FreeNAS 10 build - Prometheus 1.png

Might try it again using Zabbix (far more mature atm).
 
Last edited:

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Kicked off another run in the same VM again, this time using 4 cpu, 11GB RAM + 22GB swap. The 11GB was because I wasn't using the desktop at the same time, so could allocate the ram to the VM.

Build time (not single threaded) was 7:34:12. eg ~7 1/2 hours. That extra bit of ram does seem to help.

Also noticed something interesting during the previous build. This series of recent VM's has been created using ZFS. "top" shows a total ARC size of around 1.5GB at times during the build.

For compiling FreeNAS 10, using UFS may be better. That 1.5GB of ARC might be more useful as general system memory during the compile process.

Will probably try a run with that overnight and see how it goes. :)
 
Last edited:

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Quick status update, though there's not a "finished" result/conclusion yet.

Recreated the VM last night, giving it 4 cpu's, 65GB of space on root (/), and 8GB of swap. Used UFS instead of ZFS so ARC wouldn't be grabbing memory.

Allocated 12GB of ram to the VM, which was probably too much. VMware Fusion didn't complain, but the (host) system (my desktop) acted weirdly. Suspecting it may have caused too much contention on the host, so I'd better stick with 10-11GB or so instead from here on.

UFS vs ZFS though... not a clear result yet due to something strange. Over the last few days when using ZFS I'd allocated the VM 50GB of space (not including swap) to build in. No issues with space. Last night though, with 65GB of space (as UFS), the build failed. The VM ran out of space.

Not sure if it's due to the UFS change, or if the code update in FreeNAS itself caused it. I'm suspecting it was the large change in FreeNAS code base, but not sure.

Some graphs (from Zabbix server installed last night too), showing the stats which seem relevant. The missing stats chunk from about 10:30-11:45 was when the host filesystem the VM is on also ran out of space. ;) VMware paused the VM and popped up a warning which I didn't notice until I came back.

As expected, CPU is pretty much 100% for all of the compile phases.
CPU utilisation.png

This shows free disk space. Starts out ~60GB, and continues into the ground/failure. ;)
Free disk space.png

This shows the amount of free swap space. With 12GB of RAM, and 8GB of swap, the compile process (in this failed run) didn't get very far into swap. Not even 1 GB used. A full (successful) run may be different.
Free swap space.png

Memory usage. There seems to be only one significant section of memory pressure, ~08:00am to ~08:45am. That timeframe corresponds to the only significant swap usage period in the graph above. At a rough guess, it's probably during either the buildworld or buildkernel phase, rather than during the lengthy port building which comes after.

If we want to reduce memory usage, we could probably look into environment variables affecting buildworld/buildkernel. Maybe something to limit the # of threads there, similar to what POUDRIERE_JOBS does for port building.
Memory usage.png

Network traffic. The initial download of packages and source near the start, and not much after that:
Network traffic - em0.png

Tonight I'll try the whole run again, with maybe 100GB free disk (UFS), 11GB RAM, and 4 cpus. Hopefully that runs to completion ok. ;)
 
Last edited:

Durkatlon

Patron
Joined
Aug 19, 2011
Messages
414
Super-interesting stuff! I have also noticed that the amount of disk space required fluctuates. Like the ports build, it probably depends on which component builds end up running in parallel. In general it looks like the build cleans up after itself, such that if you do a "du" on the build directory after it completes, the size is not horrible. But during the build it has periods where it seems to snowball out of control.
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Ahhh, hadn't thought of that. Yeah, the disk space usage might really be that random. We should be able to figure out some reasonable upper bounds though. :)

As a first thought for controlling the amount of memory needed, this is a concept patch to override the number of buildworld threads that get started:

https://github.com/justinclift/freenas-infiniband/commit/45fcf6516a6e5e9a9ac92be443acdeef9cd4909c

My Python is fairly rusty though, so it's probably not yet quite right. I'll try it out probably tomorrow and see if it makes a noticeable difference.
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Build process failed again last night, due to a stuff up with the source code on my part. Will have another crack at it again tonight. ;)
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Successful run (finally) last night. ;)

The VM had 10.5GB RAM, 4 cpus, 8GB swap, and 100GB filesystem space (UFS).

The complete build process took about 8 hours. Max swap usage at any time was just over 2GB, so the general recommendation of ~12GB RAM doesn't seem too far off.

Swap space. Two significant periods of swap usage. No idea why the 2nd one happened though, as the memory usage graph below (2nd one) doesn't show a corresponding hit.
Free swap space.png


Free memory.png

CPU utilisation. Nothing looks super interesting here:
CPU utilisation.png

CPU load. I think the buildworld and buildkernel are the flat topped sections to the left, with the port building being the rough sections afterwards. Haven't investigated to confirm though.
CPU load.png


Free disk space (below) is interesting. This shows disk space used (on UFS) to be close to 90GB. I'm not sure if there's a cleanup process which would otherwise trigger earlier if disk space is getting tight. If not, then the recommendations should probably include something about having 100GB of space free too. ;) I'll try ZFS again, and see if the disk space noticeably changes.

Disk space free.png
 

JustinClift

Patron
Joined
Apr 24, 2016
Messages
287
Time for a final set of screenshots, and some sizing conclusions. ;)

New FreeNAS 10 compile run, on a bare metal system this time:

* Intel i7-920 (yeah, old box) - quad core 2.66Ghz, with hyperthreading enabled (so 8 logical cores).
* 12GB RAM
* Crucial MX200 512GB SSD. Configured as 32GB swap, and the rest as ZFS.

This was significantly quicker, taking just ~4 hours. There were a couple of periods of swap use, but only to a maximum of ~3GB swap. So, combined memory use seems to be 15-16GB with 8 logical cores.

Free swap space.png


Free memory.png

CPU utilisation. Nothing super interesting:
CPU utilisation.png

CPU load. Again, not much interesting:
CPU load.png

Seems to have only used ~45-50GB of space, which is very similar to previous ZFS runs. Unlike with UFS, where it takes ~100GB.
Free disk space.png


After all this I'm thinking:

* A reasonable rule of thumb for memory sizing might be: 10GB + 800MB per logical core.
* For a 4 core system (eg a quad with hyperthreading off), that would be 13.2GB ram, which seems about right.
* For a 8 core system (eg a quad with hyperthreading on), that's 16.4GB ram, which also seems reasonably close to the mark.

* For suitable disk sizing, choose ZFS if at all possible, and have at least 50-60GB free. If for some reason UFS needs to be used, ensure 100GB free.

If no-one objects, I'll add those as rules of thumb to the guide on the wiki (maybe this weekend).

Of course, if I've missed or forgotten something (etc), please let me know. :)
 
Last edited:

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
If no-one objects, I'll add those as rules of thumb to the guide on the wiki (maybe this weekend).
You'll hear no objections from me! Outstanding job, @JustinClift.
 
Status
Not open for further replies.
Top