So my eyes are just about bleeding from trawling through various online forums trying to solve this problem and I've come up with nothing. These are still quite handy little boxes, so I'll try to keep updated for anyone else facing the same issues in the future, but if anyone does have anything to add then please don't be shy. I currently have 2x N40L, 1x N36L (Gen7) and 1x Gen8 Microservers so I can test things as required without having too much hassle. My end objective here is to make a capable file server for use with my workstation. There are a number of things about FreeNAS that I don't fully understand and I'm limited by the hardware that I have, but I think a lot of those questions are better placed on other threads. For anyone wondering about this hardware with FreeNAS though, I hope this helps.
Online Resources:
There are a number of owner's clubs/forums online, I cannot really recommend one over the other. There is also an online wiki. Neither the wiki nor the forums seem to be very well thought out out and all seem pretty difficult to search - most have lierally hundreds of pages of people arguing about which $5 gpu is best for use with WHS and things like that. Mostly I think they are just for people who are excited that they have spent money on something and would like the world to know that they have a new toy. There is some good info in there but be prepared to spend a long time coming up with not much at all. Just to make things more fun, lots of members seem to be members of every club and mirror their posts on every website that will take them. The wiki has less of this problem but it doesn't seem to be much more than a jumbled collection of hyperlinks, often leading to online forums in which you're supposed to read through several pages of user comments to gleam several lines of information from. I found the compatible hardware/software/os information very helpful but that's about it...
BIOS Mods:
Several "step-by-step" guides on the web that also link you to online forums and pages of user comments. From what I can gather, these could all be distilled into a paragraph and 5 hyperlinks but perhaps people enjoy simplicity like they used to. Anyway, for anyone wondering there seem to be 3 BIOS currently available:
The Original (2011)
The Modded (2011)
The N54L Update (2013)
The Modded 2011 BIOS is the one that is fully unlocked, meaning that it has the most adjustable settings available. Installation is simple, download the BIOS you wish to install along with the appropriate USB formatting application from HP - the 2011 BIOS's both use the same but the 2013 BIOS requires the newer. Run the software to format a USB with the BIOS flashing software and then replace the BIOS file on the newly configured USB stick with the BIOS file that you wish to install and then boot the motherboard naked except for the usb, keyboard and monitor. The flashing process is automated and you will be given the cli "C:>" prompt once it has completed. Simply reboot and remove the usb once this appears.The entire process takes about 5mins. If you wish to roll back to the original HP Bios then simply run the software to config your USB again and do not replace the BIOS file that it puts onto your USB with a different one.
RAM:
Gen7's - I'm currently running 2x 4GB Unregistered ECC modules. Online reports claim all sorts of RAM configs but this seems to be the optimum. 2x 8gb seems to have varied results, even in the same motherboard. Sometimes it will see 8GB, sometimes it will see 16, sometimes it will see 16 and only let you use 8. If you love machines that are unpredictable then by all means, do as you will but I'm far less forgiving. Since I'm hoping to run ZFS in 2017 the weak links on these machines are the CPU and RAM capacity - you can easily fit more HDD space into these boxes than you will be able to properly control through ZFS.
Important - The Gen7 microservers are AMD based and as such, are not that highly recommended for ZFS as there is no way to verify that ECC is actually working as ECC. I'm happy to take that risk for my own home server but if you are using these in a commercial environment then spending a bit more on the outlay may give you peace of mind.
Gen8 - Max I have running is 16gb unregistered ECC, 2x 8GB.
Cooling:
Case - the single main fan can be swapped with a little bit of rewiring but it won't make much difference to overall temps or noise.
CPU - The internal USB header is perfect for wiring tiny fans up, not really required on the gen7's but in the gen8 this gives you the option of upgrading to a much more powerful cpu.
I live in Australia, so ambient temps can get well over 40C in the shade. More fans, more fans, more fans....
HDD's:
Ah, the rabbit hole...
Gen7's - These can fit massive amounts of storage in a tiny box. My record so far is 9x 3.5" drives internally mounted but I've yet to install my pico power supply, so let's see... With 10tb drives out now you can almost certainly build a super array that will never succeed a rebuild!
Gen8 - These have much less room in them but an extra drive or two can be shoe-horned in.
Networking:
Once you run out of RAM to play with, networking is the next glass ceiling...
Gen7's - 1x onboard 1GbE NIC and 1x optional add-in card for the ILO. Depending on your hardware config you may be able to add networking capabilities via PCIe.
Gen8 - 2x onboard 1GbE NICs plus built in ILO. Sadly these boxes only have a single 16x PCIe slot but it allows for possible networking upgrades, depending on your config.
HBA's & RAID Cards:
This is the bit that I'm having trouble with myself...
Gen7's - So far I have successfully used the HighPoint RocketRAID 620 2x SATA HBA, although I found access to the SATA ports annoyingly tight. I'd recommend the longest SATA cables you have if you intend to try these.
I've also purchased an IBM ServeRAID M1015/LSI 9220-8i and flashed it to the LSI 9211-8i firmware, following many online recommendations. This has not worked out for me as I am unable to boot from USB with the card installed.
The HP P410 is kinda interesting, in that it has no hardware compatibility issues and is currently available on ebay for less than $20AU including international shipping but as far as I'm aware it's a poor choice for FreeNAS. If anyone can prove me wrong on that then I'll gladly buy one...
Gen8 - HighPoint RocketRAID 620 2x SATA HBA tested and working.
IBM ServeRAID M1015/LSI 9220-8i (flashed w/ LSI 9211-8i FW) tested and working.
Note - Both Gen7 & Gen8 onboard SAS controllers work well with FreeNAS, there is only a need to add in an HBA or RAID card if you intend to use more than the 4 hot-swappable bays on each. If you don't need the extra ports then I'd recommend a networking upgrade instead. It's also worth noting that with the Gen7's, even with the modded BIOS, the internal motherboard SATA connector is unreliable. I cannot recommend using this port as part of any array. I've not yet trialed the Gen8's, nor the eSATA port on the Gen7s.
L2ARC/ZIL
From what I understand of these, they are overkill in these small systems. Overkill as in wasted money and ports, not "better safe than sorry". A Zil is not the same as a vanilla SSD Cache and will not provide the same spine tingling thrills that a hybrid array, or an SSD Caching Raid card might. Quite simply, don't get one unless it's been advised as a solution to a problem you're having... L2ARC seems to be more useful (in general) but is NOT a substitute for ARC (RAM). I think the rule of thumb is 1:5, so with a max of 16GB in any of these machines you're really just looking at wasting an SSD and a Port...
Misc.
System vDev - Both Gen7 & Gen8 Microservers have an internal USB header. If you're not using these to power anything then it's a great place to install a flash drive. In the Gen8 there is also a microSD slot, Ive successfully installed to this and even managed to mirror it with a usb drive, but I'm not sure how much of a good idea this is. I'd greatly like to hear from anybody who does know. I realise that FreeNAS doesn't write endlessly but at this point I can only recommend using the back two USB ports to mirror your system vdev. This allows the front usb ports to be used by peripherals and limits the chance of anyone wandering off with your USB.
Hardware - Obviously you can let your imagination run wild but since for me these are best utilised as file storage servers or torrent boxes, I'm going to leave it there. By adding graphics cards and so on you'll pretty much double the power usage and I personally don't enjoy listening to hard drives while I watch TV...
Software Ect - For me, finding the knife edge of these machines is to balance between massive amounts of hard drive space Vs reasonable amounts of ECC RAM. You can run all sorts of VMs and Jails and so forth but all you're doing is taking using up RAM that is better utilised for storage. People crow on and on about performance speeds and all of that but unless you've got some networking upgrades that I should know about then it's all just hot air. Sure, I could strap a Ferrari engine to my bicycle, but why? WHY?? As for overclocking and so on, I can understand the desire to get bang for your buck but you can still easily spend over $1000 for a 4bay NAS that isn't half as capable as one of these machines, so messing with RAM timings and so on is just asking for troubles that you don't need - these machines cost less now than a good night out. I previously owned an N54L and sold it because the performance increase over the N36L was marginal, yet the resale value was almost twice as high, if the prestige of a few MHz is worth it to you then go for it. Be aware though that these are peasant boxes as far as FreeNAS goes, lots of people on these forums use much, much more capable machines...
Anyway I hope that helps anyone else who can't bring themselves to throw away old hardware. I'll keep adding info as I go but I'm happy for anyone else to chip in - just please try to keep it FreeNAS, ZFS or at least NAS oriented - there are a million and one forum threads you can go and stir up about how to set up plex to transcode 4k video to your smart fridge, or how to make an easily hackable web page on WHS. You will get nothing but hostility from me, Cheers.
PS: My to do list for these is:
- Pico PSU Mod, allowing possible HDD additions.
- Networking upgrade, either higher speed interfaces or SMB Multichannel (Not quite sure where this is up to in FreeNAS)
- Find optimum maximums for FreeNAS.
- Possibly convert one or more Gen7 to drive enclosure to be used by Gen8.
