HP ProLiant ML350 G6 Server Build

HP ProLiant ML350 G6 Server Build

0773226_hewlett-packard-enterprise-proliant-ml350-gen9-server-5[1].jpeg

New members often wonder if buying a used HP server off newegg.com for $300-$600 is "worth it," why it won't boot off USB, and/or what hardware can go in it and how far the systems can be pushed. This is my journey.

As a note, the G number as in G6, G7, G8, etc. is the Generation. Servers can stand vertically (like a desktop) with a locking front door and feet that mount to the case and make the foot print wider, or in a rack by adding rails and a rack-mount bezel.

The system itself is "pretty quiet" as the fans are variable speed, though if you plan on adding a lot of drives you should plan on adding a lot of fans.

The onboard SAS RAID controller (and optional hardware cache) will not work with ZFS and must be disabled. One or more LSI Host Bus Adapters (inexpensive at around $30) can be added.

The system should be run off of an Uninterruptible Power Supply; it's an industrial-built unit and does not tolerate power issues well.

HP ProLiant ML350 G6 server Hardware List​

Mainboard​
(1) Xenon CPU (usage: 0% idle, 15% under full load)​
48 GB ECC RAM​
(2) 460W Power Supplies in Balanced mode (power draw: 195W idle, 220W under full load)​
(original mainboard suffered a power backfeed and caught fire, replaced)​
Drive Controller​
On-Board HP Smart Array P410i Controller, 256MB, v5.14 (SAS) (disabled in BIOS)
Dell H310 HBA (SAS) (@jgreco) (replaced with larger HBA)
LSI 9300-16i (16 internal port Host Bus Adapter)
Storage​
HP SFF box with 8 SAS hot-swap bays (OEM)​
Boot: (3) HP SAS HDD (300 GB 10.0K RPM (triple mirror)
Data: (4) HP SAS HDD (300 GB 10.5K RPM (worn out; retired/no longer in use)
Snapshots: Seagate SAS HDD (2 TB 7.5K RPM) (not nearly enough storage, will upgrade)
Data​
8-drive RAID-Z3 (24 TB usable storage)​
LFF 5-bay box stuffed with 6TB HGST SAS drives, (1) Noctua NF-P12
LFF 3-bay box stuffed with 6TB HGST SAS drives, (1) Noctua NF-A8
HDD Test Rig​
LFF 16-bay cage​
(new) LSI 9206-16e (16 external port Host Bus Adapter, airflow requirements)
(new) (12) 6TB HGST SAS drives (undergoing burn-in testing)
(3) 120mm fans, Aluminum Mesh Filter Grills (reduces cooling by 1°C)
Independent PSU (current 12V load: 7.5A)​
Other​
(3) HP standard chassis fans, plus 1 extra, 4 total​
Backup: Buffalo TeraStation (x2) (not nearly enough storage, will upgrade)
Offline Backup: Eddie Van Halen drives (also not nearly enough storage, will upgrade)
UPS: APC RT 1500, freshly reloaded with new Pb packs.​
TrueNAS SCALE 22.12.0 (awesome)


NOTES
The on-board HP Smart Array P410i RAID Controller sometimes lost the RAID array if the system unexpectedly lost power (requiring manual reconfig, except for the last time where the array was completely lost and required a restore from backup), plus did not work with TrueNAS. It was replaced with an LSI 8i controller from @jgreco which was later replaced with an LSI 16i due to replacing the aging data HDD array with a larger array.

Replaced the mainboard after a faulty SATA power connector lit the mainboard on FIRE! (gotta love HP...)
  • Somehow the mainboard still ran 'fine' without throwing a fault even though a small section had grenaded itself.
  • The fire took out several cables and 3 SAS drives.
  • One would think the power supply circuitry would have caught the fault, but HP is known for smoking mainboards if the 12V rail gets jumped to the 5V rail.
Added fans to lower 2.5" HDD temps from 45°C to 35°C. (The case is over-stuffed with drives since adding the 3.5" 8-drive array which upset the airflow balance.)

"Usable storage" should only be filled to 75% capacity in order to avoid fragmentation. (It's a ZFS Copy-On-Write thing).

Installing TrueNAS: Getting HP to boot off USB​

The easiest way to do this is use Rufus--very straightforward and fast. Because of HP's BIOS you may one get one shot at burning (writing) the USB stick if the stick is larger than the BIOS likes (due to how data is distributed on a USB stick), so 8GB sticks and under should work fine. If you try booting and it looks like the flash memory of the boot stick is corrupted try reformatting the stick on a different system (like a Windows PC) with a file system (like FAT32) and see if you can write to it; if you can it's HP's BIOS choking, not the stick. (details)

Another option, though more complicated, is to use Ventoy, which allows you to boot Ventoy, at which point the graphical screen comes up allowing you to boot any of the ISOs on the flash drive. (This means you burn the flash drive just once with Ventoy, then dump raw ISO images on the drive at any later point in time, Ventoy automatically finds them and adds them to the boot menu.) Regarding this, a few notes of caution:
  1. Use the Ventoy Advanced option when creating the boot configuration to keep the drive at 30GB or less to avoid HP BIOS issues. (or don't, your mileage may vary)
  2. Run sha256sum.exe (or another checker) and verify the ISO made it to the flash drive correctly. If the HP BIOS does corrupt the load you don't want it corrupting the USB stick and then not knowing if it was the initial write or the boot that corrupted it. (test everything so you know you're working with a "proven" system every step of the way)
  3. Load the ISO into RAM. When TruNAS boots it presents the TruNAS bootloader menu, one option is to load the ISO into RAM; I suggest doing that to avoid issues accessing the USB stick.
  4. Using Ventoy also allows a Copy-Paste of the System Rescue ISO onto the data partition of the USB drive alongside the TrueNAS ISO. Ventoy will automatically ask if you want to boot System Rescue or TrueNAS. System Rescue has a lot of handy tools on it, plus issuing startx starts a Graphical User Interface.
Assuming you hit F8 and
  1. set the RAID controller to remove all drives from all arrays and
  2. turned off read and write caching and
  3. act as a "dumb" Host Buss Adapter that just passes the drives through
you should be okay...probably...until the onboard RAID blocks TrueNAS direct access to the drive, at which point the ZFS array may be lost--use an LSI Host Bus Adapter in IT mode to avoid this (do not use RAID in pass-through mode, it only appears to work but then fails miserably when something goes wrong).

---
If you get that figured out, you can also reserve free space at the end of the USB drive, then create a FAT-32 partition, making:
Partition 1: USB Boot​
Partition 2: USB Data (non-writeable when booting off Partition 1)​
Partition 3: The data partition you created (mountable as read-write when booting off Partition 1)​
It's as easy as:

cd /mnt​
mkdir jumpdrv​
mount /dev/sda3 jumpdrv​
Note the 'a' in sda3 needs to be the actual jump drive letter, easily found with:

startx​
GParted​

Log files can then be written to /mnt/jumpdrv
Dismount with umount /mnt/jumpdrv before shutting down the system.

So yes, you can boot SCALE from USB on an HP server with no problem if you invest a little time to set up the USB thumb drive so the HP server BIOS can read the entire data partition if using Ventoy, if using Rufus use an 8GB or smaller thumb drive. Basically, HP BIOS can only read the first 8GB of the drive, so anything beyond 7.8GB should be slack space (or in my case a FAT-32 partition usable after booting the OS). This has nothing to do with TrueNAS and everything to do with the HP BIOS, although HP systems aren't the only ones that experience that issue (see: Rufus 'killed' my USB Drive).

Boot Device Notes​

TrueNAS uses 1 (or more, if you want) drives for the OS, and does not allow partitioning them so the boot drive can also store data. Some people think this is wasteful, but it's not, it's the easiest way to replace a failed drive and/or restore the OS drive in an emergency. At some point you might thinking, "I'll use the internal USB port on the mainboard with a thumb drive and free up a controller point at the same time." Nah, nah, nah, USB is basically SAS with beer poured over it--it'll work, you just have to realize it's drunk and misbehaves regularly, though from the USB perspective it's totally rational and reliable.*

---
* Possibly not totally accurate depending on your point of view.

Side note: One thing I noticed is installers (or the people who come after them and "upgrade" the server) don't always put the RAM in the right slots--you may wish to check that out first.

Hard Disk Drive Test Rig​

The main failure of HP Servers running Windows Server that I've run into is failing hard drives, even when running HP's RAID management software (a paid upgrade). (The second is HP power supply related issues.)

To test hard drives before putting them in a system I use smartctl to save the S.M.A.R.T. status to a jump drive, badblocks to check if the drive is storing data properly, and smartctl to check how the drive is doing after seven to ten days of badblocks hammering it with data.

LFF 16-bay cage​
(new) LSI 9206-16e (16 external port Host Bus Adapter, airflow requirements)
(new) (12) 6TB HGST SAS drives (undergoing burn-in testing)
(3) 120mm fans, Aluminum Mesh Filter Grills (reduces cooling by 1°C)
Independent PSU (current 12V load: 7.5A)​
(I run System Rescue off a Ventoy USB thumb drive which has a decent toolset, including smartctl.*)​
Notes:
  • Power supplies should not be run without at least a minimal load.
  • Inexpensive (Chinese) fans may have enough inductive feedback that if they're the only thing connected to the PSU the PSU may go into Shutdown Mode; to reset it unplug the PSU from mains power (meaning pull the cord out of the wall), add a failed hard drive as a buffer load (use a junk one so you don't wreck a good one in the event something unexpected happens), be sure the PSU has been unplugged at least 30 seconds, then plug the PSU back into the mains power.
---
* Don't ask how many sockets I've stuck my fob into (it's a bunch).

Power Usage​

HP uses a smart power supply system consisting of one (optionally two) modular power supplies, a power supply control module, power distribution module, the status of which can be monitored through the Integrated Lights-Out (iLO) system (which requires a paid upgrade if using more than the basic features--try it free for 90 days). The key note is total power output in Watts can be monitored remotely, which is very useful when putting a system together.

A note on iLO, since that section hasn't been written yet: iLO only exists on HP; Dell uses DRAC, IBM uses IMM/RSA. An additional HP support license is required to activate the iLO features; once activated iLO advanced features continue to be available even though support has expired (pg.4). (iLO Feature List)​
A note on CPU capacitors: Check the CPU filter capacitors for swelling. The original mainboard caps were starting to look iffy, and through a different issue the mainboard was replaced with new-old-stock.​
Check the mainboard cell battery. The system BIOS settings are lost when it gets weak.​

The Test Rig doesn't have such capability, though it is important to know how much power a drive array and associated fans use as to not overtax the power supply of the system the array will go into. For that I use a clamp-on DC amp meter (UNI-T UT210e, apx. $60). Set it to 200A, push the blue Mode button to switch from AC to DC mode, then put only the yellow power supply wires inside the loop, push the yellow backlight button if you're in the dark (which I frequently feel I am), and read the current in Amps.

Using the 200A range makes it easy to read a stable, useful number. You can switch it to the 2A range to see how many milliamps a hard drive or fan draws, if you hit the Zero button before clamping it on it's accurate to 1 or 2 mA.

It's an [inexpensive] Chinese meter, which is notable from the commonality Chinese grease in general isn't up to the task (mentioned as a factual observation only)*. The switch design coupled with a breakdown of aging grease can lead to inaccuracy in 3 to 5 years (even with virtually no wear observed); replacing the dial grease may rectify the issue.

UNI-T UT210e.png

---
* Given the task, the meter choice is "reasonable." While a Fluke would be more reliable, the cost (about $500) was not judged to be justified in this instance; the Fluke may also have been made in China--though to much higher standards. Note the country of origin is mentioned for informational purposes as knowing about the general grease issue serves to help maintain the equipment, which costs 1/10th as much as equipment made to higher standards.
Author
WI_Hedgehog
Views
2,610
First release
Last update
Rating
5.00 star(s) 1 ratings

More resources from WI_Hedgehog

Latest updates

  1. v0.1 - Data Dump

    As this system has [over-]grown it was moved from Hedgehog's signature to a Resource. Upcoming...

Latest reviews

Very informative for the given hardware
Top