BUILD Proposed FreeNAS build

Status
Not open for further replies.

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
I'm gonna go RAIDZ1; I've got all Enterprise Grade drives and I'll put in 1 or 2 of those 8GB archiving drives for backups, so I should be fine.
How are you expecting the drive(s) to be connected? Realize that UFS is no longer supported so you will have to go with ZFS and it's not ideal but it's better than nothing I guess. I maintain a backup of my important files on separate media. It's a manual process but that is acceptable to me and DVDs are not likely to erase themselves by accident.
 

Arwen

MVP
Joined
May 17, 2014
Messages
3,611
Well, he could use sSATA and ZFS, which is what I did for my 8TB single disk backup.
The eSATA enclosure I use has removable disk slots, without sleds, so it's a bit easier
to put in and remove the SMR / Archive disk for backups.

Eventually I may buy a second 8TB SMR / Archive so they can be rotated for backups.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
SandForce? No, S3500 and S3700 use the same controller hardware (not necessarily firmware, though).
You are totally right! I read it in a comparison table somewhere, but I cannot seem to find that anymore. So a 100GB S3700 it is now.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
RPM it nice to know these days unless you are concerned about a sensor requiring a specific minimum value.
It seems like my main board has expectations of a minimum RPM on the fans. It oscillates between fan alarm (LED blinking, all fans full speed) and normal operation in ~3 second intervals. Once FreeNAS is running, it stops doing that, probably taking these hardware interrupts from the BIOS. So I figure these are BIOS routines doing this, but I cannot find any configuration or diagnostics menu in the BIOS setup.

I now have one Noctua NF-F12 industrialPPC-2000 installed for testing. Perhaps I should get a Noctua NF-F12 industrialPPC-3000, which probably runs at a faster minimum speed.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Your problem sounds weird, but should be fixable with the procedure listed in the Supermicro X10 FAQ.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
Your problem sounds weird, but should be fixable with the procedure listed in the Supermicro X10 FAQ.
Thank you for the suggestion! I am now at work, so I have only glanced at the FAQ, but I can already see the similarities; My temperatures are reported N/A as well. I have no problems reflashing the ME, but I'm a bit surprised this may be necessary, because this board is brand-spanking-new. I'll start working on it tonight. Thanks again.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Thank you for the suggestion! I am now at work, so I have only glanced at the FAQ, but I can already see the similarities; My temperatures are reported N/A as well. I have no problems reflashing the ME, but I'm a bit surprised this may be necessary, because this board is brand-spanking-new. I'll start working on it tonight. Thanks again.
I was actually talking about the IPMI thresholds, but if some of the system sensors are borked, ensuring everything is properly updated and flashed might be a good idea.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
I was actually talking about the IPMI thresholds, but if some of the system sensors are borked, ensuring everything is properly updated and flashed might be a good idea.
Ah; I see. IPMI thresholds was actually what I was looking for, but I got side-tracked to flashing ME because of the N/A behind all sensor read-outs.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
This mainboard is starting to piss me off. I know SuperMicro has a good reputation, but this is not Enterprise Grade worthy:
1. All IPMI sensors show "N/A" and "Not present". Resetting and power-cycling was not the solution. If it were, I would have returned the board for being unreliable. Reflashing may help, but I'm not going there: If that's the solution, there's a manufacturing problem.
2. SuperMicro IPMI tools require me to give them my name, email address an probably a pint of blood. Not of this day and age.
3. Downloading IPMI tools is at 50kB/s at most. Are you kidding me? Modem download speeds?!
4. When they're finally downloaded, they turn out not to work on 64-bit Windows. I don't even know people who are still running 32-bit on workstations, let alone servers, but if that's a real limit, tell me before downloading!

So now I'm downloading the Linux version and the command line tools; They're trickling in while we rant...
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
This mainboard is starting to piss me off. I know SuperMicro has a good reputation, but this is not Enterprise Grade worthy:
1. All IPMI sensors show "N/A" and "Not present". Resetting and power-cycling was not the solution. If it were, I would have returned the board for being unreliable. Reflashing may help, but I'm not going there: If that's the solution, there's a manufacturing problem.
2. SuperMicro IPMI tools require me to give them my name, email address an probably a pint of blood. Not of this day and age.
3. Downloading IPMI tools is at 50kB/s at most. Are you kidding me? Modem download speeds?!
4. When they're finally downloaded, they turn out not to work on 64-bit Windows. I don't even know people who are still running 32-bit on workstations, let alone servers, but if that's a real limit, tell me before downloading!

So now I'm downloading the Linux version and the command line tools; They're trickling in while we rant...
You don't really need all of that, ipmitool or ipmiutil work fine. The BIOS flash can be done locally from DOS.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
Just re-flashed IPMI firmware with the latest version, the same that was already installed. Restored SDR and factory settings in the process, but nothing changed.
Code:
192.168.72.216 (S5/G2) 23:36 ASPD_T>ipmi sensor
Getting SDR data ...
Getting sensors ...
  Status | (#)Sensor                |      Reading | Low Limit | High Limit |
  ------ | ---------                |      ------- | --------- | ---------- |
         | (4) CPU Temp             |          N/A |       N/A |   95C/203F |
         | (71) System Temp         |          N/A |   -5C/23F |   80C/176F |
         | (138) Peripheral Temp    |          N/A |   -5C/23F |   80C/176F |
         | (205) MB_10G Temp        |          N/A |   -5C/23F |   95C/203F |
         | (272) DIMMA1 Temp        |          N/A |    4C/39F |   80C/176F |
         | (339) DIMMA2 Temp        |          N/A |    4C/39F |   80C/176F |
         | (406) DIMMB1 Temp        |          N/A |    4C/39F |   80C/176F |
         | (473) DIMMB2 Temp        |          N/A |    4C/39F |   80C/176F |
         | (540) FAN1               |          N/A |   700 RPM |  25300 RPM |
         | (607) FAN2               |          N/A |   700 RPM |  25300 RPM |
         | (674) FAN3               |          N/A |   700 RPM |  25300 RPM |
         | (741) VCCP               |          N/A |    1.57 V |     2.02 V |
         | (808) VDIMM              |          N/A |    1.04 V |     1.34 V |
         | (875) 12V                |          N/A |   10.78 V |    12.96 V |
         | (942) 5VCC               |          N/A |    4.48 V |     5.39 V |
         | (1009) 3.3VCC            |          N/A |    2.95 V |     3.55 V |
         | (1076) VBAT              |          N/A |    2.56 V |     3.73 V |
         | (1143) 5V Dual           |          N/A |    4.48 V |     5.37 V |
         | (1210) 3.3V AUX          |          N/A |    2.95 V |     3.55 V |
         | (1277) Chassis Intru     |          N/A |       N/A |        N/A |

The web interface reports "N/A" under every sensor's status and "Not Present!" under every sensor's reading. Note the exclamation mark: Now it's shouting at me.
When I press the 'Shown Thresholds'-button, extra columns appear, containing only "N/A" on every line in every column.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
At this point, it's best to talk with Supermicro support.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
At this point, it's best to talk with Supermicro support.
Yup, done that. No reply yet since yesterday, but I'll keep you guys posted.
(On your avatar; Were you smashing that guitar on a SuperMicro X10 with faulty IPMI firmware?)
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
(On your avatar; Were you smashing that guitar on a SuperMicro X10 with faulty IPMI firmware?)
Not faulty, just not-very-sane defaults that caused headache-inducing constant spinup and down. It gets really painful with Noctua NF-F12 IndustrialPPC 3000PWMs constantly spinning up and down.
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
Not faulty, just not-very-sane defaults that caused headache-inducing constant spinup and down. It gets really painful with Noctua NF-F12 IndustrialPPC 3000PWMs constantly spinning up and down.
Pff; I'd better get myself a guitar then, because those are exactly the fans I've got here. Does it matter what make or type of guitar I smash on it? Can you recommend one for best results?
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Pff; I'd better get myself a guitar then, because those are exactly the fans I've got here. Does it matter what make or type of guitar I smash on it? Can you recommend one for best results?
Doesn't really matter. Just smash it on an anvil or a rock (don't want Supermicro to deny an RMA because "you smashed a guitar on the server!").
 

MindBender

Explorer
Joined
Oct 12, 2015
Messages
67
Ok, I've got it working. Thanks to SuperMicro's business class support desk. It turns out that just powering up the main board isn't good enough. The board needs to be booted with an OS before the sensors are available. I think that would make a nice addition to the FAQ ;-).
Anyway, thanks the the instructions in the FAQ I was able to set the sensor limits to better values for my fans and now my fans are spinning happily.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
Ok, I've got it working. Thanks to SuperMicro's business class support desk. It turns out that just powering up the main board isn't good enough. The board needs to be booted with an OS before the sensors are available. I think that would make a nice addition to the FAQ ;-).
Anyway, thanks the the instructions in the FAQ I was able to set the sensor limits to better values for my fans and now my fans are spinning happily.
What? That's the craziest thing I've ever heard about IPMI - and IPMI is a collection of things that make you go "Holy crap, this is hacky, potentially insecure, way too obscure to be properly understood and just overall crazy."

Could you post what Supermicro said, exactly? Not doubting you, but I'd like to see what they said, exactly, to try to figure out where to go from there.
 

Something

Explorer
Joined
Jan 31, 2015
Messages
93
What? Samsung is one of the few manufacturers of DRAM that matter (along with NAND flash and processors), along with Micron and Hynix. They're tier 1.
This, Samsung and Crucial/Micron are the best of the best for DRAM.

Seasonic is good. A G-450 will work well for 6 drives (and a bit more). A G-550 is slightly more future-proof. Beyond that, I'd step up to an X-Series/Platinum.
To clarify, Seasonic isn't just good, their designs and product quality is by far exceptional. Among tier 1 PSUs, they stand above all others.

The X-series/Platinum I can vouch for.

Boot disk
For previous customers I have done extensive research on NAND flash reliability and especially power-fail robustness, all the way down to flash translations layers remapping bad erase blocks and levelling wear evenly across the NAND flash device, and doing that concurrently. They just cannot seem to get cross-page corruptions in MLC NANDs right, or at least not concurrently across a power failure. The conclusion was that there are no devices doing that 100% reliable, which was to be expected, but also that there are very, very few devices doing a pretty decent job. The investigation covered both SD-Cards and USB flash drives. I haven’t done any research on SSDs, but I have had a couple during the past few years and all failed undetected until it was too late. These drives include an OWC Mercury Extreme PCIe card, an OZC RevoDrive X2 PCIe card and multiple OCZ Vertex drives.
The Mainboard I have chosen does have an M.2 slot and again I’m tempted to put an SSD in. Selecting a PCIe V3.0 module is especially attractive, because that will make all 6 SATA connections on the board available for drives. Unfortunately the only PCIe M.2 module I could find is a €300 Samsung 950 Pro of 512GiB. That seems like a bit overkill. For that kind of money I’d rather have a smaller SLC based drive.
Rebooting will be a VERY, VERY rare occurence or should be. Stick to 2 cheap USB sticks. Sandisk Ultra Fit 16/32GBs go for $16-24.

SSDs are more for Jails, SLOG and l2ARC. To use it for a boot device is well beyond overkill and a waste of PCIe lanes, SATA Ports, m.2 ports and/or PCIe ports.

You guys have been recommending specific fans all over this forum, but I cannot remember exactly which one, so it's hard search for forum for that. Can anybody tell me what 120mm fans are recommended?
Is it perhaps the Noctua NF-F12 industrialPPC-2000 IP67 PWM 120mm? My board can handle 4-pin fans, so I'd like to use that feature. And I'm in doubt between the 2000 and the 3000 RPM version. My case will take 2 of these, so no shortage in airflow, but if the low-end is similar on both fans, I prefer the extra headroom of the 3000 RPM version.
If you need the cooling power of Noctua Industrial fans (awesome as they are, i'm a huge fan of Noctua!) you're doing something VERY wrong.

You shouldn't need that kind of performance or warranty guarantees.

I would consider that to be a pretty sane reason if we were talking about the big, chunky "thumb" drives of yore, but have you looked at the recommended drives such as SanDisk Fit/Ultra Fit? The non-USB plug portion is tiny - they don't stick out.
If you can manage to knock a Sandisk Ultra Fit out of a USB slot, you have MUCH bigger problems to worry about OP.

I now have one Noctua NF-F12 industrialPPC-2000 installed for testing. Perhaps I should get a Noctua NF-F12 industrialPPC-3000, which probably runs at a faster minimum speed.
A fan controller can easily fix that for $10.

This mainboard is starting to piss me off. I know SuperMicro has a good reputation, but this is not Enterprise Grade worthy:
1. All IPMI sensors show "N/A" and "Not present". Resetting and power-cycling was not the solution. If it were, I would have returned the board for being unreliable. Reflashing may help, but I'm not going there: If that's the solution, there's a manufacturing problem.
2. SuperMicro IPMI tools require me to give them my name, email address an probably a pint of blood. Not of this day and age.
3. Downloading IPMI tools is at 50kB/s at most. Are you kidding me? Modem download speeds?!
4. When they're finally downloaded, they turn out not to work on 64-bit Windows. I don't even know people who are still running 32-bit on workstations, let alone servers, but if that's a real limit, tell me before downloading!

So now I'm downloading the Linux version and the command line tools; They're trickling in while we rant...
LIke EricLoewe said, these are some REALLY strange problems...

1) Reinstall the firmware me thinks, and/or call up SuperMicro.
2) I never had that issue getting them from the FTP...
3) I downloaded them in probably 5 seconds from their FTP. Dunno how that happened to you.
4) Well it's best to use it as a bootable, burn it to a CD or spare USB stick for usage.

What? That's the craziest thing I've ever heard about IPMI - and IPMI is a collection of things that make you go "Holy crap, this is hacky, potentially insecure, way too obscure to be properly understood and just overall crazy."

Could you post what Supermicro said, exactly? Not doubting you, but I'd like to see what they said, exactly, to try to figure out where to go from there.
He who isn't impressed by IPMI after usage of it, is unimpressed by life.

Your quote was my initial response to IPMI. After using it I wouldn't give it up for the world. It's just so incredibly useful.
 
Last edited:

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
While not related to FreeNAS, I've built some VMware hosts before using Supermicro mini-ITX boards. The same issue cropped up... the fans cycling from slow to full speed. There is apparently a hard-coded minimum RPM expected for each fan input... dropping below this speed causes the MB to think the fan has died, so it spins every fan up to full speed - including the one that actually isn't dying. The MB then sees that all are running wide-open, so the failed state clears and the fans slow back down... below that minimum threshold. The process repeats ad nauseum.

Your options are either a fan fast enough to make the board happy, or an external fan controller.
 
Status
Not open for further replies.
Top