BUILD ASUS P9D WS

[Teva]

I wanted to share my new build, mainly to help others compare different working hardware with what they may be deciding on. Your mileage may vary.

Hardware:
Motherboard: Asus P9D WS
CPU: Intel i3-4130T
RAM: 16gb Kingston ECC RAM (KVR16LE11/8I)
Power: Corsair HX650 Power Supply - Mainly because of the modular plugs
This power supply is also silent with this low power load, the fan doesn't spin (it spins on startup so it works)
SATA: 2 x Syba SD-SA2PEX-2IR SATA II controller cards (~$20 each)
I chose these cards because i'm cheap and knew that this SATA II chip is supported by FreeNAS
Check out some of the reviews on egg for this card, seems to be a very reliable card
Harddrive: 8 x Seagate ST3000DM001 HD's - all in a single RAIDZ3 pool
Boot USB: 2 x Kingston DTSE9H/8GBZ
Case: NZXT Source 210 Elite Black Steel

Usage:
I can easily saturate the gige link. Local tests show from 300-600mb/sec on the server itself. This is being used mainly for a few iSCSI mounts from windows workstations and an ESX host. It also has samba sharing for photos and movies.
I verified via dmidecode that ECC is working in single-bit correction mode.

6 of the Seagate drives are connected to the motherboard, and the other two are individually connected to different Syba SATA II cards.

Hurdles:
Motherboard: The P9D WS doesn't like anything but a video card being put into the first PCI-E x16 slot (Yellow). When I had a network card plugged into that slot the server would randomly not boot and randomly panic'd with SATA or HD errors. Don't plug anything into that slot. The CPU I am using has built in video, and the motherboard has 3 video outs on it, so just pretend that slot isn't there.

SATA Controller: I flashed the Syba cards with the SIS 7.7.0.3 Non-Raid firmware (The b**** firmware), from a DOS or Windows PC. The default firmware is for software RAID, which you don't want for ZFS.
I went against the warnings on the SIS webpage and flashed it to the add-in card anyway. I have been running my old FreeNAS AMD system with 2 Syba cards flashed in this manner without issue for over a year. Once you flash the 7.7.0.3 firmware to it you will no longer be able to use it in windows as the PCI hardware ID's no longer match the drivers, but that isn't an issue for FreeNAS, it see's the cards and attached HD's just fine.
Don't put a SSD's on these cards as they are only SATA II, but they are working fine for me with spinning rust attached.

Thoughts:
To all those who are leaning toward Intel Avatons, stick with the Core-i3/Xeon series. Alot more flexibility, and I think from initial tests that the lowest 4th gen Core-i3 out-performs the top Avaton cpu.

 

[marbus90]

That's how you don't build a FreeNAS...

- that's a Workstation board, not a Server one with IPMI, dedicated video chipset and all slots usable for everything
- that's a low-TDP CPU which just limits the clock speed over standard CPUs which is often needed - and you don't even save that much since the CPU idles for 99% anyway.
- Kingston did enough crap with their hardware that they aren't considered anymore
- overkill on the PSU side
- cheap controllers are definitely not the way to go for a business - grab a LSI 9207-8i or 9211-8i without crossflashing-requirements instead

 

[Teva]

Sorry if I didn't clarify, this is for home use.

- Remote management isn't a requirement for me. I didn't know about the issues with the the PCIe video card slot and didn't feel like RMA'ing it since it has alot of other PCIe slots that I can use. The hurdle is there for others to read.
- I wanted temps to stay low, even under load. This motherboard supports Xeon E3-1200v3 CPUs as well, if I need more umph in the future. I get 100+MB/sec copying files over samba with the 2.9Ghz low-TDP CPU.
- I haven't heard of problems with Kingston. The model of RAM listed has been working for over a year in my old server with no problems so I just went with the same model for my new one.
- The PSU is overkill, but you really can't get a quality PSU in the 200-300 watt range. It is currently pulling around 100watts. I 'm not putting a TedsDiscountPSU in my server. Modular plugs were also a must, I don't want bundles of unused cables stuffed in my case.
- I agree. I just needed a few extra SATA ports and these work for that, I don't need to spend $200 on a SAS controller plus the breakout cables.

 

[cyberjock]

Home use or not, that board still isn't recommended for FreeNAS. The OS and the hardware aren't guaranteed to be compatible, and you may or may not run into fatal problems later. So you may justify it to yourself by saying its for home use. But anyone here with experience knows that your justification is invalid for reasons that are NOT limited to "home use".

I 'm not putting a TedsDiscountPSU in my server.

Ted is crying because you won't buy his stuff.

 

[Ericloewe]

When I was researching my build, Asus Workstation/Server stuff was at least 50% more expensive than Supermicro, too.

 

[danb35]

When I was researching my build, Asus Workstation/Server stuff was at least 50% more expensive than Supermicro, too.

Good point. For comparison's sake, the Asus P9D-WS costs $218 at newegg.com. The SuperMicro X10SLH-F-O, which uses the same Intel chipset and adds IPMI, is $203. The SATA cards are $24 ea., so the Asus motherboard (with 6 SATA ports onboard) plus four more SATA ports costs $266. The SuperMicro X10SL7-F, with 6 SATA and 8 SAS ports onboard, costs $250. Asus doesn't cost 50% more, or even really significantly more, but it's still more expensive.

By its specs, the Asus board should be adequate, but I don't see any compelling (or even interesting) reason to prefer it for a FreeNAS box over the SuperMicro boards which are known to work well. I guess if relative pricing and/or availability are significantly different in other parts of the world?

@Teva, I'm sorry if it seems like we're piling on here. It's just that when you stray from the recommended hardware, support gets harder to find. I hope the build works well for you. If it doesn't, I'm afraid you aren't going to find many others here who are using the same or similar hardware.

 

[cyberjock]

The P9D-WS used to be something like $260 a few months ago.

The problem isn't the specs IMO. Look beneath the specs.

-Asus Dr. Power Switch (is this gonna cause problems for FreeBSD?)
-Audio card (is that gonna cause problems for FreeBSD?)
-What about all the other proprietary goodies that Asus throws on there and doesn't really document (or even *test* on FreeBSD?)

This isn't just about cost. It's about avoiding things that could go very badly for you and your server.

But hey, who the hell am I? I'm just that guy that wrote the recommended hardware thread. Clearly not an expert in the field, so please don't take my advice!

 

[Teva]

Many of the SuperMicro boards have limited expandability. I like having the option to add additional network cards or hba's in the future, i don't want to be stuck in the situation where I'm out of PCIe slots. I also like the ability (no matter how unlikely) to re-purpose it in the future to be an actual workstation if needed. And i do need a new workstation, still rocking an Asus P5Q with a Intel Q9650.

I always disable the unnecessary onboard stuff in the bios like COM ports, LPT ports, Audio, etc...

If I really wanted (afford/justify) a server class system I don't think I would consider a board without redundant power inputs. I haven't seen a consumer level server board that has that ability. Not just for redundant power circuits, but protection against power supply failure on the same circuit. I'd either purchase a system from iXsystems or get a server class system from HP/DELL/IBM.

I do understand that I'll be on my own for support if something breaks, and I'm ok with that. I also checked and you are right that BSD isn't listed as a supported OS for the P9D WS, but BSD support is also not listed for the SuperMicro boards posted in this thread.

 

[cyberjock]

If you read around you'd know that disabling unnecessary stuff not only does NOT save you any power (the hardware still gets power and still uses it when its on), but the hardware can often be detected anyway. Many are "soft switches" that only stop Windows from detecting the device.

So again, I'm sorry, but your arguments for your decisions are easily negated by cold hard facts. Does it suck? Yep. I had spare hardware I wanted to use, and I had to spend $700 for a new CPU, RAM, and motherboard when I did my first build. But that's the cold hard facts and reality of things.

If turning off hardware in the BIOS was the solution we'd have the hardware as recommended hardware. We didn't do that because it doesn't solve the underlying problems.

Also worth noting is that FreeNAS isn't for everyone. It's designed to be an enterprise class operating system, and it expects you to use appropriate hardware for the situation. ZFS itself doesn't scale down to small scale particularly well either. It's really a "you vs the OS" and it quickly becomes a fight between whether you can handle what the OS needs or if another OS is better suited for the tasks you want. We've had plenty of people decide FreeNAS isn't for them. That's fine and I commend them for at least being honest with themselves when they realize they just can't do it properly. Better to use a different OS and hardware that cooperates accordingly than to fight tooth and nail (and potentially lose your data later as a result).

 

[Maturola]

many had already said it, but I would not use that MOBO, go with one from Supermicro or even ASrock and you will even save some cash.

 

[HoneyBadger]

I'd be really interested to see an example of the "soft-switch" you're talking about @cyberjock - I've seen that in laptops where Fn+F2 is a wireless toggle, for example, but at the BIOS level if you shut things off they tend to be completely MIA from an OS/PCI enumeration perspective. Now if the offending hardware in question has no "disable" option in BIOS that's a different kettle of fish.

 

[cyberjock]

@HoneyBadger

I can't remember the last time we diagnosed one. As we've gotten most everyone to realize that if they buy stuff we don't recommend, they'll often have to buy twice to get a working system, most people won't even admit when they buy "joe-blow" motherboard and it doesn't work.

I used to have a board here that did it (it was an Asus socket 1156). But I couldn't remember the model number for my life. There is a reason why I stopped buying Asus long ago

 

[Teva]

I am fully aware that disabling built-in devices does not save power. It does however save IRQ levels. I know this isn't as much/any issue nowadays with MSI interrupts for PCIe but I still do it as a best practice. It will disable the port entirely, from any OS. You can verify this by doing a hardware scan with a linux/unix OS and notice that you can't even see the hardware ID's or IO ports for the disabled devices.
You can read up on the Inception DMA attack with firewire and disabling the device in the bios prevents the attack from occurring. Here's an old article from HP about disabling devices to prevent IRQ conflicts and improve performance: http://www8.hp.com/h20195/v2/GetDocument.aspx?docname=4AA0-2267ENW
Again, not very relevant nowadays but there it is.

My point wasn't to start a great debate I was just posting what hardware is working for me. And before you counter that I may have problems in the future, I accept that and can probably fix them on my own.
Maybe someone that has the same MB but using some other OS is having the same problem I did and google indexes this page and it helps them out.

Feel free to go on an " I told you so" spree if I come back asking for help

 

[marbus90]

wasn't my point either, just that your build shouldn't be considered for someone to buy. as pointed out, there are better solutions around.