SOLVED Review of home server build

[Pliqui]

Hello Community,

Long time lurker. Today I reach out to you to tap the collective knowledge of this site in order to trial by fire my build :D and ask a couple of suggestion regarding the build.

My current setup is:

CPU: Intel(R) Xeon(R) CPU W3550 @ 3.07GHz
Mobo: Lenovo Workstation S20
RAM: 24Gb non-ECC (maxed out)
Case: Fractal Design Define R5
Raid Controller: IBM ServeRaid M5015 (Cross-flashed to LSI 9260-8i)
HDD1: 1 x Samsumg 850 EVO 250gb SSD (Boot device for VMs)
HDD2: 4 x HGST 4TB in Raid5 (1 Datastore)
NIC: 1 x 1gb

I'm running ESXi 6.5 with VMs 10 running. Nothing high performance, but the machines that doesn't boot from the SSD are painfully slow, specially the Windows ones.

I decided to separate the storage from the compute (which someday down the road will be 2 or 3 mini itx servers). One added functionality to the FreeNAS is going to be plex.

Here are the specs of the new build: * Updated to reflect components changes.

CPU: Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz - Changed from E3-1225 V6
Mobo: SUPERMICRO MBD-X11SSL-CF Micro ATX*- Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX
RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)
Case: Fractal Design Node 804
PSU: Seasonic FOCUS Plus Series SSR-650FX 650W
Boot: 1 x SSD (I will use some old one I have around) - Added boot SSD - MiniSAS-to-SATA Mobo
HDD1: 4 x HGST 8TB stripped-mirror (presented over ISCSI to the ESXi server) - MiniSAS-to-SATA Mobo
HDD2: 4 x HGST 4TB (from current server) in Raidz1 (this for media, secondary backup place and non important stuff. A portion will be also presented to the ESXi via ISCSI) - SAS-to-SATA
HDD3: 1 x Intel Solid State Drive DC S3110 128GB stripped for ZLOG - Mobo SATA
SAS Cables: 2 x Supermicro MiniSAS HD to 4x SATA
NIC1: NetApp Chelsio Dual Port SFP+ 10GbE PCIe 111-00603+A0 CC2-S320E-SR 100-1082-00 - Direct connect to ESXi for ISCSI traffic (Ebay)

* Corrected model
Here are some points were I need some guidance:

.- Do a 10Gb NIC (2 actually) would worth the extra cost for the ISCSI direct connect?. In case yes, any cheap dual port 10gbe nic that works with ESXi 6.5 and FreeNAS?
Chelsio is a brand very popular in this forum, I checked the ESXi compatibility guide and looks like these are the only models supported in version 6.5, 6.5u1 and 6.7. The T520-BT, T520-CR, T520-LL-CR, T520-SO-CR which are not cheap.

.- I can't afford a PCIE SSD like the P3700, but I do care about my data, so I wanted to used a battery backup SSD: By stripping the 2 SSD, it will allow me to set the sync=always flag without massive dropping of performance. Any issues with this configuration ?

I do perform backups at several layers (VM and files) and keep a copy outside the array in separate disks.

Please note that no purchase of disks is required, already got them.

Any Suggestions, tip, 2 cents, etc are more than welcome and appreciate it.

Thanks guys.

 

[Inxsible]

I decided to separate the storage from the compute (which someday down the road will be 2 or 3 mini itx servers). One added functionality to the FreeNAS is going to be plex.

So this new build is going to be bare-metal FreeNAS? or still ESXi and FreeNAS as VM ? What will you do with the old non-ECC build? Use it as a VM server?

If I were you, I would stay away from mini-ITX as they don't offer much in terms of expandability

 

[Inxsible]

In case yes, any cheap dual port 10gbe nic that works with ESXi 6.5 and FreeNAS?

Don't know what your definition of cheap is, but Chelsio and Intel XT540 are the 10gbe choices with FreeNAS -- in that order.

CPU: Intel Xeon E3-1225 V6 Kaby Lake 3.3 GHz

Xeon xxx5 have a graphics chip which is pointless for a FreeNAS application -- So I would stick with a Xeon xxx0 series.

RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)

DDR3 based systems might be cheaper to build in the US/Canada because DDR4 is quite expensive. But if budget is not a concern than you can go with DDR4

HBA: The IBM M1015 cross-flashed to IT mode (is a well liked choice over here) or is there any suggestion for the community here?
Boot: 2 x usb 3.0 stick in mirror (honestly haven't look that much here. I had a choice when I started to look for parts, the SanDisk 16GB Cruzer Ultra Fit USB 3.0 130MB/s Nano Flash Pen Thumb Drive CZ43 but is out of stock). Recommended ?

HDD2: 4 x HGST 4TB (from current server) in Raidz1 (this for media, secondary backup place and non important stuff. A portion will be also presented to the ESXi via ISCSI) - Connected to Mobo's SATA

If you are using a HBA for your storage disks, then you will likely have a few unused SATA ports available on the board. Even if you connect 4 disks to the motherboard, you will still have 4 unused SATA ports left. Skip the USB, and use a decent SSD drive as your boot drive. No need for a mirror SSD.

 

[Pliqui]

So this new build is going to be bare-metal FreeNAS? or still ESXi and FreeNAS as VM ? What will you do with the old non-ECC build? Use it as a VM server?

If I were you, I would stay away from mini-ITX as they don't offer much in terms of expandability

This would be a bare-metal build for FreeNAS, I will keep the workstation non-ECC for esxi (I do not run FreeNAS in a VM)

The mini itx would be for the compute only, actually the original build for FreeNAS was mini, but decided to go micro for the ram.

Thanks

 

[Zredwire]

I use ESXI 6.5 but with FreeNAS as a VM and using NFS instead of ISCSI. I would recommend you go with a single drive for your SLOG at first. Depending on the size of your writes two SLOGs striped might not be any faster. You can always add a second SLOG later and stripe it. I would purchase more drives for your striped mirror. You want IOPs but with only two VDEVs striped in your configuration you won't get many. Might be better to have more, smaller drives, like 6 or 8. Also I would not use RAIDZ1 with drives over 1TB (there is plenty of material online as to why).

 

[Zredwire]

Oh as far as 10GB. I used to have my system set up with ESXI and FreeNAS on separate hardware. When I did I used a QLOGIC QLE8152 Dual-Port 10Gbps NIC for ESXI and a Chelsio adapter for FreeNAS. This is the adapter for FreeNAS:
https://www.amazon.com/gp/product/B00O1M9UOA/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

I think the Chelsio cost about $30 and the QLOGIC was $45 (used from Ebay).

 

[Ericloewe]

Several things to point out:

Mobo: SUPERMICRO MBD-X11SSH-F-O Micro ATX

The X11SSM-F is unequivocally better, unless the X11SSH-F is significantly cheaper (30+ bucks).

RAM: 4 x Crucial CT16G4WFD824A 16Gb Ddr4 Ecc Unbuff Cl17 (64Gb maxed out)

If you're going to max out the system immediately, perhaps you should consider something larger.

HBA: The IBM M1015 cross-flashed to IT mode (is a well liked choice over here) or is there any suggestion for the community here?

Cheapest LSI SAS2008, SAS2308 or SAS3008 controller from a reputable vendor. Don't focus on the one model. Since you may be looking at other motherboards, get one with an SAS3008 onboard, if pricing is favorable.

Boot: 2 x usb 3.0 stick in mirror (honestly haven't look that much here. I had a choice when I started to look for parts, the SanDisk 16GB Cruzer Ultra Fit USB 3.0 130MB/s Nano Flash Pen Thumb Drive CZ43 but is out of stock). Recommended ?

No. If you have SATA ports (and you will), use at least one proper SSD for booting. Cheapest you can find new from Crucial, Toshiba/SanDisk/WD, Samsung or Intel.

By stripping the 2 SSD, it will allow me to set the sync=always flag without massive dropping of performance. Any issues with this configuration ?

Higher probability of failure, but otherwise no.

 

[Pliqui]

Don't know what your definition of cheap is, but Chelsio and Intel XT540 are the 10gbe choices with FreeNAS -- in that order.
Xeon xxx5 have a graphics chip which is pointless for a FreeNAS application -- So I would stick with a Xeon xxx0 series.
DDR3 based systems might be cheaper to build in the US/Canada because DDR4 is quite expensive. But if budget is not a concern than you can go with DDR4

If you are using a HBA for your storage disks, then you will likely have a few unused SATA ports available on the board. Even if you connect 4 disks to the motherboard, you will still have 4 unused SATA ports left. Skip the USB, and use a decent SSD drive as your boot drive. No need for a mirror SSD.

.- By cheap 10Gb I meant no more than $100USD.
.- Good catch ! I do not need a video card on the CPU, will update the build with Intel Xeon E3-1230 V6 Kaby Lake 3.5 GHz and get a little bit of extra power there.
.- DDR4 is already on the budget, so no issue here.
.- The reason for going with the HBA is because the case have 8 x 3.5 bays and 2 x 2.5 internal, so the 8 SATA ports of the mobo would not suffice. There will be some free ports of course, but I cannot fill with more HDDs. I have to stay micro atx.
.- I will get a cheap SSD for boot instead of the USB.

Thanks

 

[Ericloewe]

I have to stay micro atx.

Then look at an X11SSL-CF. It has integrated SAS.

 

[Pliqui]

I use ESXI 6.5 but with Freenas as a VM and using NFS instead of ISCSI. I would recommend you go with a single drive for your SLOG at first. Depending on the size of your writes two SLOGs striped might not be any faster. You can always add a second SLOG later and stripe it. I would purchase more drives for your striped mirror. You want IOPs but with only two VDEVs striped in your configuration you won't get many. Might be better to have more, smaller drives, like 6 or 8. Also I would not use RAIDZ1 with drives over 1TB (there is plenty of material online as to why).

Thank you Zredwire, I'll go with 1 SLOG and take it from there. I don't think I can buy more 8TB drives, but going to evaluate if I can squeeze 2 more. About the RAIDZ1, i'm well aware of that is not a recommended at all, but I'm going for space and not protection. This will store media files and non-important stuff, so I don't mind if while replacing a disk the another drive fails.

 

[Pliqui]

Oh as far as 10GB. I used to have my system set up with ESXI and Freenas on separate hardware. When I did I used a QLOGIC QLE8152 Dual-Port 10Gbps NIC for ESXI and a Chelsio adapter for Freenas. This is the adapter for Freenas:
https://www.amazon.com/gp/product/B00O1M9UOA/ref=oh_aui_search_detailpage?ie=UTF8&psc=1

I think the Chelsio cost about $30 and the QLOGIC was $45 (used from Ebay).

You sir ! deserve a cookie. Already are in my cart at Ebay, thank you very much.

 

[Pliqui]

Then look at an X11SSL-CF. It has integrated SAS.

Thank you, looking into the board right now and also checking if the LSI 3008 needs to be cross-flasshed to IT mode or is it ready to go for FreeNAS.

I found so far this https://forums.freenas.org/index.php?threads/lsi-sas-3008-p14-firmware.55169/ - but need to dig deeper.

I have a Supermicro X11SSL-cf Motherboard with an onboard LSI 3008 HBA flashed in IT mode....

Thanks

 

[Ericloewe]

LSI 3008 needs to be cross-flasshed to IT mode

No, it does not.

It probably takes IR firmware, too, but it's just a matter of flashing IT. No need for tricky crossflashes.

 

[Pliqui]

Thank you all for your feedback, I changed the new server list to reflect the changes.

Will tidy up things, still need to check fans and then buy it.

Cheers,

 

[Inxsible]

Mobo: SUPERMICRO MBD-X11SSL-F Micro ATX - Changed from SUPERMICRO MBD-X11SSH-F-O Micro ATX

Did you mean to change to X11SSL-F or X11SSL-CF?

It's the -CF that has onboard SAS ports not the -F. If you are changing to -F , then why not change to X11SSM-F or stay with X11SSH-F?

 

[Pliqui]

Did you mean to change to X11SSL-F or X11SSL-CF?

It's the -CF that has onboard SAS ports not the -F. If you are changing to -F , then why not change to X11SSM-F or stay with X11SSH-F?

It is the -CF, thank you for the catch, corrected it with the right link.

 

[Scorch95]

The X11SSM-F is unequivocally better, unless the X11SSH-F is significantly cheaper (30+ bucks).

Would you mind explaining this? I do not doubt you at all but as I am designing my first system and these two boards are the ones I'm debating between. I liked the idea of the X11SSH-f vs X11SSM-f as it has a m.2 drive built in.


EDIT: I found your post about the reason in another thread explaining why.

The X11SSH-F has two wasted PCIe lanes, which would be used by the extra NICs in the -LN4F. The M.2 slot is crippled and has only two lanes.
This package is nearly always more expensive than an X11SSM-F plus an M.2 adapter.

 

[Ericloewe]

Would you mind explaining this? I do not doubt you at all but as I am designing my first system and these two boards are the ones I'm debating between. I liked the idea of the X11SSH-f vs X11SSM-f as it has a m.2 drive built in.

https://forums.freenas.org/index.php?resources/so-you’ve-decided-to-buy-a-supermicro-x11-board.13/

 

[Pliqui]

Hey Guys,

Thank you very much for the points. Build is on his way (small hiccup with the SAS cables, will update the list parts).

Just another thing, I want to confirm if this 10Gb cable will work 10Gtek for Cisco SFP-H10GB-CU1M,10GBASE-CU Direct Attach Copper Cable, Twinax Cable, Passive, 1-Meter

Thanks,