My shy build - require some help

[zwf]

Hi All,

I am reading all the posts and doc since a few weeks and my eyes are starting to bleed :)
I am a software person and last time i really play with parts in a server/computer was when i was building my own PC back when 486 was a thing :) so please don't laugh :)

My friend work at a recycling company and his house (well his computer room) is like a shop lol
So last sunday i decide to bring sushi and get the afternon over there.... this guy have way to much stock for the time he have lol.
Nothing new and shiny but still good stuff i think....
Strangely no server MB only consumer sadly so i will need to buy that but i found some instresting part i think ....

All that are free for me (well cost of a nice sushi lunch and i eat as well :)), Keeping the budget low help to get approval form the boss as well (my wife)

• CPU : Intel Xeon X5675
• Memory : Samsung DDR3 RDIMM 8GB - M393B1K70CH0-YH9 (I think they are ECC) (12 of them)
• SAS card, he had, many was old but i noted 2 of them :
  • Sun 375-3536-04 R50 StorageTek 8-Port SAS PCI-E Server Raid Controller Card
  • Sun 375-3701-01 8 PORT 6GPS SAS-2 PORT RAID PCIE + BATTERY LOW PROFILE
• Bunch of 1TB Disk i think i have 10 or 12 of them

I would like to start with that let say 6 x 1TB disk and upgrade later .
My use is not big for so a couple TB would be fine to start with, but i think i will upgarde when opportunity come
My use will be for Backup the house, no streaming etc.... just File Server. and we do not have ton of data....

After reading some more thread i think i would like for case :
Fractal Design 804 or Thermaltake Core V21
Or anything you guys can suggest in the same range....

So my issue is the MB even after reading for hours i cant decide :( for something where the part i have can fit.
so decide to write here to see if the gurus can suggest MB and the other small stuff i would need.

Thanks in advance :)

 

[rvassar]

The problem with those two PCIe SAS controllers... They're RAID, which you do not want with ZFS at all. The bigger problem, they're second-sourced by a company that for all intents and purposes, hasn't existed since 2011. So it becomes a case of figuring out if you can flash one to IT mode or not.

The 375-3536-01 is I believe an Adaptec board, used by StorageTek, which was acquired by Sun, and then Oracle. Adaptec got swallowed up via a chain of acquisitions into Microchip Technology, Inc. Not sure how much support you'll find in FreeNAS for it. Also, this board has external facing SAS connectors, so you'd be looking at a secondary disk shelf.

Oracle Datasheet

The 375-3701-01 appears to be an LSI2108 chip set. You might be able to find a way to flash this one.

Oracle Datasheet

 

[Pitfrr]

Considering the 12 RAM modules, I'd suggest a dual CPU board.
I'm not sure you'll find a single CPU board with 12 DIMM slots?
And you'll need to get a second X5675....

You could also have a look to some second hand servers (without CPU, RAM and hard disks), you could probably get it quite cheap.
I'm using similar CPU on a HP DL380G6 for example. You just need to get an extra HBA.
And I would directly start with 8 disks... why 6? :) If you add an HBA with 8 ports, then I don't see why you shouldn't go with 8 drives (if you have them anyway).

And if you go with a dual X5675 and 96GB of RAM... then you could think about virtualizing FreeNAS? Since, as you said, you're not going to have and extensive use of FreeNAS, then you can dedicate 32GB to FreeNAS and still have plenty to use for other VM.
But virtualizing FreeNAS is a bit touchy (well, you have to be careful before going that road and you have to do it right) but a fun adventure.

 

[zwf]

The problem with those two PCIe SAS controllers... They're RAID, which you do not want with ZFS at all. The bigger problem, they're second-sourced by a company that for all intents and purposes, hasn't existed since 2011. So it becomes a case of figuring out if you can flash one to IT mode or not.

The 375-3536-01 is I believe an Adaptec board, used by StorageTek, which was acquired by Sun, and then Oracle. Adaptec got swallowed up via a chain of acquisitions into Microchip Technology, Inc. Not sure how much support you'll find in FreeNAS for it. Also, this board has external facing SAS connectors, so you'd be looking at a secondary disk shelf.

Oracle Datasheet

The 375-3701-01 appears to be an LSI2108 chip set. You might be able to find a way to flash this one.

Oracle Datasheet

considering all of that , i guess i should go to the buy SAS card road
something like https://www.ebay.com/itm/163391043203
i was already pretty much convince of the bad idea of using those 2 sun part...

 

[zwf]

Considering the 12 RAM modules, I'd suggest a dual CPU board.
I'm note sure you'll find a single CPU board with 12 DIMM slots?
And you'll need to get a second X5675....

Dont need to use all 12 sticks.... i can bring back what unsed to him no issue

And I would directly start with 8 disks... why 6? :) If you add an HBA with 8 ports, then I don't see why you shouldn't go with 8 drives (if you have them anyway).

depend of the case ...

 

[Jessep]

I would suggest not going any older than E5-XXXX v1 (v2 would be better) for compatibility reasons. The 55XX and 56XX are getting very long in the tooth and will be deprecated into oblivion soon.

If this is exclusively a lab toy then fine, if you intend to put your precious data on it long term go with something newer.

 

[Yorick]

Is your heart set on uATX, that is a case like the Fractal 804? If so, using a different CPU would make sense, but then you’re also into using different memory. I am looking at https://www.supermicro.com/products/motherboard/archive/ and I am seeing some great ATX candidates, but nothing uATX for socket 1366. The low-cost option that I'd advocate for is to use what you have in an ATX case.

Did the CPU come with a heatsink? See the manual at https://www.supermicro.com/manuals/motherboard/5500/MNL-1137.pdf for pictures of what those look like. Given that you're not going into a case with a shroud, something with a fan attached seems prudent, along the lines of https://www.amazon.com/Dynatron-G199-Active-Blower-Cooler/dp/B002BYMHFY .

The Fractal Define R5 gets used a lot by people in this forum and can take ATX as well as 8 drives, and two SSDs for booting, and keep it all neat and quiet. Looks like a great choice for this use case.
Keep in mind that with ZFS, you can't just expand your raidz(2/3) with another disk. You'd have to build a whole another raidz vdev of the same size and add it to your pool. Not that two times 6 disks isn't lovely, but now what case does that go into? Using 8 from the word go is a great choice. The fact that ZFS will probably learn to expand raidz sometime in 2019-2020 notwithstanding. :)

You say bunch of disks but not what type ... all SAS? Got a model number? That's key to make sure you have the right controller and power/data connectors. Below I'm assuming they're SAS2 and use SFF-8482. If that's not so, let's adjust the BoM.

The x8dtl-6(f) goes on eBay between 120 and 200, and has 8 sas-2 ports on it. Also room for two CPUs, not that you have to use both sockets. SATA to connect a small ssd to boot from. You wouldn’t need a separate controller, as long as those drives are sas-2 compatible. If not, other models in that same line are available that have SATA or sas instead. I'd get the 6f for the ability to manage BIOS without needing to connect a monitor.

Room for three memory sticks per CPU, gives you 24GB with one. Plenty for a file server with 8 disks. Raidz2, you have 6TB to play with.

So that’s my thought right now for hardware. Define r5, a board off eBay that matches your disks, no hba needed, and a small sata ssd (between 32GB and 64GB, 128GB is okay too) to boot from. Size of that disk may depend on what you already have. If you’re feeling ambitious, use two boot drives in a mirror, but one SSD is quite reliable. Just back up your config. I have no particular opinion about SSD brands, here's one option. https://www.newegg.com/Product/Product.aspx?Item=9SIAJ3G8PV9914, and here's another, if you want a drive that is "too large" but could be repurposed for a small laptop or such: https://www.newegg.com/Product/Product.aspx?Item=9SIA12K7547820 , and here's one from eBay https://www.ebay.com/itm/292919340779

Power supply, something ATX that can power 8 SAS drives. Eight of these cables https://www.amazon.com/Goliton-SFF-8482-Connected-Motherboard-Adapter/dp/B01874KQP4 with a Seasonic 650 https://www.newegg.com/Product/Product.aspx?Item=N82E16817151192 will power your board and drives, and you'd need one of https://www.newegg.com/Product/Product.aspx?Item=12-200-060 to power the boot drive(s).

This bit about the CPU going EOL ... I’m not sure I follow. FreeBSD will continue to work on it, and newer AVX instructions and such aren’t needed or wanted. Educate me on why that Xeon is making you nervous, @Jessep .

 

[rvassar]

This bit about the CPU going EOL ... I’m not sure I follow. FreeBSD will continue to work on it, and newer AVX instructions and such aren’t needed or wanted. Educate me on why that Xeon is making you nervous, @Jessep .

There are no Spectre/Meltdown microcode updates for the socket 1366 CPU's. They are entirely dependent on FreeBSD getting the Retpoline mitigations correct, and will still suffer Spectre variant 2.

Here's the list as of last April. I can't seem to find a newer list.

Intel's Microcode Guidance - April 2018

 

[Yorick]

Oooo Infosec stuff, I actually happen to know a bit about that.

The concern about Spectre / Meltdown and other such sidechannel vulns is that evil code on the machine can read data it should not have access to, such as say crypt keys.

Note the prerequisite: Evil code on the machine. Something has to already be installed that’s malicious. There is a scenario where that comes in via plugin on FreeNAS, but that’s a very targeted attack to ... do what? Break the crypto on my NAS server? If I even have data important enough to encrypt in the first place.

On desktop machines, the concern is JavaScript. Browser mitigations will be good enough in most all cases I can think of.

The people that care are the ones running hypervisors with public workloads.

So, for a home server, particularly a FreeNAS one: Time to relax. There won’t be malicious code making its way on there to then use Spectre to read data it shouldn’t.

We haven’t even seen attacks on easier targets exploiting these vulns, because it’s too much work. So much easier to just phish people.

As always, devil is in the details. So many details with these sidechannel vulns.

 

[rvassar]

The people that care are the ones running hypervisors with public workloads.

Yes, and those people have for the most part EOL'd those servers, which is why there's such a plethora of cheap socket 1366 kit on the secondary market. I still have two of them myself, a pair Dell T3500's. They're still useful, and I'm not moving to get rid of them yet. I simply don't use them for anything public facing.

The other problem I have with them is they're not terribly power efficient, and unless you run something that takes registered RAM, you're stuck at 4Gb per stick densities on a lot of the motherboards. The 6 core chips are up in the 95 to 130 watt TDP range, as are the older 4-core W35xx chips, and that's enough to heat up my home office. Skipping to socket 1155 gets you 8Gb sticks of RAM and a nice performance bump @70w TDP, and the Spectre/Meltdown fixes, for very little additional money. It's just a better position to start a build from in 2019.

 

[Yorick]

@rvassar Yeah, I get that. It’s a matter of dollars and cents. Let’s see what OP comes back with. 6TB of disk to run home backups to: That’s a hobby. One might as well just buy a couple WD elements and be done, back up locally. But where’s the fun in that.

I’ve been attempting to help by coming up with hardware that uses the free kit he got. So noted to do another scrub on those memory sticks and make sure they’ll work in that board. Whether he’ll go for that, spend more and do a “modern build”, or spend much less and just back up locally, that’s up to OP.

 

[Yorick]

Just because, monies needed for "use what you have" and "get slightly newer" as per https://forums.freenas.org/index.ph...ting-expensive-need-advice.71880/#post-497585 via @Chris Moore

Common:
Case - Fractal Define R5 $122
SSD (times two) from eBay (Intel 40GB) and power cable and two SATA cables - $30 and $4 and $6
PSU - $110
8 x 1TB SATA drives, free - 6TB-ish usable with raidz2
==
USD 272

Old hw:
Existing Xeon
Existing memory
Supermicro x8dtl-6(f) from eBay - $160 or thereabouts, watch out that it's "PCB 2.01 or later version" to support your RDIMMs, see mobo manual
Active cooler - $34
SATA cables - $24
==
USD 210 to 280, depending on board cost

Slightly newer as per C.Moore:
Xeon E5-2650 V2 from eBay - $85
16GB or 32GB RAM from eBay - $30 to $60
Supermicro X9SRL-F from eBay - $200 or thereabouts, maybe 180 if lucky
Active cooler from eBay - $43, Thermal Compound if you don't have any - $8
SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode from eBay - $43
SAS cables to 4 times SATA times two - $26
==
USD 415 to 465, depending on board cost and amount of memory

520 bucks - ish for using what @zwf has, vs 680 bucks -ish for a "newer used" build with Xeon v2. Those 160 extra may be worth it. Up to the person with the free hardware.

Vs buying a WD elements for $100-150 a drive, 4TB to 8TB each, and using that as either external or internal backup storage, one per PC or one 8TB external that gets plugged in manually and kicks off the backup on connection. Or indeed a WD MyCloud, 4TB for 160 or 8TB for 300, network-connected.

A cheap hobby it ain't, building a FreeNAS server. It does offer redundancy, the ability to expand, and media streaming abilities. Just for backup for a couple TB, one would need to want to build it pretty badly to justify it.

Edit: C. Moore is at it again with a budget idea.

Dell PowerEdge T30. Comes with Xeon E3-1225 v6 and 8GB of ECC RAM, extend to 16GB when/if needed. Room for up to 4 hard drives. You'd boot from USB with this one, to keep the SATA ports for drives.
Boot: Sandisk 16GB, two in a mirror. This is an edit, USB 3.0 drives, I am learning, tend to overheat and fail.
SATA cables: Two SATA cables with right angle, short

Total without drives: USD 388

Add two shucked WD Elements 8TB, which most likely have HGST Ultrastar @5400 in them, for 150 each, and for USD 688 (plus Canada uplift) you have 7.3 TiB of space. Not too shabby. You might need to do minor surgery to get those Ultrastars to spin up, depending on how the Dell power supply behaves.

 

[rvassar]

Supermicro X9SRL-F from eBay - $200 or thereabouts, maybe 180 if lucky

If I could get my hands on one of these... One of my T3500's would go away. $200 is still a bit rich for me.

 

[zwf]

Thanks I will be reading all that and will get back to you when i get back home next week.
but more i read more i am asking myself if my idea is good anymore.. :(
and i am in canada everything is just a bit more expensive than you guys :)

 

[zwf]

So i Am Back home and here are the info on the disk you asked

• Hitachi GST Ultrastar A7K1000 HUA721010KLA330 1TB 7200 RPM 32MB Cache SATA 3.0Gb/s 3.5"

Thanks

 

[Yorick]

Makes the cabling a little easier, it's just SATA. Updated my post above with SATA cables.

I'd also look into the recent "what kind of used systems are out there" thread, that's at https://forums.freenas.org/index.php?threads/custom-build-or-ixsystems-freenas-mini.73121/page-2

 

[zwf]

Thanks i will check that.
travelling a lot at the moment so hard to start anything