suggestions for a low power system to replace that old dual xeon macpro

[hotdog]

hi everyone
a while ago (april 2014, according to my own posts here in this forum) I made a freenas out of an old mac pro.

Mac Pro "Eight Core" 2.8 (2008) Specs (Early 2008, MA970LL/A, MacPro3,1, A1186, 2180): EveryMac.com

Technical specifications for the Mac Pro "Eight Core" 2.8 (2008). Dates sold, processor type, memory info, hard drive details, price and more.

www.everymac.com

never had problems, running 24/7 since then, no disk fail but I feel now it's time to replace the thing:
the 3 mirrored 4gb wd reds are old and the power consumption of those dual xeons just will not go down by itself anytime soon...

the new thing should:
- act as zfs fileserver for 4 users max, I would like to keep it simple with the setup. triple mirror 8gb is enough. option for total 6 sata hdds
- make encrypted rsyncs to an external file hoster as backup
- host a VM mini windows just for running lic-server services
- maybe hosting owncloud, not sure yet.
- write an read speed is ok as it is now.
- option to upgrade to 10gb lan via pci card
- maybe have a supermicro board (because I like supermicro)
- not draw to many watts
- price not to important

some thoughts about pool size:
- now we are using 2.5 tb out of 4 tb, lz4 compression on, but doing no snapshots
- I would like to do a snapshot every day going back 30 days.
- our content is mostly static and just growing over time (with project files etc.).


xeon-d? I read the hardware suggestions guide but I think something newer than a v3 xeon maybe could be more power efficient?

any suggestions are much appreciated
cheers

 

[hotdog]

ok let me ask more specific: First I see two options, used vs newer, example:

Combo 1:
X10SL7-F / E3-1231 / 16 GB RAM

Combo 2:
X11SSM-F / Xeon E3 1230 v5 or v6 / 16 GB RAM

Except for the significant price difference, will there be a difference in
- power consumption?
- limitation on future freenas features in the next 5 years?

the plan is to build a simple pool with 3 disks first like the old one, but bigger disks (8 or 12 TB), so enough sata port by default will come in handy when growing the pool.

Another Question:
I feel like the E3 1230 could be overkill. Will it draw more power while idling than a E3 1220? Or is it finally just a price question?

tnx for your inputs.

 

[Jessep]

v5/v6 should pull significantly less wattage.

For drives larger than 2TB or so it's best to use RaidZ2 or RaidZ3 due rebuilt times. This would require 4 or more drives with the best space/redundancy balance being around 6 drives.

High stress of the drives during rebuild and the length of the rebuild, it's possible to have a second failure (or drive drop out) that could cause data loss. During rebuild of a mirror or RaidZ1 you have no redundancy. This is less of a concern with mirrors as the rebuild faster, still much more concerning than having full redundancy during rebuild as with RaidZ2 or RaidZ3. It's up to you to balance your risk tolerance vs. price/convenience.

 

[anmnz]

v5/v6 should pull significantly less wattage.

I think idle power consumption has only seen small incremental improvements in successive processor generations (since Ivy Bridge anyway), so would not expect a significant difference here. If that's wrong I'd really like to see some concrete details!

I feel like the E3 1230 could be overkill. Will it draw more power while idling than a E3 1220?

I believe the conventional wisdom is that idle power consumption is pretty much the same across CPUs of the same generation, these days, so would not expect a significant difference.

 

[Constantin]

The Xeons In the old MacPros were pretty power hungry even at idle, IIRC.

If you want something prebuilt, the current mini XL is pretty future-proof for your use case. Drop in 5+ drives with a Z2 array and you should be good to go. Power consumption should be a lot lower, performance should be pretty excellent as a file server.

Keep in mind that every additional drive will draw about 7 watts of power (verify via spec sheet) so your aim to reduce power consumption should include a cost/benefit analysis re # of drives.

Also consider a 10Gbe trunk to your switch.

 

[hotdog]

For drives larger than 2TB or so it's best to use RaidZ2 or RaidZ3 due rebuilt times. This would require 4 or more drives with the best space/redundancy balance being around 6 drives.

High stress of the drives during rebuild and the length of the rebuild, it's possible to have a second failure (or drive drop out) that could cause data loss. During rebuild of a mirror or RaidZ1 you have no redundancy. This is less of a concern with mirrors as the rebuild faster, still much more concerning than having full redundancy during rebuild as with RaidZ2 or RaidZ3. It's up to you to balance your risk tolerance vs. price/convenience.

tnx for your input.

I understand. I just like the "simpleness" of the tripple mirror which I am used to for the last 5 years. Never had a disk failure. The data will be synced in a datacenter offsite every night. An eventual data recovery is not that time sensitive (mostly our architecture projects).
I don't mind the "waste" of the tripple mirror as long as our data fits on one drive. Write times on the MacPro are 125 MB/s for mixed files and at the moment I am happy with it.

BTW:
Do larger drives fail more often?
Or: Is a drive failure more likely on 6 4TB drives or 3 8TB ones?

 

[hotdog]

The Xeons In the old MacPros were pretty power hungry even at idle, IIRC.

Yeah THAT IS TRUE :)

If you want something prebuilt, the current mini XL is pretty future-proof for your use case. Drop in 5+ drives with a Z2 array and you should be good to go. Power consumption should be a lot lower, performance should be pretty excellent as a file server.

I built every single one of my computers except my first IBM Compatible XT 2.7Hz and that MacPro I was talking about.
Kinda like doing that..

Also consider a 10Gbe trunk to your switch.

I do, should not be a problem via PCI slot in the future?

But then I also considered a X11SDV-4C-TP8F, but I then need to manufacture/customize a quiet cooler I guess, since the box will not stay in a air conditioned server room and should be reasonably quiet.

 

[Greg_E]

You could possibly use one of the Supermicro Atom powered main boards, but finding a chassis to hold three or four 3.5 inch drives might be difficult. 11.2-u6 will "run" on my old Atom 525d computers, but limited to 4gb of ram makes it a bad idea so probably moving that test machine back to version 9.x, not completely certain yet.

I have two servers running the x10 boards, no problems there and I recently upgraded storage on one to eight 10tb drives and dual 10gbe cards. The x11ssm-f that I have are doing different things so can't comment on those. They are pretty low power machines but might fit well in a chassis that holds the drives you need.

 

[Constantin]

If you enjoy building systems, there is a lot to choose from out there. I'm inclined towards the use of SuperMicro boards on account of the company focus on the server space. For a limited set of SATA ports, I'd consider the X10SDV-2C-TLN2F

It's Mini-ITX, relatively inexpensive, offers 4 memory slots, 6 SATA ports, one PCIe 3.0 x16 slot, onboard 10Gbe, 1Gbe, etc. copper, one PCI-E 3.0 x4 m2 slot (great for a SLOG like mine), etc. all in a 25W TDP package and about $370 at Newegg. I'd put it in a well-ventilated case along with 32GB of RAM (ideally two 16GB sticks for future expansion, if needed). That buys you a lot of expandability in the future (L2ARC, SLOG, HBA, etc.) if you ever need it.

A SATADOM is a nice-to-have as a boot device but it's not a must have. If you go SATADOM, go for a 32GB or so, that reduces capacity anxiety. Or, just buy a small SSD. I would not boot off a USB stick in a production server, however.

These D-15xxx boards are a great value and the power savings vs. atom series are minimal. D15xx offers much better expandability (far more PCIe lanes to play with) yet the pricing on the D-15xx series is almost the same as the atom series.

 

[hotdog]

If you enjoy building systems, there is a lot to choose from out there. I'm inclined towards the use of SuperMicro boards on account of the company focus on the server space. For a limited set of SATA ports, I'd consider the X10SDV-2C-TLN2F

It's Mini-ITX, relatively inexpensive, offers 4 memory slots, 6 SATA ports, one PCIe 3.0 x16 slot, onboard 10Gbe, 1Gbe, etc. copper, one PCI-E 3.0 x4 m2 slot (great for a SLOG like mine), etc. all in a 25W TDP package and about $370 at Newegg. I'd put it in a well-ventilated case along with 32GB of RAM (ideally two 16GB sticks for future expansion, if needed). That buys you a lot of expandability in the future (L2ARC, SLOG, HBA, etc.) if you ever need it.

A SATADOM is a nice-to-have as a boot device but it's not a must have. If you go SATADOM, go for a 32GB or so, that reduces capacity anxiety. Or, just buy a small SSD. I would not boot off a USB stick in a production server, however.

These D-15xxx boards are a great value and the power savings vs. atom series are minimal. D15xx offers much better expandability (far more PCIe lanes to play with) yet the pricing on the D-15xx series is almost the same as the atom series.

This sounds like a good option, thanks for pointing in that direction. Do you think the 1508 will be beefy enough for the tasks I am planning listed in my first post?
A D1528 would be harder to cool silently I guess.

 

[Constantin]

The 1508 is also offered in one size up on a flex atx basis. There, with the help of a HBA Chip it can handle up to 20 SATA drives with just the ports on the MB.

For SMB and AFP, clock speed is key because these protocols are single-threaded. The 1508 runs faster than the 1537 in my rig and in retrospect the 1508 likely would have been a better choice for me. Less heat, less power, and likely faster performance for my use case.

so unless you have plans to get into VMs and like projects, I doubt the other D series chips will offer a great deal of benefit. Remember, you will automatically limit yourself to maybe 160MB/s even with 10Gbe networking on account of your pool setup. If you want faster performance, you’ll need more vdevs (and by extension, more drives).

that’s the reason I went with a motherboard that can handle 20 drives if need be. I have no current plans to do so, but adding another VDEV is only hampered by the current physical room in my server. The motherboard and the rest of the infrastructure are not the bottleneck, the single VDEV is.

coming back to your rig, one option you could consider is a pool with two VDEVs, consisting of two-three mirrored drives each. That would buy you ~320MB/s sequential write performance. Just be sure to set up those vdevs identically!

 

[hotdog]

The 1508 is also offered in one size up on a flex atx basis. There, with the help of a HBA Chip it can handle up to 20 SATA drives with just the ports on the MB.

For SMB and AFP, clock speed is key because these protocols are single-threaded. The 1508 runs faster than the 1537 in my rig and in retrospect the 1508 likely would have been a better choice for me. Less heat, less power, and likely faster performance for my use case.

so unless you have plans to get into VMs and like projects, I doubt the other D series chips will offer a great deal of benefit. Remember, you will automatically limit yourself to maybe 160MB/s even with 10Gbe networking on account of your pool setup. If you want faster performance, you’ll need more vdevs (and by extension, more drives).

that’s the reason I went with a motherboard that can handle 20 drives if need be. I have no current plans to do so, but adding another VDEV is only hampered by the current physical room in my server. The motherboard and the rest of the infrastructure are not the bottleneck, the single VDEV is.

coming back to your rig, one option you could consider is a pool with two VDEVs, consisting of two-three mirrored drives each. That would buy you ~320MB/s sequential write performance. Just be sure to set up those vdevs identically!

I am aware I can double write performance by doubleing the drive count. I am not sure if I need that write performance, right now, we are happy.
If I decide so, maybe I will go with 2.5" drives to keep power consumtion low (do you think this is a good idea?).

About workload, for:
- making nightly encrypted rsyncs to an external file hoster as backup
- a VM running something like micro windows xp (running lic-server services which need windows only)
- maybe hosting owncloud,
you still think the 1508 is a good choice?

tnx a lot for your time, and a happy new year to you all!


EDIT: you mean the X10SDV-2C-7TP4F I guess. Starting looking into that, really like what I saw till now.

 

[jgreco]

I think idle power consumption has only seen small incremental improvements in successive processor generations (since Ivy Bridge anyway), so would not expect a significant difference here. If that's wrong I'd really like to see some concrete details!

The trashcan Mac Pro sported an Ivy generation CPU, but no one's making a NAS out of that. I would expect that the Mac Pro in question is substantially older and probably substantially piggier at power.

Balancing the power sip of the Xeon D's (and generally slow clock) against piggier full Xeon (where you can get great clock) is a tricky thing. As long as you stick to a single socket board, though, you're definitely going to wind up with a lower power solution.

 

[Constantin]

The 4,1 MacPro was using Nehalems (up to 2) with a TDP <100W each. The later 5,1 generation was also power hungry, though I guess the Westmere's weren't as bad? Running a older MacPro under load is going to do a great job of heating whatever room its sitting in. I still have a G5 in the same type case and mine was one of the water-cooled weirdos.

Coming back to the OP, I'd like to think that a 2-core, 4-thread machine would work just fine for storage. I cannot speak to VM or own cloud, have never tried using it here. However, even if you were to max out one thread with SMB/AFP traffic, that leaves three other threads to do everything else. However, I'd get advice from someone more knowledgeable. However, I'd consider upping the RAM if you want to run a bunch of jails to ensure the FreeNAS is not starved for memory.

As for the X10SDV-2C-7TP4F, yes that is the board I am thinking of. The only downside I have discovered thus far is that the HBA chip runs pretty hot and if you ever want to use the second PCIe lane, then any retrofit cooling solution has to somehow fit under the card. I use a very flat cooler and most of the time, the HBA stays below 50*C, which I guess is on the cool side for these chips.

This board is built for high airflow in a limited enclosure (SM ships them either with a plastic cover to direct air flow or in an server enclosure with limited vertical height. Either way, the board is designed to get a lot of air flow. I use multiple fans and a DIY shield to direct the air flow across the CPU HX and the HBA as well.

What I love about the X10SDV-2C-7TP4F series is the SFP+ port (greater flexibility re: 10Gbe; fiber or copper), two PCie slots, a m2 for my L2ARC, a PCI 3.0x4 NVME for my SLOG, 4 memory slots for RAM, two SATADOMs for boot pool protection, etc. It's an all around amazing board for data storage applications.

However, be sure that the board will fit in whatever case you choose! Flex ATX is pretty rare!

 

[anmnz]

I would expect that the Mac Pro in question is substantially older and probably substantially piggier at power.

Yeah I should maybe have quoted more carefully; I was actually trying to address the OP's apparent question whether going from a Xeon E3 (v3/v4, in an X10SL7-F) to a Xeon E3 v5/v6 (in an X11SSM-F) would save much power.

No doubt at all, either will be much more efficient than the old Mac Pro.

 

[Apollo]

Xeon E3 V3 is kind of "poussif" (French for pooling behind). My G4600 seeems to be doing better though.
E3 V5/V6 might bring marginal performance and power efficiency compared against AMD.

 

[Constantin]

The use case suggests that pretty much any modern board will suffice. There are minimal disks, maybe one VM, a few other processes like own cloud. As described, I’d wager that the system will be limited by disk I/o.

the nice thing about boards with removable processors is that they can be upgraded later. No such luck with the d- or atom series, they’re embedded. So for this OP, a low power board with an e3 or whatever might be the ticket and if there is a future need for 20 VMs and plenty more memory, the processor / RAM can be upgraded.

I’d choose a board either off the resource list of recommended platforms or go with one of the embedded units discussed here.

 

[hotdog]

Yeah I should maybe have quoted more carefully; I was actually trying to address the OP's apparent question whether going from a Xeon E3 (v3/v4, in an X10SL7-F) to a Xeon E3 v5/v6 (in an X11SSM-F) would save much power.

No doubt at all, either will be much more efficient than the old Mac Pro.

yeah the mac pro thing was more like an introduction, only to say that I decided to build a new file server. If I remember correctly, the mac pro draws about 180 Watts from the wall, idleing (I measured it I year ago, but it's up since then and I don't like disturbing it...).
the thing has 14 GB of fully buffered ecc ram. All I did 5 years ago is swapping back the stock Radeon GPU against the mac special edition Nvidia quadro card which would have made it much worser again. What can I say: the thing was standing around and basically just fullfilled the system requirements of the freenas version from these days. and it is still weekly scrubbing my pool..

Anyway thanks for all your help, I tend to try an embedded board like the X10SDV-2C-7TP4F. Unfortunately especially that one must be ordered for me.

Multiple VDEVs also seem to be a better future proof choice.

BTW I looked up my local prices (Switzerland)
I figured, maybe I could reduce my initial space requirements, let's say 8 TB. I'll then go with 4TB drives.

2 VDEVs built out of:
2 tripple mirror 3.5 HDDs (6 drives in total, WD Reds) -717$
or
2 tripple mirror 2.5 HDDs (6 drives in total, Seagate BarraCuda) - 1050$
or
2 mirrored 2.5 SSDs (4 drives in total, Samsung 860 QVO) - 1950$ --> insane write speed over 10Gb Network?

I guess resilvering with SSDs is way safer than with spinning disks? Still a very pricy option...

 

[John Doe]

maybe a few hints to power consumption:

try to avoid putting more stuff into your machine.

E.G. better to find a cpu and a board which has sufficient amount of sata ports (or whatever you want) Instead of adding HBAs later on.
Or better to have 1x 16gb ram instead of 4 times 4 gb
unbuffered ECC RAM should use less power than reg ECC

that might also apply to 10 gbit nics. there are supermicro boards available with 10 gbit nics. But I have no experience with that (if it works good, reliable, fast etc)

Discs with helium usually run cooler and use less power

lower RPM for spinning HDDs is also beneficial to safe power.

in general; there is a break even point, between saving power and pay higher prices for the components.
Since the price for energy in switzerland is low (compared to other European countries) i would not go for very expensive products in order so safe a few watts.

i felt very comfortable using ssds for the jails and the logs. in order to safe power, i set the HDD drives to keep spinning but the head is in parking position (should be lowest power state without spindown) if you have the jails and log on the HDDs there are continuous writes, so in essence they will not safe energy (that I experienced in my system).

 

[joeinaz]

hi everyone
a while ago (april 2014, according to my own posts here in this forum) I made a freenas out of an old mac pro.

Mac Pro "Eight Core" 2.8 (2008) Specs (Early 2008, MA970LL/A, MacPro3,1, A1186, 2180): EveryMac.com

Technical specifications for the Mac Pro "Eight Core" 2.8 (2008). Dates sold, processor type, memory info, hard drive details, price and more.

www.everymac.com

never had problems, running 24/7 since then, no disk fail but I feel now it's time to replace the thing:
the 3 mirrored 4gb wd reds are old and the power consumption of those dual xeons just will not go down by itself anytime soon...

the new thing should:
- act as zfs fileserver for 4 users max, I would like to keep it simple with the setup. triple mirror 8gb is enough. option for total 6 sata hdds
- make encrypted rsyncs to an external file hoster as backup
- host a VM mini windows just for running lic-server services
- maybe hosting owncloud, not sure yet.
- write an read speed is ok as it is now.
- option to upgrade to 10gb lan via pci card
- maybe have a supermicro board (because like supermicro)
- not draw to many watts
- price not too important

some thoughts about pool size:
- now we are using 2.5 tb out of 4 tb, lz4 compression on, but doing no snapshots
- I would like to do a snapshot every day going back 30 days.
- our content is mostly static and just growing over time (with project files etc.).


xeon-d? I read the hardware suggestions guide but I think something newer than a v3 xeon maybe could be more power efficient?

any suggestions are much appreciated
cheers

I am trying to understand why such a focus on CPU power usage for your application. Usually CPU power consumption is a big deal in a data center where there are dozens or even hundreds of servers running. The cost of power and cooling is significant such a high density environment and companies are willing to pay a premium to get similar computes with less power. In an office or home environment with a small number of servers it can be difficult to justify the cost savings of a low power/high performance CPU. Consider the following options:

Dual Intel L5640 CPUs vs one E5-2630-V2 vs one E3-1230-V6. Each of these CPU solutions provide similar performance on the PassMark tests. The annual power costs (using CA residential power rates, 24/7 usage and 25% avg. CPU usage) are as follows: Dual L5640s: $46.32; 2630 V2: $26.88; 1230 V6 $24.29.

The good news is the 1230V6 (assuming 24/7 usage) will save about $22 a year over the dual L5640s and about $1.50 annually over the 2630 V2. There will potentially also be savings in cooling costs as well. The challenge is the cost of newer CPUs, memory and motherboards, make the older hardware a better value. For example (on eBay) a pair of L5640s will run less than $20. A single 2630 V2 will cost less than $25. A single E3-1270 V6 starts at about $220. Newer CPUs are on a steep price curve compared to older CPUs.

One last thought; If you are looking for lower power and good performance, consider CPUs with an "L" at the end. The 2630L V2 (6 cores) or better yet the 2648L V2 (10 cores) offers quite a bit of performance at less the 75w per CPU. Pair that with a SuperMicro board like an X9SRI-F and you get the power savings you want and the cores you need for virtualization at a lower cost.