Engineering Sample Xeon E5-26xx CPUs on Supermicro X10 Mobo?

[pclausen]

So I have started having issues with my current Xeon E5-1620 v3 processor running out of juice when having to transcode media content to certain clients on my network. I thought about upgrading to a 6 core E5-1650 v3, but I think that would only postpone the issue for a little while.

I started looking at the E5-2600 V3 series, but quickly got sticker shock when I realized the 14 - 18 core ones retail north of $2000!

So I was checking out eBay and it looks like "Engineering Sample" versions of these are available for much less. Here's a nice 14 core one for under $400:

http://www.ebay.com/itm/Intel-Xeon-...965653?hash=item235b42e395:g:6v8AAOSwVL1V~YkV

I sent the seller a message asking about compatibility with my motherboard, which I think it will be.

But then I got to thinking, is there *ANY* change that, even if this CPU works fine for me, that I could somehow corrupt my ZFS filesystem?

 

[jgreco]

Anything's possible. Without knowing the very specific quirks of a particular batch of ES CPU's, it's so hard to say. It SEEMS unlikely that it'd corrupt your data. No promises.

More generally, if the 1620 is running out of steam, the 1650 is only ~50% more oomph. However, I'll note this. And it isn't a direct translation to your sitch... I migrated stuff from a pair of E3-1230's onto an E5-1650 box and ended up with some free space overhead. If you look at the Geekbench numbers for the Sandy 1230 (~10000) and the 1650v3 (~20000) you would think that loading two of the former onto the latter would result in approximately the full thing being used. But it doesn't work out quite like that unless you're actually maxxing out the CPU constantly. If you're dealing with slightly bursty workload, the extra cores actually help things out because you're getting the work done faster, and you end up aggregating what was probably little bits and pieces of idle time on the 1230's into larger amounts of free space on the 1650, especially if the traffic is bursty.

Now, we've got some of the 1650v3 here and it's a very pleasant CPU, but, yes, it is limited. It is also the best bang-for-the-buck server CPU you can get in the Xeon lineup, plus some of the best individual core speeds.

On the other hand, a few years back I managed to weasel a Xeon E5-2697 v2 (~$2700?) slightly before the official release date. That box is an amazing performer. It's often running as many as a hundred VM's (though many of them aren't particularly busy) and I don't often regret the silly price of the CPU. A decent hypervisor box based on the 1650v3 costs ~$4-$5K so one can kinda justify a pricey CPU if it stops you from buying a whole 'nother box.

 

[pclausen]

The seller got back to me and confirmed that it should work on my particular motherboard and BIOS revision. He offers a return if it doesn't and all correspondence was through eBay, so I went ahead and pulled the trigger. Fingers crossed it will work without issues.

Yeah, my E5-1620 is running out of steam even with just 2 clients transcoding content and then when a 3rd client, that doesn't need to transcode, tries to navigate the Emby media library, it will often loose the session to the server, and is generally very sluggish. The interesting thing is that until recently, the Emby server (very similar to Plex) was running on a standalone Windows 10 server with a i7-4770K CPU and I never had any issues. The win10 box was of course accessing the media content from the FreeNAS server. On paper, the 4770 and 1620 has Pass Mark scores that are very close, with a slight nudge to the 4770k.

Here's the 1620 with 2 transcodes and a 3rd client just browsing:

1 client transcoding, then 2 clients. No other activity.

Me running 3 transcodes from local workstation browser:

4 concurrent transcodes with top showing server on left and Emby jail on right:

This test was performed from my workstation which now has the 4770k in it.

I was never able to get 5 streams going at the same time. I guess each ffmpeg process "owns" a core while it's running?

It will be interesting to repeat this test once I get the E5-2683 V3 up and running. If I still run out of steam, then I guess it's time to trade in my single socket X10 mobo for a dual socket one and pick up another 2683. :D

 

[rogerh]

It will be interesting to repeat this test once I get the E5-2683 V3 up and running. If I still run out of steam, then I guess it's time to trade in my single socket X10 mobo for a dual socket one and pick up another 2683. :D

As a matter of interest (because I've been out of touch with such things for years) can run two processors with two separately obtained ones nowadays? ISTR you generally had to get identical stepping ones, preferably sold certified to work together.

 

[jgreco]

Yes, they need to be the same stepping.

 

[pclausen]

Yep, I learned that the hard way back on my X7 motherboard running dual 5400 Xeons. Fortunately I was able to pick up a 2nd 5492 with a matching stepping for less than what I was able to sell the one that didn't match for. What a power hog that setup was when doing something that used up all available resources!

 

[cyberjock]

Well, consider these potential problems:

1. The CPU is illegal to own. Not entirely your fault, unless you know its illegal to own.
2. The CPU errata is guaranteed to not match the CPU errata for the 'official' CPUs that are out there. Normally your compiler works around them, and other tricks make stuff not go sideways in certain scenarios, but you no longer have that guarantee. So running code on the CPU may work fine and one day you hit some edge case that the CPU errata doesn't handle and... kaboom.. end of your data.
3. The BIOS may or may not be compatible today, or some time in the future with an update.

It's really a big grey area as to how 'important' the server is.

Ignoring the illegal aspects for a minute...

For a desktop, it's probably a fairly safe bet from a reliability standpoint since the consequences of a failure aren't severe. If it goes sideways the worst that happens is your desktop stops working properly. Maybe you need a new OS install. For a server (which you are relying on to be 100% stable without any possibility of problems) you're taking some risks and certainly shouldn't 'expect' 100% as you are basically using a CPU that was/is never tested for your BIOS, has a totally undocumented errata list that nobody has a compiler to deal with, etc.

Its your data and your risk. Enjoy the money saved, and if you lose your data just remember that you lost it trying to save some cash on a CPU that could best be labeled as "experimental".

A friend bought an ES CPU some years back. It worked great in Windows. He never had any major problems with it that he could point the finger at the CPU for blame. But linux was a different story. On bootup he had errors that were directly attributable to the CPU, and things went downhill from there until the system went unstable and crashed. Hit the reset button and rinse and repeat, ad nauseum. That being said, ultimately he had major stability problems as corruption to the file system ultimately resulted and he ended up losing about 900GB of data (which at that time was a MASSIVE amount of data).

 

[anodos]

There's a saying in a part of the world I used to live in 'we're not rich enough to buy cheap things'.

 

[koifish59]

For desktop Engineering sample CPU, I might have taken the cost savings, since system crashing or corrupting is not as crucial. But for a mission-critical server, I would never even consider ES CPUs, no matter how cheap. These CPUs are meant for TESTING, and as a result, might not calculate things 100% the same as a retail Xeon (I don't know the terminology, but cyberjock mentioned "errata"). It might seem like it's running fine, but you can't detect micro code and errors that are not apparent. I also know for a fact that many desktop CPUs have been heavily abused (I use to be into the benchmarking and overclock scenery), although I'm not sure if this is true for server chips.

But either way, as per Intel's contract when they send out ES chips, you're not allowed to sell these (so likewise, you shouldn't buy them either). There are different serial numbers for every ES chip.

 

[pclausen]

Appreciate the feedback. This server isn't really mission critical and I agree that if my business depended on this server I would never run a ES CPU. This server is just for serving up media to various devices around the house. I would think that if Intel really had an issue with the 100's of adds on eBay selling ES version CPUs, they would have contacted eBay and have them take those adds down.

So I got the CPU today from Japan and I'm not convinced its even an ES one. There's no Intel Confidential stamp on the front and the rest of the labeling looks productionish from what I can tell.

And the back even has a Q.C. PASSED sticker on it!

I got a bigger heatsink for it since I'm sure it will generate more heat than the E5-1620 V3 did.

Supermicro mobo booted right up:

Microcode revision 14 it would seem. Not sure how that compares to production models, if this in fact isn't one?

FreeNAS picked up the 28 Threads:

Once everything was booted up, I started up 6 ~25 mbps 1080p transcoding sessions to 6 Emby clients:

Temp are holding steady in the mid 60's and server is plenty responsive for other tasks. I could probably have started another 6 ffmpeg sessions with no issues.

The S-Spec number SR1XH would appear to be a production version according to the following link:

http://www.cpu-world.com/sspec/SR/SR1XH.html

So who knows, maybe I got lucky and got a $2,000 production unit for $350? I'll keep pushing it to see if I can make it break, but so far so good!

 

[pclausen]

Trying to find some command line utils to get me some CPUZ like info. Not finding much. This is all that I could come up with:

Code:

[root@freenas] ~# sysctl hw.model hw.machine hw.ncpu
hw.model: Genuine Intel(R) CPU @ 2.00GHz
hw.machine: amd64
hw.ncpu: 28
[root@freenas] ~# grep -i cpu /var/run/dmesg.boot
CPU: Genuine Intel(R) CPU @ 2.00GHz (2000.04-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 28 CPUs

I did stumble across this which is pretty cool:

Code:

[root@freenas] ~# sysctl -a |egrep -E "cpu\.[0-9]+\.temp"
dev.cpu.0.temperature: 48.0C
dev.cpu.1.temperature: 48.0C
dev.cpu.2.temperature: 46.0C
dev.cpu.3.temperature: 46.0C
dev.cpu.4.temperature: 50.0C
dev.cpu.5.temperature: 51.0C
dev.cpu.6.temperature: 49.0C
dev.cpu.7.temperature: 49.0C
dev.cpu.8.temperature: 44.0C
dev.cpu.9.temperature: 44.0C
dev.cpu.10.temperature: 44.0C
dev.cpu.11.temperature: 44.0C
dev.cpu.12.temperature: 51.0C
dev.cpu.13.temperature: 51.0C
dev.cpu.14.temperature: 45.0C
dev.cpu.15.temperature: 45.0C
dev.cpu.16.temperature: 49.0C
dev.cpu.17.temperature: 49.0C
dev.cpu.18.temperature: 46.0C
dev.cpu.19.temperature: 46.0C
dev.cpu.20.temperature: 48.0C
dev.cpu.21.temperature: 48.0C
dev.cpu.22.temperature: 49.0C
dev.cpu.23.temperature: 49.0C
dev.cpu.24.temperature: 49.0C
dev.cpu.25.temperature: 49.0C
dev.cpu.26.temperature: 51.0C
dev.cpu.27.temperature: 51.0C

 

[koifish59]

Hmm looks like you got pretty lucky there. Nothing laser-etched on it says that's it's an ES. One way to be sure is if you called up intel and gave them the serial number. But of course, if it's really an ES based on serial number, you might have to give it up ;)

 

[cyberjock]

Try this program... http://www.intel.com/support/processors/tools/piu/

That program is basically the definitive answer to whether you are running an ES, QS, Retail, etc.

 

[pclausen]

Thanks. Unfortunately the bootable version can only be extracted using a 32-bit version of windows. Everything I have is 64-bit these days.

I'll probably yank the LSI cards and install/boot into windows and run the windows version and CPUZ and a few benchmarks while I'm at it, to find out exactly what I got.

 

[Ericloewe]

The QC sticker is really odd. How the hell did it end up there?

 

[rogerh]

The QC sticker is really odd. How the hell did it end up there?

Could cause problems if it fell off at a strategic moment.

 

[Ericloewe]

Could cause problems if it fell off at a strategic moment.

I'd definitely never leave it attached to the CPU.

 

[Jeff Armstrong]

Well, consider these potential problems:

1. The CPU is illegal to own. Not entirely your fault, unless you know its illegal to own.
2. The CPU errata is guaranteed to not match the CPU errata for the 'official' CPUs that are out there. Normally your compiler works around them, and other tricks make stuff not go sideways in certain scenarios, but you no longer have that guarantee. So running code on the CPU may work fine and one day you hit some edge case that the CPU errata doesn't handle and... kaboom.. end of your data.
3. The BIOS may or may not be compatible today, or some time in the future with an update.
... .

You are quite correct. The picture the OP posted in the top post of the advertised CPU is a true engineering sample. The picture he posted of what he received is an authentic, non-engineering sample, so he is safe overall.

Engineering samples should not be used in general (forget the legality) because you don't know exactly what was done to the CPU. Engineering samples are sent for all sorts of reasons, some are even given away to employees, and putting them up on Ebay is generally not liked. I know of a few cases employees who sold their loaner chips on ebay were terminated forthwith.

 

[jgreco]

The QC sticker is really odd. How the hell did it end up there?

Someone stuck it there for ironic purposes.