X11 is the latest recommendation?

[ISJ]

Hello,

I was reading the "Hardware 2021 R2a.pdf" posted here and I'm a bit concerned I'm buying the improper motherboard given a lot of the recommendations are old posts. Has nothing much changed? Are the X12 system incompatible in some way?

Is X11 really the latest motherboard to use? Whats the step up if any?

I'm looking for 64GB ideally up to 128GB of ECC for my 80TB and later 160TB system. Im not doing De-dupe or anything but encryption at rest so it should be okay with 64GB to 80TB and 128GB at 160TB I heard.

Thought?

Thanks.

 

[jgreco]

In the server world, it's rarely necessary to have the absolute latest and greatest, because this usually comes with less maturity as well. Your typical X9 or X10 system is quite serviceable, and X11 is a well-known and stable quantity at this point. X12 will be a little bit "living on the edge", and until lots of users have had positive problem-free experiences with them, are you really meaning to imply that we should be suggesting something we don't have confidence in?

If you want to be buying the latest and greatest, great. Go ahead. Someone has to be a guinea pig. There will be bumps and bruises along the way, since the mainline OS's will be trying to integrate support for new features like P-cores and E-cores. You can be sure that in a year or two, you'll have a nice stable system, but until then, it may be a bit bumpy.

 

[Redcoat]

I found this thread interesting: https://www.truenas.com/community/t...icro-board-recommendations.98353/#post-678667

​

 

[ISJ]

are you really meaning to imply that we should be suggesting something we don't have confidence in?

Not at all, its just counter to my expectations and seems to be limited to 64GB in a lot of cases. If you're saying in Q2 2022 that x11 is fine then im sold, just need to find one with 128GB ideally.

 

[ISJ]

I found this thread interesting: https://www.truenas.com/community/t...icro-board-recommendations.98353/#post-678667

​

Thanks, reading it now.

 

[jgreco]

Not at all, its just counter to my expectations and seems to be limited to 64GB in a lot of cases.

Intel Xeon's tend to come in "small" and "large" variants, such as the E3-1230 which is an X9 generation capable of 32GB of RAM, and an E5-1650 which is capable of 256GB. The differentiation between E3 and E5 used to make it super-easy.

For X11, you basically have choices between the X11SC board series, such as the X11SCL which takes 8th/9th gen E-21xx/E-22xx CPU's and up to 128GB which is the newer "Intel Scalable", which was the follow-on to the X11SS which take 6th/7th gen E3-12xx which are limited to 64GB, which is I suspect what you are seeing for a 64GB limit. These are both "smaller" boards for their respective CPU classes.

For "larger", it's more complicated, because you have X11SR such as the X11SRL which takes W-21xx/W-22xx CPU's and up to 1TB LRDIMM, or the X11SP such as the X11SPL which takes 2nd Gen Scalable Socket P which goes up to 2TB.

You will also find some limited boards such as X11SCL-iF boards limited to 64GB by virtue of only two DIMM slots, but that's a board design limit rather than architectural.

There's also really nothing stopping you from going X12 other than resetting your expectations to be realistic about the potential for problems and remediations. For example, I've got a compelling need to have a few workloads with access to recent Intel QuickSync, which is Really Very Hard to find. Been looking at an SYS-110T-M with E-2388G as one of the few plausible options, because Intel normally doesn't package QuickSync with the Xeons, except for a limited selection of workstation CPU's. Finding stock is an issue, and overall the frustration factor with X12 means lots of careful vetting of stuff with VMware's HCL and other joyful crap.

 

[ISJ]

Intel Xeon's tend to come in "small" and "large" variants, such as the E3-1230 which is an X9 generation capable of 32GB of RAM, and an E5-1650 which is capable of 256GB. The differentiation between E3 and E5 used to make it super-easy.

For X11, you basically have choices between the X11SC board series, such as the X11SCL which takes 8th/9th gen E-21xx/E-22xx CPU's and up to 128GB which is the newer "Intel Scalable", which was the follow-on to the X11SS which take 6th/7th gen E3-12xx which are limited to 64GB, which is I suspect what you are seeing for a 64GB limit. These are both "smaller" boards for their respective CPU classes.

For "larger", it's more complicated, because you have X11SR such as the X11SRL which takes W-21xx/W-22xx CPU's and up to 1TB LRDIMM, or the X11SP such as the X11SPL which takes 2nd Gen Scalable Socket P which goes up to 2TB.

You will also find some limited boards such as X11SCL-iF boards limited to 64GB by virtue of only two DIMM slots, but that's a board design limit rather than architectural.

There's also really nothing stopping you from going X12 other than resetting your expectations to be realistic about the potential for problems and remediations. For example, I've got a compelling need to have a few workloads with access to recent Intel QuickSync, which is Really Very Hard to find. Been looking at an SYS-110T-M with E-2388G as one of the few plausible options, because Intel normally doesn't package QuickSync with the Xeons, except for a limited selection of workstation CPU's. Finding stock is an issue, and overall the frustration factor with X12 means lots of careful vetting of stuff with VMware's HCL and other joyful crap.

Thanks so much, that cleared a bunch up actually. Interestingly I found a X10 board that surpassed what I could find in the X11 range, any thoughts?

- MBD-X10SRL-F-B (memory size and SATA port density)
- Xeon E5-2690 V4 (highest range CPU for under $500)
- 2x Supermicro MEM-DR464L-CL01-LR24 64GB DDR4-2400 LRDIMM ECC (128GB total)

God willing that it all comes undamaged and works together. I did the best I can to determine compatibility but god knows there's always something lol

 

[jgreco]

The X10SRL is one of the nicest general purpose boards for X10 and we've sold some as hypervisors in the past. It's pleasant, slot-ful, competent hardware. We run a bunch of X10SRW (Supermicro specialized form factor version of the X10SRL) as well.

The E5-2690v4 should be a decent CPU and should work on that board. We bought this stuff new back in the day and are mostly running E5-1650v3's on them. If your board refuses to boot with the v4 CPU, note that the BIOS may need to be updated in order to boot v4 CPU's; this can be done via the IPMI interface. If you have problems, ASK HERE or contact Supermicro support. The memory you've selected is on the X10SRL's qualified memory list so it is basically guaranteed to work.

If this is for a dedicated NAS, I think you're on a reasonable track. As I noted above, you do not need the latest and greatest hardware. My only concern here might be the choice of CPU. It's a meaty CPU, but doesn't have high clock speeds per core (2.6/3.5). But you get 14(!) of them. My selection for this would have been E5-1650v4 (3.6/4.0, six cores) which is attractive for singlethreaded processes like Samba, but they're often hard to find on the used market, AND the 16xx do not do LRDIMM, so DO NOT do that using your selected memory. The E5-2643v4 (3.4/3.7, six cores) has more cache and is usually easier to find.

I think if you're going to be doing stuff like jails or VM's or other stuff, the 2690's a great choice, but if you want to be able to get the most out of 10G networking and Samba filesharing in the future, stop and think about a 2643.

 

[ISJ]

The X10SRL is one of the nicest general purpose boards for X10 and we've sold some as hypervisors in the past. It's pleasant, slot-ful, competent hardware. We run a bunch of X10SRW (Supermicro specialized form factor version of the X10SRL) as well.

The E5-2690v4 should be a decent CPU and should work on that board. We bought this stuff new back in the day and are mostly running E5-1650v3's on them. If your board refuses to boot with the v4 CPU, note that the BIOS may need to be updated in order to boot v4 CPU's; this can be done via the IPMI interface. If you have problems, ASK HERE or contact Supermicro support. The memory you've selected is on the X10SRL's qualified memory list so it is basically guaranteed to work.

If this is for a dedicated NAS, I think you're on a reasonable track. As I noted above, you do not need the latest and greatest hardware. My only concern here might be the choice of CPU. It's a meaty CPU, but doesn't have high clock speeds per core (2.6/3.5). But you get 14(!) of them. My selection for this would have been E5-1650v4 (3.6/4.0, six cores) which is attractive for singlethreaded processes like Samba, but they're often hard to find on the used market, AND the 16xx do not do LRDIMM, so DO NOT do that using your selected memory. The E5-2643v4 (3.4/3.7, six cores) has more cache and is usually easier to find.

I think if you're going to be doing stuff like jails or VM's or other stuff, the 2690's a great choice, but if you want to be able to get the most out of 10G networking and Samba filesharing in the future, stop and think about a 2643.

Fantastic to hear you're in support of that mobo choice.

Took your advice and I bought a E5-2643 v4, ill be returning the other. My methodology was to use passmark.com to sort by speed, this E5-2643 is half the speed as the E5-2690 but since it is a pure NAS you're totally correct that the core speed is more important. I have several dedicated hypervisors for that role.

I will likely need to use the IPMI to upgrade then but that shouldn't be too hard, thanks fort he heads up! My plan was to just take it to the datacenter and ask the guys for a spare v3 if they had one on hand, its best to leave the rack monkeys alone though as they are busy and easily angered.

 

[jgreco]

I bought a E5-2643 v4,

Great, glad I invested a little time to discuss that, then. I looked on eBay and it looks like all the CPU's have gotten "cheapish." At least considering what their original prices were.

 

[ISJ]

Great, glad I invested a little time to discuss that,

Speaking of that, thank you so much for your help across several posts Jgreco, I truly appreciate your support and experience here sir. With kids and work this has been a lot of effort to keep up and you have helped immensely.

Thanks.

 

[jgreco]

Speaking of that, thank you so much for your help across several posts Jgreco, I truly appreciate your support and experience here sir. With kids and work this has been a lot of effort to keep up and you have helped immensely.

You are certainly most welcome. I invest time here because I've got professional experience and because I get a bit excited when things work out well, so letting me know that it was helpful is the best thanks.

 

[snorpleblar]

Thanks so much, that cleared a bunch up actually. Interestingly I found a X10 board that surpassed what I could find in the X11 range, any thoughts?

- MBD-X10SRL-F-B (memory size and SATA port density)
- Xeon E5-2690 V4 (highest range CPU for under $500)
- 2x Supermicro MEM-DR464L-CL01-LR24 64GB DDR4-2400 LRDIMM ECC (128GB total)

God willing that it all comes undamaged and works together. I did the best I can to determine compatibility but god knows there's always something lol

How'd this build end up turning out for you? Assuming you went with the E5-2643v4.

I'm in the same boat here, waffling between X10/X11. Having a hard time finding any new X10SRLs :( And the X10SRA I did find didn't have any onboard video and no IPMI, tough to troubleshoot if things go wrong.

Not sure if I want to risk "gently used" variants.

 

[Dave Hob]

May I ask what case did you use for this setup?

I am considering same MB and CPU

thank you

 

[Davvo]

Not sure if I want to risk "gently used" variants.

You do, server-grade hardware has a long lifespan and the price saving alone usually makes it worth the shortened lifespan.

 

[jgreco]

You do, server-grade hardware has a long lifespan and the price saving alone usually makes it worth the shortened lifespan.

Most server-grade hardware gets shipped to data centers, installed, sits there for N years, gets pulled out of the rack, and maybe only once or twice is even opened for a tech to paw around inside to add RAM or PCIe cards. The gear is designed with components rated to work 24/7 in harsher environments.

@Davvo is also right about the price savings. I recently commented on costs of used gear in another thread, where I noted that an old X9S* system with 2011-era CPU would run you about $100 used, and probably still has several years of life in it, while a new modern system might cost $1200ish. So the new gear is likely to last quite a bit longer, that's true. Out of a sample size of maybe 70 X9-X10 generation systems under management here, I can think of only two catastrophic failures, one appears to be a power distribution board failure in a 4U chassis, the other appears to be either bad RAM or a CPU in need of reseating in a system that's unfortunately 2500 miles away. Both of those happened around the ten year mark in the equipment's lifecycle. This could be a compelling price savings, but it is certainly a bit of a risk in that the remaining service life of used gear is reduced.

 

[Constantin]

Not at all, its just counter to my expectations and seems to be limited to 64GB in a lot of cases. If you're saying in Q2 2022 that x11 is fine then im sold, just need to find one with 128GB ideally.

That’s really dependent among other things on the CPU in use. For example, my favorite SOHO AFP/SMB dedicated file server board is a X10 and can handle 128GB of RAM. I’m sure there are others with even higher capacities.

Bottom line, SuperMicro has a very wide scope of motherboard designs in every generation ranging from CPU workhorses with tons of cores, memory, etc to dedicated file servers whose CPU capacity may be limited but whose SATA ports are plentiful. It really pays to think about what you want to use the system for and then to optimize the hardware around those use cases.

In my limited experience, the biggest benefit of later (newer) systems is around power consumption - both by the cores as well as the PSU effiency. SM server case PSUs are trivial to upgrade and usually cheaper than the ATX competition once you account for capacity and efficiency. But older CPU generations can be the electrical companies best friend, imposing a considerable load continuously.