Build check and RAM recommendations for X11SSL build

[nickt]

Hi all,

Having finally accepted that it is time to move on from my beloved (but very dead... for the second time) C2750D4I box, I think I've settled on a replacement build. I'd appreciate any comments on my plan as well as help with RAM selection.

The added challenge for me is that I am in Australia, and availability of parts is either limited, outrageously expensive or both. I will buy from overseas, but only if I absolutely have to. I had a terrible time getting support for my dead C2750D4I(s) from NewEgg and ASRock Rack, mostly because I bought from NewEgg as an overseas import.

Anyway, I'm thinking the following:

• Supermicro X11SSL-F-O mobo (yes, the SSM or SSH would be a better choice, but see previous comments about Australia)
• Xeon E3-1230 v6 CPU
• 2x 16 GB ECC RAM - having trouble, see comments below
• Fractal Design Node 804 case
• Seasonic Focus Gold 450W PSU
• I retain my 6x 3 TB WD red/green drives and 2x Cruzer Fit USB sticks from previous build

ECC RAM is either impossible to get or 100% more expensive than non-ECC RAM in Australia. I think I have no choice but to take a deep breath and buy from NewEgg (Amazon US won't ship to Australia, Amazon Australia is incredibly limited). I can't find any of the models listed in the SuperMicro QVL, but the Crucial RAM checking tool propose the following two options:

• CT16G4WFD8266
• CT16G4XFD8266

Both are available on NewEgg. The only difference I can see is that the second option appears to be a low profile board. Currently, the second option is cheaper than the first. Can anyone confirm these are suitable for use in the X11SSL-F-O?

Lastly a dumb question. When a FreeNAS boot drive is taken out of an old machine and put into a new machine, does it "Just Work"? Do I need to rebuild it, or will FreeNAS / FreeBSD figure out the hardware differences and get on with it?

Any other comments welcome!

 

[Chris Moore]

Sorry,

in Australia.

Many people have had a difficult time getting the hardware they needed at what we might consider reasonable prices in the US.
@Stux any suggestions?

 

[Chris Moore]

When a FreeNAS boot drive is taken out of an old machine and put into a new machine, does it "Just Work"?

Yes, except you will probably need to go in from the console to setup your IP address.

 

[Yorick]

That second RAM option comes up as non-ECC on the Crucial site. Only use that if you like, but not love, your data - as in, this is merely a backup, or you can restore from backup, data loss in case of memory corruption is acceptable. Otherwise choose the ECC stick.

32GB is a serious amount of RAM. What’s your use case? You may be able to run 16GB and be just fine. What 32GB gives you is a bigger ARC and more room for jails and VMs.

E3-1230v6: Perfectly fine CPU. Are you looking to use Plex on this? Edit: x11ssl, my bad. C232 doesn’t support the GPU, an 1225v6 won’t be able to hw transcode. You’d need an ssm or ssh for that. Nope for (If so, consider the 1225v6. Come 11.2-U4, you may be able to use hw transcode, which would make that CPU perform better than the 1230v6 in terms of transcode.)

If you will only file serve, still consider the 1225v6 depending on price. It’ll serve files just as well as the 1230v6 but might save you some cost.

 

[nickt]

Thank you both for the quick replies. It's hard not to get sour about how difficult it is to get good gear here - it really shouldn't be so hard...

@Yorick: I'm fairly sure both models are ECC - I've been back to the Crucial website to check. The only difference I can find between the two is the height of the module - so it seems to me it should be fine. But it would be great to hear that someone else has successfully used either model with an X11SSL/M/H board (CT16G4XFD8266 being the lower height module). The SuperMicro memory QVL is rather short.

Why so much memory? Because I am running a bunch of iohyve VMs. Four currently. One very light weight DNS server. One quite large docker factory (things like nginx reverse proxy with let's encrypt, Crashplan, Nextcloud, InfluxDB, Grafana, VPN server and a few monitoring scripts). Another one running Gitlab (such a memory hog I put it in its own VM). And a fourth with a handful of other services. Plex I still haven't migrated away from the old jail plugin - it still works fine, and I am using it less these days anyway.

I previously only had 16 GB, and Gitlab broke the bank. I started seeing swap getting thrashed, which I'd never seen before - not good for performance. I upgraded to 32 GB and all was well (until the mobo died 2 weeks later...)

The other thing I do quite a bit of is plain file serving. I use my FreeNAS box to store the majority of my raw photos, which I process in Lightroom. SMB performance is key - Lightroom (stupidly) reloads the same raw file quite frequently while working on it - I am hoping the step up from the C2750 to a higher clock rate Xeon will significantly improve SMB performance (single threaded, as I understand it).

The GPU transcode is an interesting point I hadn't considered. In practice, I don't think I do enough to worry. Goodness - the C2750 kept up with all the transcoding jobs I could throw at it.

In any case, very interested for any feedback on the RAM. I'm quite willing to have a go at other brands if there are any recommendations.

 

[Yorick]

Yeah you’re right about both being ECC. Not sure what happened when I first searched for that model number; somehow I got the wrong crucial page.

And you’re right, with that many jails and VMs, as much RAM as you can manage is a good idea. For Lightroom, it sounds like you’re getting a lot of cache misses. You can easily check that. If so, I’d add an L2ARC that’s large enough to hold your Lightroom files, that should help with read. A simple SATA SSD will do the trick. Doesn’t do anything for write, but it sounds like reading is the issue here. Check first, see what your ARC hit rates look like, and then make that decision.

 

[nickt]

Thanks @Yorick. I just noticed your build signature - looks remarkably similar to what I am planning (with a few small differences). In particular, can you comment on the Crucial RAM you are using? Is what you are using the same as either of the models currently available on NewEgg?

 

[Yorick]

My memory is not the same as yours, though similar. 16GiB Unbuffered ECC DDR4 UDIMM, PC4-19400, 2400Mhz. Yours is 2666Mhz memory. The officially supported speed for an e3-1230v6 is 2400Mhz. You can use faster memory; I'm expecting the board to just downclock it. If Crucial says it's compatible, it typically is. I checked the Crucial site - looks like they stopped selling 2400Mhz entirely and just sell the 2666Mhz now.

Crucial changed the model number by now, they do that. bhphoto still lists that kit for reference:

https://www.bhphotovideo.com/c/product/1193720-REG/crucial_ct2k16g4wfd824a_32gb_2_x_16gb.html

In dmidecode you can see that it's Micron memory as expected:

Type: DDR4
Type Detail: Synchronous
Speed: 2400 MT/s
Manufacturer: Micron
Serial Number: 1C23EE00
Asset Tag: DIMMA2_AssetTag(18/17)
Part Number: 18ASF2G72AZ-2G3B1
Rank: 2
Configured Clock Speed: 2400 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V

 

[Yorick]

That Lightroom use case has me intrigued now. Talking to The Best Husband Of All Possible Worlds, raw files are only 5-50MiB in size. Which means they'll fit into ARC, easily. Sometime this coming week, I'll ask him to edit a raw file from local HDD and from FreeNAS, and see how the performance differs.

Is this a Mac or a Windows machine for Lightroom? I'm wondering whether it's write, not read, and the Mac's "sync-write SMB" behavior.

 

[nickt]

Yes indeed - raw files are not that big - although part of my workflow is (occasionally) sending raw files to Photoshop for further editing. This can quickly get into the GBs.

In reality, there are three different demands on the NAS for my photography editing workflow, which will shape optimal system design:

• Frequent reads of small raw files: as you say ARC should be fine. I assume the network link will be the limiting factor
• Less frequent reads of large photoshop files: the network link again will be limiting, but I'm sure I've seen SMB max out a core during file transfer, which suggests CPU single thread performance is important and limiting transfer speed
• Massive transfer of raw files (typically 20 MB each) during import. Typically 100s of files in one go. Involves tons of simultaneous writes AND reads: Lightroom sends each file to the NAS then reads it back to render previews, which are stored locally. Beside the network link, I'm not too sure whether this is more dominated by CPU single- or multi-thread performance - I don't know whether SMB is able to run a thread per file, or it is all done sequentially.

I'm using Macs, sometimes connected by GB ethernet, sometimes by AC WiFi (very close to AP, able to sustain 800-900 Mbps over the air measured by iperf). Doesn't make much difference whether wired or wireless.

The other thing to know: Lightroom is a pig when it comes to performance. It does silly things all the time. Can get completely different results by quitting / restarting Lightroom or even restarting the machine. Makes performance benchmarking very difficult.

I've never investigated this as much as I would like, but as above, I have seen SMB maxing out a core more often than not. I have played with things like forcing a particular SMB version (never made any difference, but I usually kept it at the highest level).

I'm left with only two things when it comes to system build: RAM is good and high single-thread performance in CPU is good.

 

[Yorick]

There's a thread here somewhere about Macs doing sync-writes on SMB. That'll slow things down a ton. You can test whether that's an issue by telling FreeNAS to never honor sync and always do async. If that solves the performance issue, you can either run it that way or install a SLOG and then go back to the default sync behavior.
Here's the thread about strict sync: https://www.ixsystems.com/community/threads/smb-share-slow-and-capped-at-30mbps.63748/#post-456871

SMB read of large files: I'm getting about 100MB/s transfer, as close to gig wire speed as I'll get with protocol overhead. One core busy at 15-18% on this E3. I think you're right, good single-core performance helps. Tested again with two files - each at around 50MB/s, CPU utilization the same as with one file. I'm definitely capped by my gig link.