Upgrade alternatives

[eirik.lars1]

I’m thinking about maybe upgrading my 2016 system. This is what I’m running today:

CPU: intel xeon E3 1220 v5
MB: supermicro x11ssm-f
RAM: 32GB Samsung ddr4 2133MHz ecc
Disks: 6x 6TB wd red nas disks configured in raidz2
Case: fractal design R5
Psu: silverstone st55f-g
Boot: Samsung 16gb sata dom
Gpu: quadro p400
10GbE: mellanox connect-x3

I’m running TrueNAS scale and the system is used for personal file storage and as a Plex media server (currently 5 users including myself). I'm running 13 docker applications at the moment including photoprism.

I want to be able to experiment with VMs and maybe run a minecraft server.

I have been looking at different xeons and supermicro boards. But there is a lot to choose from so I'm hoping someone can guide me in the right direction. I was looking at the xeon E-2286G after reading this test. Could this be a reasonable upgrade? I'm thinking to double RAM capacity to 64GB. The disks and case will be reused. PSU will be swapped with a new Seasonic FOCUS GX 750W.

Currently I'm running my docker applications of the main pool. Would it be better to put them on SSDs? I see the new supermicro boards come with M.2 slots. I have a couple of 250GB samsung 840 lying around. If I get a supermicro board with 8 sata slots I could put the 6 disks for the pool and the two SSDs on there and then run the boot drive of the M.2 slot. Or would it be better to run the boot of of the sata ports and put the storage for the applications on the M.2 slot? I'm running photoprism and as I understand it its better to put that on faster storage.

The old system will be moved off-site and reused as a backup server using some old 3TB wd reds that I have.

 

[eirik.lars1]

No one? Anyone running truenas with the E-2286G?

 

[Davvo]

The E-2286G looks like a substantial upgrade from the E3-1220v5, both in performance and price.
Why would you upgrade though? You current configuration can handle up to 64GB of RAM. Are you not getting the desired performances bceause of a CPU bottleneck?
What is in your limits of "reasonable" upgrade?

Ideally, anything can benefit from being in a SSD instead than in a spinning disk. M2 generally has lower latency than SATA SSD. Just make sure your vdevs have enough redundancy: you can lose a boot drive and replace it with minial (proven you have a backup of your configuration) hassle, but you can't do the same for any vdev composed of a single drive (just writing this causes me pain).

As for the MoBo suggestion, other than a (relatively useless) "go with supermicro, and anyway a server-grade board" I don't know. Socket is the same, but I can't understand the naming scheme or the CPU line of the 2286G, so can't verify compatibility. Intel confuses me more that it should.
EDIT: look at the X11SCA-F, should be compatible with most of your requirements (8 SATA ports, integrated graphics, 4 UDIMM slots, M2 slots). And in case it wasn't clear enough, I don't use and have never used a system with the E-2286G.

 

[eirik.lars1]

The E-2286G looks like a substantial upgrade from the E3-1220v5, both in performance and price.
Why would you upgrade though? You current configuration can handle up to 64GB of RAM. Are you not getting the desired performances bceause of a CPU bottleneck?
What is in your limits of "reasonable" upgrade?

Ideally, anything can benefit from being in a SSD instead than in a spinning disk. M2 generally has lower latency than SATA SSD. Just make sure your vdevs have enough redundancy: you can lose a boot drive and replace it with minial (proven you have a backup of your configuration) hassle, but you can't do the same for any vdev composed of a single drive (just writing this causes me pain).

As for the MoBo suggestion, other than a (relatively useless) "go with supermicro, and anyway a server-grade board" I don't know. Socket is the same, but I can't understand the naming scheme or the CPU line of the 2286G, so can't verify compatibility. Intel confuses me more that it should.
EDIT: look at the X11SCA-F, should be compatible with most of your requirements (8 SATA ports, integrated graphics, 4 UDIMM slots, M2 slots). And in case it wasn't clear enough, I don't use and have never used a system with the E-2286G.

Thank you for your reply! I was 100% sure that my current system could only support 32GB of RAM. After reading your post I had to recheck and sure enough, both the E3-1220v5 and my motherboard support up to 64GB RAM.

I wanted to upgrade because with my current 32GB RAM I don't have enough for running more apps or a VM.

This changes things. Now I'm thinking that I could go for a cheaper RAM upgrade and keep the current system. I would however like to utilize my two SSDs for app storage. In that case I guess I would need to get a SAS-controller to get more connectivity? I have no experience with this, but I have read through the hardware recommendation guide. Seems like I have to get an LSI/Avago/Broadcom SAS controller.

Would it be better to put the pool on the controller or to put the boot/app storage on the controller? I'm already running the GPU and 10GbE on PCI, so don't know if this will create some bottleneck?

So you would put the two SSDs in a mirror configuration for redundancy, correct?

 

[Davvo]

Thank you for your reply! I was 100% sure that my current system could only support 32GB of RAM. After reading your post I had to recheck and sure enough, both the E3-1220v5 and my motherboard support up to 64GB RAM.

I wanted to upgrade because with my current 32GB RAM I don't have enough for running more apps or a VM.

I suspected so, glad I could help.

This changes things. Now I'm thinking that I could go for a cheaper RAM upgrade and keep the current system. I would however like to utilize my two SSDs for app storage. In that case I guess I would need to get a SAS-controller to get more connectivity? I have no experience with this, but I have read through the hardware recommendation guide. Seems like I have to get an LSI/Avago/Broadcom SAS controller.

Yes, you would need an HBA since you currently use 7 out of the 8 SATA connectors of your motherboard. Please read the following resources.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

Would it be better to put the pool on the controller or to put the boot/app storage on the controller? I'm already running the GPU and 10GbE on PCI, so don't know if this will create some bottleneck?

With a solid HBA there should be basically no difference. With enough redundancy (generally mirrors) you can do things like splitting the drives in order to create additional redundancy in case the card fail, but it's generally not usable for RAIDZ layouts.
Yes, your motherboard PCIE slots could be an issue. You have two PICE x8 (one in x16 slot) and two x4 (both in x8 slots) lines (all is PCIE 3.0), which depending on your situation could cause problems. However, I'm no expert here so maybe wait for someone like @jgreco to have a look.

So you would put the two SSDs in a mirror configuration for redundancy, correct?

Absolutely, in my system they would probably form a separate pool as well. You can get without redundancy if you don't really care about your VMs and dockers, but I higly doubt this is your case.

I recommend you reading this resource as well if you haven't already.

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

 

[eirik.lars1]

I suspected so, glad I could help.

Yes, you would need an HBA since you currently use 7 out of the 8 SATA connectors of your motherboard. Please read the following resources.

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

With a solid HBA there should be basically no difference. With enough redundancy (generally mirrors) you can do things like splitting the drives in order to create additional redundancy in case the card fail, but it's generally not usable for RAIDZ layouts.
Yes, your motherboard PCIE slots could be an issue. You have two PICE x8 (one in x16 slot) and two x4 (both in x8 slots) lines (all is PCIE 3.0), which depending on your situation could cause problems. However, I'm no expert here so maybe wait for someone like @jgreco to have a look.

Absolutely, in my system they would probably form a separate pool as well. You can get without redundancy if you don't really care about your VMs and dockers, but I higly doubt this is your case.

I recommend you reading this resource as well if you haven't already.

Introduction to ZFS

This is a short introduction to ZFS. It is really only intended to convey the bare minimum knowledge needed to start diving into ZFS and is in no way meant to cut Michael W. Lucas' and Allan Jude's book income. It is a bit of a spiritual...

www.truenas.com

Thank you again. I'm looking at the motherboard manual now and PCIe slot 6 and 7 where I'm running the 10GbE ethernet card and p400 GPU are CPU slots, while slot 4 and 5 are PCH slots. Does this mean that these slots share bandwidth with onboard LAN, onboard SATA, USB etc.?

 

[eirik.lars1]

I will do some more research on SAS controllers, but they seem pretty expensive (looked up some LSI cards real quick). What would be the cheapest card I could get to achieve what I want and still be within recommended hardware?

 

[Davvo]

Thank you again. I'm looking at the motherboard manual now and PCIe slot 6 and 7 where I'm running the 10GbE ethernet card and p400 GPU are CPU slots, while slot 4 and 5 are PCH slots. Does this mean that these slots share bandwidth with onboard LAN, onboard SATA, USB etc.?

According to this thread (and the diagram) yes.

https://www.truenas.com/community/threads/can-i-add-a-10gbe-card-to-my-server.82122/

I will do some more research on SAS controllers, but they seem pretty expensive (looked up some LSI cards real quick). What would be the cheapest card I could get to achieve what I want and still be within recommended hardware?

What's a Good Hard Drive Controller Card?

Hi, I'm building a FreeNAS system with about 8 hard drives. They are 6gbs drives. I've been reading and reading and can't seem to get a straight answer on which card to use. If someone could point me to a ready to go card for FreeNAS 11 using zfs that would be great. I'd rather buy it off...

www.truenas.com

You can get OEM LSI cards (Dell H200, H310, IBM M1015, etc) in the $30 range on eBay if you shop carefully and patiently.

 

[eirik.lars1]

According to this thread (and the diagram) yes.

https://www.truenas.com/community/threads/can-i-add-a-10gbe-card-to-my-server.82122/

What's a Good Hard Drive Controller Card?

Hi, I'm building a FreeNAS system with about 8 hard drives. They are 6gbs drives. I've been reading and reading and can't seem to get a straight answer on which card to use. If someone could point me to a ready to go card for FreeNAS 11 using zfs that would be great. I'd rather buy it off...

www.truenas.com

Ordered two identical sticks of ram to the ones I already have. That will get me to 64GB. Also ordered an IBM 1015 HBA. Total was 400 USD. Lets see how it goes.