We are using three amateur servers and need help

[huseyinozsut]

Me and my friend are using three truenas scale servers. One was at my friend's office. It had 12 2TB disks. My preferred zfs setup was 2 Raidz Vdevs with two spares in each. It had 16gb ram (non ecc), intel i5 3570 cpu.

In my home I have 6 4TB disks. My preferred setup is 3 mirrored Vdevs. Again 3570 cpu with 24gb ram.

In my office I have 6 6TB disks. My preferred setup is 3 mirrored Vdevs. With 3550 cpu with 16gb ram.

---------------------------

We are going to change some of the configurations.

First, I don't need two servers anymore. My home server will be enough. In my office my need is minimal. So my plan is to migrate my data at home to 6TB disks. Then combine a server with best parts at hand and use 6tb disks in it and use it at home.

My plan is to use 4tb disks at my friend's system. He allready owns one 4tb disk. We will buy one more. Our plan is to create 4 mirrored vdevs in his system. We will probably increase the ram to 32gb.

On monday X540 10gbit network cards will arrive. In our systems we will start using 10gbit network cards.

-----------------------------

Our concern is not using ecc ram. Here is the thing: THEY ARE EXPENSIVE! Rdimms are very cheap in my country; but udimms are seriously expensive. I can find cheap mainboards that support Udimm; but I could not find a cheap server motherboard that supports Rdimm. Is this that important? Or can you suggest a cheap mainboard that supports rdimm, from same era with i5 3570, have at least two PCI-e 8x (16x is also ok). Form factor is not important for me. I can use any mainboard in my case. So, a non atx is also ok for me. I can print something to mount the card to the case, then cut the back of the case etc...

Second, with 4 mirrored 4tb vdevs, considering that we are not using VMs, will 24gb ram be sufficient? Or should I go with 32gb? With 3 mirrored 6TB vdevs I will go with 32gb. I can't pass 32gb since it is the mainboard's limit in both of the systems.

 

[ChrisRJ]

Why are you so fixed on keeping the CPUs? They are probably the cheapest part to replace.

Also, I am not sure that 10 Gbps makes sense for your overall performance.

You cannot mix UDIMM and RDIMM.

What is your use-case?

 

[huseyinozsut]

Why are you so fixed on keeping the CPUs? They are probably the cheapest part to replace.

Also, I am not sure that 10 Gbps makes sense for your overall performance.

You cannot mix UDIMM and RDIMM.

What is your use-case?

I am not fixed on keeping the CPUs. It is very easy to sell a 3570 and buy a xeon for me. The problem is finding a cheap motherboard that accepts rdimm. If I can find it, rest is very easy. In Turkey rdimms are cheaper than normal desktop dimms. So I will sell my rams and buy rdimms.

1gbit connection is very slow for our use case. In my house I am using 2.5gbit and still it is a little bit slow for us; and 2.5 gbit cards are low quality in general. What we do is simple actually, from server to the main pc we use high speed connection (in this case it will be 10gbit). From server to the rest of the network we use 1gbit connection.

My friend is an architect and has 10TB of archive. When working with architectural programs, he does not copy the files he is using in his design to the desktop; but using them directly from the server. That reduces the data mess.

In my usage, I cut&paste big chunks of data time to time. (900gb to 2TB) And also I use VR headset time to time. The data is huge. If connection is faster, my wait time is lower.

So our main aim to increase the speed for just one pc, and for the rest, 1gbit is enough.
---------------------------------
In a 3 sets of raid 0 setup, I beleive minimum 750 MB/s read speeds, 400MB/s write speeds will be possible. It will probably higher than that. So it means somewhere around 6gbit.

 

[ChrisRJ]

No idea how it is in Turkey specifically. But in general there are plenty of used server boards for little money on eBay. So I assume from your comments that it is indeed rather different in your place.

 

[huseyinozsut]

No idea how it is in Turkey specifically. But in general there are plenty of used server boards for little money on eBay. So I assume from your comments that it is indeed rather different in your place.

I made some deeper research. As I understand, x540 can work on pci-e 2.0 with full speed. Also I own old SAS controllers like LSI 9201-16i or SAS9212 (something else actually, which is very similar but supports disks that are bigger then 2TB).

All of these are actually pci-e 2.0. So, I don't need pci-e 3.0. When I look like this, I can only find two cpu boards. They are not cheap; but not very expensive either. For example, SuperMicro X8DAL-I is 60usd. Super Micro 113M-5 with 2 x E5620 and 32gb Registered ECC ram is 117usd. This is whole system, including 1U case, fans, 550W psu. But I bet this server rack is seriously loud. So I don't know which one to buy. Or should I buy one. If I am going to do this, I will do this for just ECC ram and nothing else.

--------------------------

Another option is a little bit gamble; but I am curious wether it will work or not:

Setting up a server with dual xeon E5-2697 v4.

The second pc at my friend's home-office (his wife is also an architect) is AMD 2700x with 48gb ram and gtx 970 GPU. We can sell these stuff and buy a dual cpu mainboard, dual xeon E5-2697 v4, 128gb ram (or even more), creating a powerfull render machine + HDD server. They need rendering power time to time.

In theory this can become the main pc also. This server can handle 3080ti without bottleneck. Therefore my friend's 5900x system can be his wife's system.

But this is a little bit gamble for us. Especially if we are stuck with server psu (I don't know whether rack server mainboards have ATX psu connection), rack server case, it would be seriously loud.

In this option either truenas, or the system should work on VM.

--------------------------------

I will continue my research.

 

[Stux]

I’m still using Broadwell era hardware personally (Xeon v4).

This is super micro X10 generation and is available cheaply now, supports PCIe3 and DDR4-2400 RDIMMs.

I would not recommend X8 hardware today.

 

[ChrisRJ]

@huseyinozsut , what is your budget?

 

[huseyinozsut]

@huseyinozsut , what is your budget?

If we are going to build a cheap cpu render machine with truenas capability, our budget can go up to 470usd. We have two 700w psu, two cpu water cooler. So, we would need mainboard, cpu, ram.

For beginning 64gb ram would be enough. We can add later. As I understand, each cpu needs it's own ram. So If I want dual channel, I need 4 slots of ram. Or am I wrong? Can I use two 32gb ram in the system and make it dual channel? I am not sure. 2933mhz 16gb Samsung rams are 22 usd here. 2400mhz 16gb rams are 17 usd. 32gb 2400mhz rams are 30 usd. 32gb 2666mhz rams are 42usd.

I found E5-2697 v4 cpus for 88 usd each.

So for 4x16gb = 64gb 2933mhz ram, my cost will be 88 usd. I would prefer to go with 32gb rams; but for the beginning I can buy two of them.
CPU cost is 176 usd.
I have around 100 usd left for mainboard. HUANANZHI X99-F8D ?? Because in Turkey, Super Micro X10 is way over 160usd.

------------------------------

My aim is to compete with 5950x in rendering. I guess I can do that with this setup. If I can find a mainboard which has 8-10 sata ports, I can sell the sas card also. Although I don't know whether I can do pci passthrough to mainboard sata.

 

[asap2go]

If we are going to build a cheap cpu render machine with truenas capability, our budget can go up to 470usd. We have two 700w psu, two cpu water cooler. So, we would need mainboard, cpu, ram.

For beginning 64gb ram would be enough. We can add later. As I understand, each cpu needs it's own ram. So If I want dual channel, I need 4 slots of ram. Or am I wrong? Can I use two 32gb ram in the system and make it dual channel? I am not sure. 2933mhz 16gb Samsung rams are 22 usd here. 2400mhz 16gb rams are 17 usd. 32gb 2400mhz rams are 30 usd. 32gb 2666mhz rams are 42usd.

I found E5-2697 v4 cpus for 88 usd each.

So for 4x16gb = 64gb 2933mhz ram, my cost will be 88 usd. I would prefer to go with 32gb rams; but for the beginning I can buy two of them.
CPU cost is 176 usd.
I have around 100 usd left for mainboard. HUANANZHI X99-F8D ?? Because in Turkey, Super Micro X10 is way over 160usd.

------------------------------

My aim is to compete with 5950x in rendering. I guess I can do that with this setup. If I can find a mainboard which has 8-10 sata ports, I can sell the sas card also. Although I don't know whether I can do pci passthrough to mainboard sata.

You could just go with a Ryzen 3900 or 5900.
They are faster than the xeon.
The 5950X is probably twice as fast as the E5-2697 v4 or do you need the PCIe lanes?
Then go with a used Epyc 7551p for example.
Also good availability and 32 cores should perform at least on par with the xeon.

 

[huseyinozsut]

You could just go with a Ryzen 3900 or 5900.
They are faster than the xeon.
The 5950X is probably twice as fast as the E5-2697 v4 or do you need the PCIe lanes?
Then go with a used Epyc 7551p for example.
Also good availability and 32 cores should perform at least on par with the xeon.

We are talking about dual E5-2697 and I can find them for 70 usd each. I can even buy E5 - 2698 for nearly the same price. So, when talking about cpu rendering, I guess dual E5-2697 will perform near to 5950x.

This setup will be used instead of a 2700x machine. It will be hdd server, second pc at the home office (work pc of my friend's wife) and cpu render machine when needed.

We will either use 970 or 1080ti with this machine. My friend's wife does not need serious GPU power. She does not have any problem with 970. So 1080ti will work with her for years.

We will use 8 4TB HDD. I found a motherboard that has 8 sata ports. If these ports can be used pci-e passthrough, I won't need a sas card.

We will use a 10gbit ethernet card.

So I will need one pci-e 3.0 16x lane and one or two pci-e 2.0 8x lane (one is for ethernet, one is for sas card if needed).

----------------------------------

Main pc at the home office is a 5900x with 64GB ram. We are using 1080ti with it. 1080ti is on limit for my friend's use. We will probably buy either rtx 3080 or 3080ti. So we will use 1080ti in the server or sell it and use 970 in the server. We did not decided that yet.

----------------------------------

So now we have three pc. One is 5900x, other is 2700x, and the last one is the Truenas server, i5 3570. We will reduce the pc count to two and increase the render speed and use ecc ram. Either we will install truenas scale and use windows vm with pci passthrough for gpu, or install windows and use truenas in vm with pci passthrough of sata ports.

The key here is: will two E52697 or 2698 will perform close to 5950x ?