New server - setup recommendations, ideas.

[-Adam-]

Hi!

It's been a while that I posted. My first setup (Intel® Xeon(R) CPU E31220 @ 3.10GHz, Intel® Server Board S1200BTS, 4x4GB 1333MHz Kingston ECC Memory, WD RED's) is working flawlessly for couple of years now. We use it for general document stuff, photos and base of our cloud system. In meantime I have built another setup, strictly as a server for photographs (Supermicro X11SSL-F + Pentium G4400 + Samsung (2Rx8) 288-pin DDR4 Unbuffered DIMM with ECC 1Gx72 (2Rx8), PC4-19200 + 6x3TB WDRED) which is also doing its job perfectly for over a year.

I'm very grateful for many guides that I found here and for your previous help which I got here, in this forum!

Now it is time to build a replacement for the first build which was actually a hardware recovery from different server and a test for first FreeNAS installation!

Principals:
- I would like to build a machine that will last couple of years and to be as much upgradable in terms of efficiency as possible in the future.
- The plan is to install the newest 11.2 v. of FreeNAS.
- Usage: mostly CIFS and AFP shares
- no jails, no virtualization (for this we have a second server)
- daily RSYNC backup to current FreeNAS (current setup will be wiped, reinstalled and configured as a backup solution)

Here is my hardware picks:
1/ Chassis: CSE-825TQC-600LPB - https://www.supermicro.com/products/chassis/2U/825/SC825TQC-600LPB
- thinking about redundant power supply
- 8x 3,5" bays - we will use probably RAIDZ2 4x6TB or 4x4TB ord RAIDZ3 5x4TB or 5x6TB as a start
2/ Board: X10DRi-T - https://www.supermicro.com/products/motherboard/xeon/c600/x10dri-t.cfm
- 2x Xenon processors
- 16x memory slots
- 2x 10Gbit ethernet
3/ Processor: Intel Xeon E5-2620 v4 Octa-core (8 Core) 2.10 GHz
- The plan is to install one and have an option to upgrade with second in the future
4/ Memory: Samsung DDR4-2666 CL19 ECC reg. SR
- 4x16GB of ECC RAM
- I have a possibility install a lot more in the future
5/ Disks - WD REDS - all my FreeNAS setups use WD RED HD. Extremely happy!

My concerns:
- I don't know whether more processor(s) power is needed. Will FreeNAS use the power of two xeons or is it an overkill? The price difference between single-processor motherboard is not that much.
- After reading this: https://www.freenas.org/blog/a-comp...-design-part-iii-pools-performance-and-cache/ - I don't want to use caching, but I'm willing to buy more ram if needed. I think it is safer in the long run. Good idea?
- Will FreeNAS take advantage of 10Gbit? I have a 10Gbit switch, rsync will be done between two servers with 10Gbit connections. What about the 1Gbit network? Will the performance be better between in setup like: COMPUTER <-1GBIT->SWITCH<-10GBIT->FREENAS than standard 1GBBIT all the way?

Thanks for your input and time, feel free to recommend your changes!
Adam

 

[MatthewSteinhoff]

1. Two Xeons is overkill in terms of CPU power. Typically, CPU clock speed is a priority over CPUs/threads all else being equal. That said, in some cases, the only way to get the amount of memory you want is additional CPUs.

2. L2ARC and SLOG seem like a really good idea but you must have the right use case to see the benefit. More RAM more better in most cases. At 64G of RAM, you're way over the suggested amount and may not see a meaningful benefit of adding more. (Though I do love me some RAM.)

3. Yes. If you need more than 1G, 10G is the way to go. Without any performance tuning, we see about 6-7G throughput with Intel NICs and low-end Cisco switches. (This is far better than when we had four 1G NICs in a LAG/LACP configuration.)

Other thoughts...

* You don't need RAIDZ3.

* You've only got eight disk bays. From day one, you're using four or five of those bays. If you're using five bays, how much expansion room do you really have? You wouldn't add another RAIDZ2 or Z3 VDEV because you'd be losing way too much capacity to redundancy. Your best bet would be to add a mirror. Does that meet your expansion requirements? I might consider a 12-bay chassis.

* If you're doing RAIDZ2 with just four drives, you'd get the same storage space and twice the performance with a strip of mirrors. That would also make expansion easier because you could add two drives at a time.

* DDR4 is still a wallet stinger compared to DDR3. For just file services - no jails, no VMs - you might be better off going back a generation on the motherboard, CPU And RAM. That'll save a bunch of money and, given just file services, you'll never know the difference, especially if the older chip has a faster clock rate. (Which is why your old 3.1 GHz E3 might seem faster for SMB/CIFS than your fancy E5 at 2.10 GHz.)

* You may not need 10G to meet your performance requirements.

Cheers,
Matt

 

[freedombacon]

I agree with above. Over-specced and over priced. That is a lot of compute power and RAM with not much storage and no VMs. Look at your reports for CPU and RAM on your current server. Are you using it? I'm betting no. That 5 disk Z3 is crazy, even using consumer drives. I don't think you need Z3 until you get to 8+ disks.

 

[-Adam-]

Thanks Matt and freedombacon for your inputs!

1. I didn't know that CPU clock speed is more important than CPUs/threads. Does FreeNAS takes any advantage from 2 instead of 1 CPU? In my case would it be better to get one cpu with let's say 3,5Ghz, E3 Xeon?

2. Noted. Everybody says the same - more ram = better.

3. I'm reaching the 1G limit (ca. 100-110MB/s). I red that I will never reach 10G speed transfer due to the Hard Drive limitations. Is that true?

You are right about RAIDZ3 - 8 bay does not make any sense. RAIDZ3 is safer though due to the 3rd parity (3 disk failure tolerance), but having 2 backups, I think RAIDZ2 is OK.

I'm reading your last comment about DDR, Matt - You think that going back few years in generation (DDR4->DDR3 and E5 -> E3) will have no influence on speed/efficiency? Do I understand you correctly?

I just checked my stats on current setup - you are right - CPU usage is iddle 90% of the time.

So where should I look for improvement to my current setup, when I look for faster transfer rate and general efficiency?

 

[MatthewSteinhoff]

1. SMB/CIFS is single-threaded. A single thread running at 3GHz is going to pump out more data than a single thread at 2GHz. There is some nuance there but, as a rule of thumb, if that's your protocol, high clock speeds matter. This is more true with fewer connections and less true with more connections.

3. When you have a single RAIDZn VDEV in your pool, your throughput is, basically, the speed of the slowest drive. A 6TB WD Red is going to top out at 175 MB/s. So, four drives or eight drives, your RAIDZ2 VDEV is never going to be faster than 175 MB/s.

In a four-drive, 6TB RAIDZ2 VDEV, you have 12 TB of space, 175 MB/s and 100 IOPS (give or take).

In a four-drive, 6B striped mirror VDEV, you have 12 TB of space, 350 MB/s and 200 IOPS (give or take) - roughly twice the performance.

To expand your storage, you add two drives to the striped mirror pool and go from 350/200 to 525/300. Meanwhile, adding four drives as RAIDZ2 to the first configuration brings you to just 350/200, same as the striped mirror you've been running in the meantime.

With just four drives in play and eight slots, I can make a compelling case for a striped mirror.

You think that going back few years in generation (DDR4->DDR3 and E5 -> E3) will have no influence on speed/efficiency?

Efficiency, yes. Performance? Meh. Hard to say. My guess is that your disk configuration is the bottleneck.

You could use netperf and dd to measure network and disk performance individually and find the practical maximums for each. If you have a spare SSD sitting around, you could add it to the server, create a new single-disk pool just for testing and see if you get better network performance from that SSD than you do your conventional drives. You could drop a 10G NIC to the existing server to see if performance improves.

Cheers,
Matt

 

[Chris Moore]

I didn't know that CPU clock speed is more important than CPUs/threads. Does FreeNAS takes any advantage from 2 instead of 1 CPU? In my case would it be better to get one cpu with let's say 3,5Ghz, E3 Xeon?

Actually, because of the potential for delay when one processor needs data held in the RAM of the other processor, it is usually slower to have a multi socket system vs a single socket system. Each socket is a NUMA node.
Some light reading on that if you are interested:
http://frankdenneman.nl/2016/07/07/numa-deep-dive-part-1-uma-numa/
If you can do the job on a single socket system, that is the fastest memory access and FreeNAS is very dependent on memory speed because of the way ARC works.

Noted. Everybody says the same - more ram = better.

That is because of ARC (adaptive replacement cache) which is something ZFS does to make storage faster. If you are not familiar with these things, here is some reading to help you out:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Lots of discussion of ARC on the forum, just look around...

ARC Size/consumption and performance
https://forums.freenas.org/index.php?threads/arc-size-consumption-and-performance.61021/

What are your ARC statistics?
https://forums.freenas.org/index.php?threads/what-are-your-arc-statistics.28122/

I'm reaching the 1G limit (ca. 100-110MB/s). I red that I will never reach 10G speed transfer due to the Hard Drive limitations. Is that true?

That depends on the potential of your drive configuration. We have had a few users that were able to saturate a 10Gb link but it depends on the hardware (processor and RAM) as well as the disks and how the pool is designed.

So where should I look for improvement to my current setup, when I look for faster transfer rate and general efficiency?

Is this the system you want to improve the performance of?

FreeNAS-9.10.X, Intel® Xeon(R) CPU E31220 @ 3.10GHz, Intel® Server Board S1200BTS, 4x4GB 1333MHz Kingston ECC Memory, 2x3TB WD RED (RAIDZ)+ 2x3TB WD RED (RAIDZ)

My first question would be what is the pool layout because I can only guess based on the way this is written that you have two vdevs in one pool?

 

[-Adam-]

Thanks for replies!

What are your ARC statistics?
https://forums.freenas.org/index.php?threads/what-are-your-arc-statistics.28122/

• Put your data type(s) here...
• 8:54AM up 55 days, 19:19, 2 users, load averages: 0.64, 0.60, 0.45
• 2.09TiB / 2.72TiB (MIRROR_2TB) <- old name, should _3TB)
• 2.11TiB / 2.72TiB (MIRROR_3TB)
• 2.70GiB / 14.9GiB (freenas-boot)
• 12.58GiB (MRU: 9.84GiB, MFU: 2.75) / 16.00GiB
• Hit ratio -> 1.72% (higher is better)
• Prefetch -> 6.31% (higher is better)
• Hit MFU:MRU -> 27.49%:33.30% (higher ratio is better)
• Hit MRU Ghost -> 0.89% (lower is better)
• Hit MFU Ghost -> 1.41% (lower is better)

That depends on the potential of your drive configuration. We have had a few users that were able to saturate a 10Gb link but it depends on the hardware (processor and RAM) as well as the disks and how the pool is designed.

Current setup is two separate mirrors. On the next setup I was planning to make one raidz2 from 4disks.

Is this the system you want to improve the performance of?
My first question would be what is the pool layout because I can only guess based on the way this is written that you have two vdevs in one pool?

Correct, on the current setup I have 2 mirrors, 2 pools, not connected:

zpool status -v

pool: MIRROR_2TB
state: ONLINE
scan: scrub repaired 0 in 10h48m with 0 errors on Sun Nov 11 10:48:14 2018
config:

NAME STATE READ WRITE CKSUM
MIRROR_2TB ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/125d0036-5fe2-11e7-b018-001b21b6e94c ONLINE 0 0 0
gptid/6d427c1c-5045-11e7-b973-001b21b6e94c ONLINE 0 0 0

pool: MIRROR_3TB
state: ONLINE
scan: scrub repaired 0 in 10h2m with 0 errors on Sun Nov 4 10:02:23 2018
config:

NAME STATE READ WRITE CKSUM
MIRROR_3TB ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/99a4317f-d800-11e6-a8da-001b21b6e94c ONLINE 0 0 0
gptid/9a650db9-d800-11e6-a8da-001b21b6e94c ONLINE 0 0 0

 

[-Adam-]

Thanks for clearing things up, I will get back to my supplier, and check:
- I think I will go for one socket solution
- check 12 bay system
- go for higher frequency E3 Xeon
- go for 10GBIT ethernet

 

[-Adam-]

Just out of curiosity, I have checked ARC on my second system:

FreeNAS-9.10.X, Intel(R) Pentium(R) CPU G4400 @ 3.30GHz, Supermicro MBD-X11SSL-F-O, 2x8GB 2400MHz Samsung ECC M391A2K43BB1-CRC, 4x4TB WD RED (RAIDZ2) + 2x2TB WD RED (RAIDZ)

• Put your data type(s) here...
• 9:37AM up 5 days, 18:59, 1 user, load averages: 0.23, 0.16, 0.14
• 10.9TiB / 14.5TiB (DREAMPHOTO_16TB_Z2)
• 1.20TiB / 1.81TiB (DREAMPHOTO_MIRROR)
• 1.26GiB / 14.2GiB (freenas-boot)
• 12.45GiB (MRU: 11.93GiB, MFU: 541.70) / 16.00GiB
• Hit ratio -> 65.82% (higher is better)
• Prefetch -> 86.11% (higher is better)
• Hit MFU:MRU -> 65.35%:13.78% (higher ratio is better)
• Hit MRU Ghost -> 0.28% (lower is better)
• Hit MFU Ghost -> 1.92% (lower is better)

pool: NAME1
state: ONLINE
scan: scrub repaired 0 in 8h43m with 0 errors on Sun Nov 18 08:43:48 2018
config:

NAME STATE READ WRITE CKSUM
NAME1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/9e145c6f-4ba1-11e7-8da4-0cc47ae33f42 ONLINE 0 0 0
gptid/9ebac94d-4ba1-11e7-8da4-0cc47ae33f42 ONLINE 0 0 0
gptid/9f5abe85-4ba1-11e7-8da4-0cc47ae33f42 ONLINE 0 0 0
gptid/a00097e6-4ba1-11e7-8da4-0cc47ae33f42 ONLINE 0 0 0

pool: NAME2
state: ONLINE
scan: scrub repaired 0 in 4h18m with 0 errors on Sun Nov 18 04:18:35 2018
config:

NAME STATE READ WRITE CKSUM
NAME2 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/17d6d28d-600a-11e7-8519-0cc47ae33f42 ONLINE 0 0 0
gptid/6783c9e7-b2ae-11e7-b127-0cc47ae33f42 ONLINE 0 0 0

 

[Allan Jude]

- daily RSYNC backup to current FreeNAS (current setup will be wiped, reinstalled and configured as a backup solution)

Do you have a reason for wanting to use rsync to backup the data from one FreeNAS to another? Using ZFS's replication will be much much faster.

 

[-Adam-]

Do you have a reason for wanting to use rsync to backup the data from one FreeNAS to another? Using ZFS's replication will be much much faster.

Never had two FREENAS machines in the same network, so I thought Rsync is the way, thanks for the tip!

 

[Chris Moore]

Correct, on the current setup I have 2 mirrors, 2 pools, not connected:

We recently, just last week, discussed this in another thread.
You might want to read through the discussion here and review the links provided by @jgreco
https://forums.freenas.org/index.ph...s-in-production-environment.71878/post-497598
If you are looking for greater speed, you will need more vdevs in your pool.