What network speed is an Atom C3758 capable of?

[thomas-hn]

Hi,

does someone here know which data transfer speed over 10 GBit/s network (e.g. via NFS protocol) will an Atom C3758 with two RAIDZ2 pools be capable of? If the drives of a single pool can deliver around 150 MB/s, will the system be able to transfer data from both pools at around 300 MB/s over the network or will the CPU be a limiting factor?

Thanks,

Thomas

 

[Constantin]

For HDD based pools, I’d expect the pools to be the limiting factor. A pool with multiple vdevs is likely a much better bet than multiple single-VDEV pools though.

You could consider helping your pool with SSDs (sVDEV) but that carries its own set of risks.

 

[thomas-hn]

For HDD based pools, I’d expect the pools to be the limiting factor. A pool with multiple vdevs is likely a much better bet than multiple single-VDEV pools though.

You could consider helping your pool with SSDs (sVDEV) but that carries its own set of risks.

I want to go with two RAIDZ2 pools, each pool with 4 HDDs. This fits to my storage needs and separates my "Private" and "Multimedia" data on to different Vdevs, which allows later an easier disksize upgrade by only exchanging four drives.
So, if I will get the full HDD speed with an Atom C3758 over a 10 GBit/s network, this should be fine. At the moment I don't want to go with SSD pools, mainly because of the costs. Maybe, I will add a SSD for SLOG later.

 

[NugentS]

Just as an aside - SLOG only really works on iSCSI or NFS. Its not a cache and doesn't do what you probably think it does. Its purely a data safety thing. Oh and any old SSD won't do - it needs something that is very fast, very long endurance and with PLP otherwise its largely useless.

 

[Constantin]

I’d also consider the X10SDV-2C-7TP4F mother board in case you’re not planning on using VMs or transcoding. More PCIe lanes, more SATA slots, and onboard SFP+ at about the same price point as the atom boards from ASRock. The major delta being SM quality and customer support.

If you buy that c3578 via iXSystems (mini xl?) then all is good. IXSystems gives *awesome* customer support - I know since my c2758 motherboard was replaced multiple times by them. But ASRock has a bit of a Uneven support history with direct end users here.

I still don’t understand your description of need as I see your approach losing more space to parity than necessary. A single z3 VDEV will provide more redundancy and 5 volumes of storage vs. your approach. Performance will be slower, for sure, but your stated use case doesn’t suggest a need for multiple vdevs in the first place.

The two sets of data can reside on the same disks but get shared to different users as needed. I have a single pool here and it has all sorts of files on it, which are shared by user, etc.

 

[thomas-hn]

Just as an aside - SLOG only really works on iSCSI or NFS. Its not a cache and doesn't do what you probably think it does. Its purely a data safety thing. Oh and any old SSD won't do - it needs something that is very fast, very long endurance and with PLP otherwise its largely useless.

Thanks for the information, I am already aware of this.

 

[thomas-hn]

I still don’t understand your description of need as I see your approach losing more space to parity than necessary. A single z3 VDEV will provide more redundancy and 5 volumes of storage vs. your approach. Performance will be slower, for sure, but your stated use case doesn’t suggest a need for multiple vdevs in the first place.

The two sets of data can reside on the same disks but get shared to different users as needed. I have a single pool here and it has all sorts of files on it, which are shared by user, etc.

The point is that I need around 16 TB for personal data (so I would go 4x 8TB RAIDZ2) and around 28 TB multimedia data (so I would go 4x 14TB RAIDZ2). This would allow an easier disk upgrade later, because I only have to change the 4 drives of the required Vdev.

 

[Constantin]

I'd counsel against that on two counts.

1. If speed is a consideration as described in your OP, a pool will behave faster with 2 VDEVs (which you can still upgrade as you describe, or add-to with another VDEV) than two separate pools. With two VDEVs filled with HDDs, I doubt you'll get much above 400MB/s consistently (esp. with smaller files w/o using a sVDEV... and a sVDEV requires thoughtful setup - both in planning and day-to-day operation. )
2. Using a single set of disks is preferable since it's good policy to have at least one qualified cold spare on hand in case a drive fails. Standardizing around a single capacity / type reduces the # of cold spares you have to carry. Some folk here like to live lucky (and with the expectation that HDD prices will continue to fall) but past experience (post Thai floods, for example) suggests that having spares on hand is better than betting on the market being able to supply them as the need arises.

Remember, the VDEVs do not have to consist of the same drive size. Good practice is to have the same # and config in each VDEV, but the drive capacity used in a given VDEV group can vary from VDEV to VDEV. So one VDEV could be a 4TB drive cluster, the next one can use 14TB drives, etc. However, standardizing one one drive size can be helpful from a maintenance / replacement POV.

 

[Etorix]

The point is that I need around 16 TB for personal data (so I would go 4x 8TB RAIDZ2) and around 28 TB multimedia data (so I would go 4x 14TB RAIDZ2). This would allow an easier disk upgrade later, because I only have to change the 4 drives of the required Vdev.

Another point: I hope you have taken into account that a pool should not be more than 75-80% full, and that initial design should aim at 50% full or less.
If you have 16 + 28 TB of data to store right now, you need more drives and/or bigger drives than what you plan.

 

[Constantin]

That’s illogical. You should never rely on the pool to be your only source of the data unless it’s a scratch drive. Assuming you have backups, disconnecting, upgrading, and resilvering a VDEV with larger drives one at a time will give you more redundancy than pulling all four drives and then relying on a backup to reconstitute the data.

The OP mentions that the data cannot be swapped back and forth between pools due to the size of the need. It’s simply not an option.

Having redundant pools in the same enclosure is better than nothing but no panacea because a local point failure can still hose both pools (fire, flooding, whatever)

 

[thomas-hn]

I’d also consider the X10SDV-2C-7TP4F mother board in case you’re not planning on using VMs or transcoding. More PCIe lanes, more SATA slots, and onboard SFP+ at about the same price point as the atom boards from ASRock. The major delta being SM quality and customer support.

If you buy that c3578 via iXSystems (mini xl?) then all is good. IXSystems gives *awesome* customer support - I know since my c2758 motherboard was replaced multiple times by them. But ASRock has a bit of a Uneven support history with direct end users here.

After all your comments here, I will think again about the pool/vdev layout. Thanks for all the input.

Let me direct the discussion back to the mainboard & CPU:

• Has the mainboard mentioned by you a comparable power consumption compared to an C3758 board?
• The Pentium D1508 has only 3MB Cache compared to 16MB of the C3758 and the CPU seems also to be 5 years older (which could result in an increased idle power consumption). Also it only has 2C/4T instead of 8C/8T as the C3758, however, at the turbo frequency the D1508 is 0.4 GHz faster. Would you really prefer an D1508 over the C3758? Also the Atom is used in the official iXSystems builds (which I don't plan to use), but it seems that the Atom is highly tested with TrueNAS. Does a Pentium have any drawbacks against an Atom?
• Do you think one of the smaller Xeons (D, E3, E-21xx) could reach a power consumption at idle or low load compared to the power consumption of the Pentium D1508 or Atom3758?
• To have a low power system is important to me, nevertheless, it should have the 10GBit/s network (just to be able to reach the full pool speed instead to be not limited to the 1 GBit/s speed of the interface).

 

[Constantin]

Has the mainboard mentioned by you a comparable power consumption compared to an C3758 board?

Yes. TDP is 25W but that's a crummy measure. See ServeTheHome for reviews of various boards. They all consume about the same amount of power and the much bigger determinant ultimately is the HDD pool attached to the motherboard. For that reason, I'd also consider going helium-filled for the drives, since they consume less power than air-filled drives.

• The Pentium D1508 has only 3MB Cache compared to 16MB of the C3758 and the CPU seems also to be 5 years older (which could result in an increased idle power consumption). Also it only has 2C/4T instead of 8C/8T as the C3758, however, at the turbo frequency the D1508 is 0.4 GHz faster. Would you really prefer an D1508 over the C3758?

Yes, I would prefer the D-1508 but that's for my use case. Every use case is different and you might decide that transcoding is cool, or that you want to run 20 jails, etc. I'm using my NAS strictly for storage... not everyone does! But here is why I think the D-15xx series of embedded boards are a superior choice vs. Atom for storage-oriented use cases:

1. Because SMB is a single-threaded process, clock speed is the limiting factor, not cores. If you plan on using NFS, etc. the equation changes.
2. Bigger board and more PCIe lanes for the CPU means more room on board like SFP+, mSATA (for L2ARC), two PCIe 3.0x8, One PCI 3.0x4 m.2 (for a SLOG), onboard LSI HBA, etc. Atom boards always feature compromises due to the lack of PCIe lanes from the CPU.

I'm still kicking myself for buying the D-1537 version of this board which has more cores but a slower clock speed and a $500 higher price. Ultimately, I found the cores to be too slow for VMs like Windows running BlueIris, so I could have had potentially faster SMB at a lower price than I do.

• Also the Atom is used in the official iXSystems builds (which I don't plan to use), but it seems that the Atom is highly tested with TrueNAS. Does a Pentium have any drawbacks against an Atom?

Yes, iXsystems offerings are built to a price point and they prefer a Mini-ITX solution. The board I reference needs a larger case. That in turn limits your universe of case options. I am pretty happy with my Lian Li A76 to keep everything cool yet offer a decent amount of access. However, compared to a iXsystems solution, it is huge, much noisier, and costs more. Comes down to how comfortable you are to only have one PCIe slot available for future expansion and how hot you like your stuff to run.

Bigger cases typically give you better options to rig the right fans, etc. My old MiniXL case featured drive temps near 50*C during scrubs with the default BIOS settings. I upgraded the rear fan and also made the fan settings more aggressive. All that helped bring the max temp down to 40*C. My current crop of HDDs in the Lian Li case never go over 33*C during scrubs unless the fans fail. Otherwise, they loaf along around 28*C.

• Do you think one of the smaller Xeons (D, E3, E-21xx) could reach a power consumption at idle or low load compared to the power consumption of the Pentium D1508 or Atom3758?

FWIW, the plug load of my system is around 108Watts with that power being consumed by the HDDs (8x7.1 watts), the SSDs (~5x3 watts), the fans (~6x3 watts) and the balance being a function of the power supply (platinum should be about 95% efficient), suggesting the CPU and motherboard combined consume about 8 watts. Now, are my SSD / fan allowances perhaps high? Potentially, but there simply is not a lot of "room" for the CPU to consume a lot of power.

Some of the posts here certainly have shown other systems performing at the same plug load as mine despite not featuring a low-power embedded processor. The days of CPU-based house warmers are likely behind us unless you're into crypto mining or other CPU-intensive tasks. Pretty much all X10 or later based motherboards from SM offer lower idle power consumption.

• To have a low power system is important to me, nevertheless, it should have the 10GBit/s network (just to be able to reach the full pool speed instead to be not limited to the 1 GBit/s speed of the interface).

This board has on-board SFP+, which is a better low-power choice than copper 10GbE. It's also a better choice for your switch re: power consumption. Copper 10GbE runs hot because it uses a lot of power in comparison.

 

[Constantin]

PS: Someone here deleted their posts so some of my answers may no longer make complete sense.