C2750D4I FreeNas 9.2 Supported or Not Supported?

[cyberjock]

Those tests would be more useful if run on FreeNAS 9.2. The tunables for ZFS on FreeNAS don't quite match FreeBSD from what I've seen in the past. Performance will be different for that reason(and perhaps a little more reliable too).

You'd be better off doing something like a 50GB test file than multiple small tests.

dd if=/dev/zero of=testfile bs=1m count=50000

 

[jgreco]

with 32GB RAM, more like 100GB or more...

 

[Nindustries]

Oh boy! Thanks for the post! I was hoping it'd be compatible.
Any chance you're also going to test over NFS?

It is a bit slow, given the lack of usb 3.0 on the mother board

uh? You mean the startup of the system? meh. :D

 

[forfiter]

Unfortunately, it seems it doesnt install anymore. I am not sure what's the cause of that, but I've dd'd all disks in the machine and tried to reinstall FreeNAS from scratch, what seems to be working, until the reboot, where the booting stops on trying to mount root as on the pictures below.

 

[jgreco]

Scroll up and look to see if your USB boot device is listed (betting: not).

 

[forfiter]

It seems I had issues with labels on my pendrives, and that was the problem with new FreeNAS installations, apparently it doesnt wipe them entirely, and some leftovers from installations of other operating systems can cause issues. I've installed it fine now and I am doing some more benchmarks of the platform. This one looks interesting:

mkdir /mnt/ram && mdmfs -M -S -o async -s 10240m md10 /mnt/ram && dd if=/dev/zero of=/mnt/ram/test bs=1M count=8192

with results like:

8589934592 bytes transferred in 19.693936 secs (436171553 bytes/sec)

what seems quite low, right? Would it be /dev/zero that slow on Avoton, or the memory bandwidth so narrow (these are 1600Mhz Kingstone's KVR16E11-8) or is there some form of FreeNAS optimization that can be done via sysctls?

 

[Cyanon]

Its ~12% lower than my system, (mine ran around 470-485 MBPS vs your 416 MBPS) with the same test. But it doesn't seem like much of a bottleneck for Gigabit, even dual.

I'm not sure what a Xeon would do though, but I thought the Avotons had the same bandwidth.

 

[jgreco]

I'm not sure what a Xeon would do though, but I thought the Avotons had the same bandwidth.

Code:

CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.00-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306e4  Family = 0x6  Model = 0x3e  Stepping = 4
  Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0x9e982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 34359738368 (32768 MB)
avail memory = 33105321984 (31571 MB)
Event timer "LAPIC" quality 600
[...]
# mkdir /tmp/ram && mdmfs -M -S -o async -s 10240m md10 /tmp/ram && dd if=/dev/zero of=/tmp/ram/test bs=1M count=8192 && umount /tmp/ram && mdconfig -d -u md10 && rmdir /tmp/ram
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 9.839446 secs (873009999 bytes/sec)
#

Code:

CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3198.93-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206a7  Family = 6  Model = 2a  Stepping = 7
  Features=0xfe3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS>
  Features2=0x82982203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AESNI,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 8589934592 (8192 MB)
avail memory = 8242987008 (7861 MB)
[...]
# mkdir /tmp/ram && mdmfs -M -S -o async -s 10240m md10 /tmp/ram && dd if=/dev/zero of=/tmp/ram/test bs=1M count=8192 && umount /tmp/ram && mdconfig -d -u md10 && rmdir /tmp/ram
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 9.223455 secs (931314202 bytes/sec)

This looks to be more a CPU core speed test than anything else; the E3 has the higher clock speed and performs better. The Avoton appears to pull in at about half as fast, which is generally consistent with what is known about the performance of that part so far.

I'm unclear as to the point that was being made though.

 

[Cyanon]

Ah, good to know its more of a CPU thing, thanks.

Not much of a point really, Forfiter was just expecting higher numbers.

 

[forfiter]

Here are the numbers when writing from /dev/zero to raidz2 with geli encryption enabled:

[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 367.135149 secs (233972002 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test2 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 406.815534 secs (211150604 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test3 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 411.059273 secs (208970705 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test4 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 435.488609 secs (197248204 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test5 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 386.742930 secs (222109673 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test6 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 420.131136 secs (204458414 bytes/sec)
[root@freenas] ~# dd if=/dev/zero of=/mnt/pool/test7 bs=1M count=81920
81920+0 records in
81920+0 records out
85899345920 bytes transferred in 413.116867 secs (207929893 bytes/sec)

Anyone interested in any other tests in particular?

 

[jgreco]

Ah okay. No coffee yet. I was getting distracted by the filesystem creation, which is pointless overhead. md isn't real fast in any case in my experience.

Since the data path on that E5 ought to be quite fast (12.8GB/sec per channel, eight modules, two on each channel, theoretically 50+GB/sec) ...

Code:

# mdconfig -a -t malloc -o reserve -s 10240m -u md10 && dd if=/dev/zero of=/dev/md10 bs=1M count=8192 && dd if=/dev/zero of=/dev/md10 bs=1M count=8192 && mdconfig -d -u md10
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 3.590026 secs (2392722018 bytes/sec)
8192+0 records in
8192+0 records out
8589934592 bytes transferred in 3.309956 secs (2595180842 bytes/sec)

is a slightly more useful test but still obviously nowhere near the potential memory bandwidth. Too tired to think more about it right now.

 

[cyberjock]

RAM drives in FreeBSD are not exactly record breaking performance. I had problems when trying to benchmark the new AES encryption code FreeNAS uses in 9.2.0 because my bottleneck was the ramdrive. Ironically, my 18 disk pool outperforms the ramdrive on the system in throughput. So what's the point of using a ramdrive for benchmarking? For my benchmarking it was pointless. In fact, I was unable to definitively ascertain what the AES-NI performance limit was. On my system it was nothing less than 1GB/sec or so. I have no clue what the upper limit was because I couldn't get past certain limits.

So yeah, I wouldn't take that ramtest to mean anything except the ramdrive driver isn't very fast.

 

[Nindustries]

Could you try a transfer test using NFS? Or is AFP multithreaded too?

 

[jonnn]

Got a kill a watt?

Please test power if so.

 

[forfiter]

Got a kill a watt?

Please test power if so.

Sorry, I dont own one.

 

[cadamwil]

Does anyone know how much better the performance would be for a Xeon E3 on a motherboard like http://www.asrockrack.com/general/productdetail.asp?Model=E3C224D4I-14S with a C224 chipset and 32 GB of ECC ram spread across 4 sticks? It won't really matter in my use, but if I can build a better box for not much more, I want to do so.

 

[cadamwil]

I would be using either the E3-1270 or E3-1280. FYI.

 

[cadamwil]

It's great that it looks like the Asrock C2750 board works 100%. Tough decisions.

 

[jgreco]

I would be using either the E3-1270 or E3-1280. FYI.

The price premium for the -1280 over the -1230 is kind of ridiculous.

I don't think anybody's done a head-to-head comparison, but generally speaking the Avoton cores are about half the speed of the Xeon E3-1230 cores. Since there are twice as many, for general purpose applications that means that it is very possibly a contender, but for NAS uses we typically notice that core speed is much more important than core count, so my suspicion is that in an application where speed is key, Xeon.

I am still very intrigued by the Avoton in part because it has the potential to go to 64GB RAM. This puts it in an unusual position of potentially being better for ZFS on a NAS, where ethernet speeds limit your overall throughput anyways.

 

[cadamwil]

Yeah, I don't know what I was thinking. I was trying to remember the number and I was way off. It would be E3-1230V3 or E3-1240V3. Is the V just a revision version number?