10 Gbit on a budget

[Durandal]

Hi,

I used FreeNAS a couple of years ago with very good results in general. Nowdays my fileserver is using Windows Server but i'm planning to migrate over to FreeNAS again - escpecially when you read of all the new cool features the new releases of FreeNAS has.

I have designed a new computer but one problem is that i have a low budget. Another "problem" is that i want to use 10 Gbit on it.

Chassi: Ri-vier 2U 12Bay
PSU: Be Quiet TFX 300W
Motherboard: SuperMicro X10SLM-F-O
Processor: Intel Core i3 4130T
RAM: 2x Kingston KVR16E11/8 8GB 1600 MHz ECC
HDD Controller: Intel SASUC8I (for the 1 TB disks, already have this)
NIC: SuperMicro AOC-STG-I2 (already have this)

I've been thinking alot about ECC but that's going to increase the cost with about $150-200 which i cannot afford at the moment. The motherboard supports Xeon-CPUs and ECC memory so it's possible for me to upgrade it in the future if i want ECC. One other setup could be to buy a Xeon 1225 v3 CPU now and upgrade to ECC memory later. From where i buy it's around $120 difference between 4130T and 1225v3.

I'm starting with 8x 1 TB disks and 2x 2 TB disks as storage but i will probably migrate to a better controller card (like the IBM M1015) and 4 TB disks within a year. I will run FreeNAS from a 8 GB KingSpec SSD. I'm also planning to play around with iSCSI boot, and iSCSI in general. Since the processor has AES-NI i'm also thinking of using some type of encryption.

What performance can i expect from 8x 1 TBdisks with ZFS in RAIDZ1/RAIDZ2 with a 10 Gigabit card and with similar performance on the "other side"? Have anyone tried AOC-STG-I2 with FreeNAS? I would really like some hints on the setup since it was a while i used FreeNAS. I'm thankful for all type of comments.

 

[warri]

Can't say anything about performance, but just wanted to say that the i3 also supports ECC RAM - no need to upgrade to a more expensive Xeon (well, except if you'll need more CPU power I guess).

 

[Durandal]

Can't say anything about performance, but just wanted to say that the i3 also supports ECC RAM - no need to upgrade to a more expensive Xeon (well, except if you'll need more CPU power I guess).

That was good news! I totally forgot that some i3-models support ECC. Then it's just a question of getting hold of cheap ECC-memory then.

 

[DrKK]

Just for the record, some of the SuperMicro boards **REQUIRE** that you use ECC RAM. Let me say that again: While most of the SuperMicro X9 and X10 boards we talk about for FreeNAS builds do support ECC RAM (obviously), some of the models ***REQUIRE*** ECC RAM, and WILL NOT POST with non-ECC RAM.

So do your research before you ASSUME that a given SuperMicro motherboard will even accept non-ECC RAM.

 

[Durandal]

Just for the record, some of the SuperMicro boards **REQUIRE** that you use ECC RAM. Let me say that again: While most of the SuperMicro X9 and X10 boards we talk about for FreeNAS builds do support ECC RAM (obviously), some of the models ***REQUIRE*** ECC RAM, and WILL NOT POST with non-ECC RAM.

So do your research before you ASSUME that a given SuperMicro motherboard will even accept non-ECC RAM.

Hi - thanks for the info. It seems like the motherboard in the X10-series im checking out the most (SuperMicro X10SLM-F) does not in fact support non-ECC memory. Well, that only gets me to buy those ECC memorys. I'll guess i'd start with one 8 GB module and later on increase to 16/32 GB.

 

[joeschmuck]

So the title of this thread got me to read it. I see you have a single 10gb card for the server, what about the other computers on the network, how much will that cost or do you already have those components?

As for performance, unfortunately I doubt you will find anyone here who can speak from personal experience on using a 10gb network with FreeNAS but based on what little I know, you mentioned encrypting the drives, I would think your first bottleneck could be there so it depends on how much CPU horsepower you purchase. I feel another bottleneck will be the drives you choose and the controller. You are also only using 8GB RAM, that isn't very much and caching by having more available RAM actually contributes a huge speed factor plus I don't know if encrypting/decrypting will use much RAM or not since it's a CPU hardware function.

Anyway, good luck and if you build this system, please post your results and what works better or worse. We all would like to know.

 

[Durandal]

So the title of this thread got me to read it. I see you have a single 10gb card for the server, what about the other computers on the network, how much will that cost or do you already have those components?

As for performance, unfortunately I doubt you will find anyone here who can speak from personal experience on using a 10gb network with FreeNAS but based on what little I know, you mentioned encrypting the drives, I would think your first bottleneck could be there so it depends on how much CPU horsepower you purchase. I feel another bottleneck will be the drives you choose and the controller. You are also only using 8GB RAM, that isn't very much and caching by having more available RAM actually contributes a huge speed factor plus I don't know if encrypting/decrypting will use much RAM or not since it's a CPU hardware function.

Anyway, good luck and if you build this system, please post your results and what works better or worse. We all would like to know.

I have two dual port CX4 PCIe x8 cards and one PCI-X card and one Woven Brocade LB4 switch (48p Gbit, 4p CX4), but i'm planning to buy at least one more network card. One for my workstation, one for the filserver and one for the webserver. I probably going to try to trunk 2 ports from the new FreeNAS filserver when i get it to see if i can gain performance with serveral sessions.

I thought that the FreeNAS community was really into 10 Gbit :). I have read some guides on how to tune FreeBSD for 10 Gbit, but since i don't have a FreeNAS machine online right now i cannot really try it. I also interested if FreeBSD in general have good support for older 10 GbE-cards? The Intel 82598EB controller seems to be supported with the ixgbe driver in FreeNAS at least, but i would love to hear someone who has tried it out.

I will try with encrypted drives, since the processor can speed it up through the AES-NI instructions. But if the encryption becomes the bottleneck i will probably not use it. I'm hoping to get "good" performance of 8 1 TB drives, but not great. I have some other plans with iSCSI with SSD:s though, and i really hope it can perform really good. If i upgrade the 1 TB disks to 4 TB i can probably get alot better sequential performance. As far as i know the IBM M1015 with IT-firmware has a throughput of around 2-2.2 GB/s.

RAM is always a problem, escpecially when it comes to the price of ECC memory. I'll probably start with 8 TB and upgrade within a couple of months to 16/32, i know ZFS loves RAM :)

 

[joeschmuck]

I thought that the FreeNAS community was really into 10 Gbit :).

We all would love 10gb connections if they were free or at a low cost.

For me encryption is not something I need. None of my financial data is without encryption built into the software so encrypting drives is a waste of resources and adds a level of difficulty for use in the FreeNAS environment. If I were a company then I would only encrypt if accessing my data required complex passwords to access it. Data safety/security is a serious issue and just encrypting some hard drives is only one small piece of the puzzle. I'm not saying you don't use complex passwords but I'd say a fair amount of people here using FreeNAS likely do not. I use complex passwords at work, very complex and sometimes difficult to remember depending on which classified system I log into, and especially after I have changed it the day before.

Off my soap box for now.

 

[Durandal]

Motherboard is now ordered, i'm checking diffrent memorys but i will probably go with Kingston 16 GB (2x8GB) 1333 MHz CL9 ECC or 16 GB (2x8GB) 1600 CL11 ECC memory.

However, i'm beginning to doubt the CPU. The Core i3 4130T has 2 cores (4 virtual with HT) at 2.9 GHz and 35W TDP. Is it possible that the CPU can be a bottleneck when using a 10 Gigabit network? I have seen other CPUs such as Core i3 4330T or the Core i3 4340 which a pretty inexpensive but seems to perform pretty good. Is one of those a better choice or does an 4130T give enough power? I rather use a low power CPU but not at the cost of too much performance.

I will try to use as much of the 10 Gbit network as possible with iSCSI and SSD:s for example, but i don't really know how much CPU is needed. I have a small websever with a Celeron 847 CPU right now and it can send around 4-5 Gbit/s before it becomes a bottleneck. My workstation with a Core i7 3820 does not have this bottleneck.

Anyone know what internal performance i can expect from 8-10 1TB disks in RAIDZ2?

 

[cyberjock]

Anyone know what internal performance i can expect from 8-10 1TB disks in RAIDZ2?

That's not a question that has an answer. There's many factors that affect pool performance. And what you get when you first setup the pool is not what you'll get in 6 months. In fact, what you get in 6 months will be a function of the history of the pool, so it could be only slightly slower or you might want to jump off the nearest cliff.

The only good answer you'll get is "you'll find out when you build it". I'd expect 300MB/sec or so in a best case scenario though.

 

[joeschmuck]

However, i'm beginning to doubt the CPU. The Core i3 4130T has 2 cores (4 virtual with HT) at 2.9 GHz and 35W TDP. Is it possible that the CPU can be a bottleneck when using a 10 Gigabit network? I have seen other CPUs such as Core i3 4330T or the Core i3 4340 which a pretty inexpensive but seems to perform pretty good. Is one of those a better choice or does an 4130T give enough power? I rather use a low power CPU but not at the cost of too much performance.

I will try to use as much of the 10 Gbit network as possible with iSCSI and SSD:s for example, but i don't really know how much CPU is needed. I have a small websever with a Celeron 847 CPU right now and it can send around 4-5 Gbit/s before it becomes a bottleneck. My workstation with a Core i7 3820 does not have this bottleneck.

As for the CPU bottleneck, I think it's going to come down to the interface between your hard drives and how your pool in configured more than your CPU unless you are encrypting your drives and then the CPU has a huge affect. I feel the CPUs you listed are pretty speedy for file movements. Something you might try once you get a system up and running, create a UFS drive and see how fast you can pass information. This will have no ZFS overhead and should give you a great indicator of the maximum speed your system is capable of. When you throw in ZFS you can expect a slowdown but its a trade for data reliability. This is just my opinion.