Building a Premium FreeNas system - Am I on track?

[journich]

Hi,

I'm looking to build my first FreeNas server and budget is not really a concern (within reason).

What I am looking for is 10 hard drives (I'll probably go with 12 or 14 TB drives) with the ability to expand in the future. We create video courses for Youtube and other places, and all our videos are 4K, so you can probably imagine we are eating through hard drive space. I have 2 x Synologsy ds918 with expansion bays each doing about 75 TB of space. I plan to keep one of these to backup FreeNas and to get rid of the other one.

I'm looking at a case like the Fractal design R6 that can fit 10 X 3.5" drives (I have to buy 4 extra brackets to fit all 10 drives).

In addition, it has 2 dedicates 2.5 " brackets which I can use for 2 X SSD's Mirrored for booting.

I've been researching the motherboards and am pretty much decided on a SuperMicro board. Right now I am thinking the X10SRH-F. This supports 10 SATA drives plus the SSD's by the looks, but it has support for up to 4 x 1GB RJ45 Lan ports. I was looking at 1 or 2 10G RJ45 ports.

Does anyone have a recommendation for another SuperMicro board that is well supported by FreeNas and has single or dual 10G RJ45 ports. Or, should I just buy the 10G Nic's? If so, are the Chelsio 10GbE NICs still the way to go?

I'm planning to run FreeNas, but also looking to run one or more VM's. Would probably go with 96 or 128GB ECC Ram.

I'm stuck on the CPU, any advice? I am happy to go with future flexibility by adding a more powerful CPU or even a dual CPU system, especially given I plan to use VM's. I am not sure what is or is not possible, but I'd like to have a render system on this system - Is that a dumb idea? We have tons of videos to render as we havd 39 courses and more being added regularly. We have a dedicated Windows 10 machine that my team can connect to and render videos.

Silence is very important to me. The case is already relatively quiet and is built to reduce noise, and I'd probably replace the fans with Noctua to get as quiet experience as possible. The server would be next to me in a largish office. So I want to do everything possible to make a machine as quiet as possible.

Right now we are using Google File stream for the company data allowing each team to download what they need to use and upload. I plan to setup some sort of process to continually sync the data from google file stream to freenas. So really this is a backup to the live data which resides on google team drive because my team is remote and don't work at my location.

Am I on the right track, is there better options for me to consider. Remember I'm not looking to save money, I want to do things right and allow for future upgradability. As I would be maxing out the case initially, I was thinking I could add a 24 bay expander and SAS card later, right? Or should I be looking at a rackmount server now?

Any help/advise sincerely appreciated.

Thanks

 

[sretalla]

Silence is very important to me. The case is already relatively quiet and is built to reduce noise, and I'd probably replace the fans with Noctua to get as quiet experience as possible. The server would be next to me in a largish office. So I want to do everything possible to make a machine as quiet as possible.

I would recommend looking into using this https://www.ixsystems.com/community/threads/pid-fan-controller-perl-script.50908/ since you will be using a supermicro board (assuming you will also have the IPMI chip).

That script does the best job (which you can adjust as you see fit) of balancing noise and cool drives.

 

[Chris Moore]

If so, are the Chelsio 10GbE NICs still the way to go?

I have used Chelsio 10Gb cards in my FreeNAS systems, but I recently was told that these worked well for another forum member:
https://www.ebay.com/itm/Solarflare...rt-10G-Ethernet-10GbE-PCIe-w-SFP/113540380624
I ordered two of them myself to do some testing with, but I have not had a chance to install them yet.
If you can get Chelsio cards at a good price, like these:
https://www.ebay.com/itm/CC2-N320E-...LP-Server-Adapter-Card-w-SFP-QTY/352569766559
It might be more about personal preference.

 

[danb35]

wrt the Chelsio cards, IIRC, the T3-series cards will be unable to give full 10G bandwidth; they're limited to somewhere around 6 GB/sec as a result of their PCIe implementation. For full bandwidth, you'd want a T4-series or better. Those are (of course) more expensive, but still pretty reasonable on the used market.

 

[Chris Moore]

14 TB drives

Don't do that. The only 14TB models available right now are using SMR (Shingled Magnetic Recording) and it really kills the performance.

 

[jgreco]

I have used Chelsio 10Gb cards in my FreeNAS systems, but I recently was told that these worked well for another forum member:
https://www.ebay.com/itm/Solarflare...rt-10G-Ethernet-10GbE-PCIe-w-SFP/113540380624
I ordered two of them myself to do some testing with, but I have not had a chance to install them yet.

I should warn you that I didn't say I'd used them with FreeNAS. They seem great for ESXi though. Apparently @depasseg and @c32767a have used the SFN5162F in the past on FreeNAS. It does appear to be a vendor-provided/supported driver. But really for $30-$40 you can't go wrong.

 

[Chris Moore]

I am not sure what is or is not possible, but I'd like to have a render system on this system - Is that a dumb idea?

I don't think it will be able to do that. I am not an expert on the subject, but most of the commercial products I have seen employ video cards and that is simply not going to work inside FreeNAS.
Wendel at Level1News did a two part video series on the render system they setup. You might want to take a look:
Part 1:
https://www.youtube.com/watch?v=96e9grnOTZE
Part 2:
https://www.youtube.com/watch?v=60QDV_fHQuE

 

[Chris Moore]

I should warn you that I didn't say I'd used them with FreeNAS

No, it was another forum member: @dvc9 ... Although that member mistakenly called it a "Solarstorm"
https://www.ixsystems.com/community...name-is-no-longer-available.72491/post-520179

But really for $30-$40 you can't go wrong.

I figured I would give it a shot. I just had some coding work in assembly to do this weekend and I didn't get around to doing any FreeNAS experimentation.

 

[Chris Moore]

Silence is very important to me.

That in mind and your later comment about needing more storage and potentially going to a rack mount and for a 24 bay expander for even more storage. Also, your comment about not really needing to be a low-budget solution. I think your should invest in a larger capacity system now so that you can simply add drives in the future.

Am I on the right track, is there better options for me to consider. Remember I'm not looking to save money, I want to do things right and allow for future upgradability. As I would be maxing out the case initially, I was thinking I could add a 24 bay expander and SAS card later, right? Or should I be looking at a rackmount server now?

I am pretty sure there is a vendor in Australia that sells Supermicro servers and I can (from personal experience) recommend this model:
https://www.thinkmate.com/system/superstorage-server-6048r-e1cr60l
Which is this item on Supermicro's site:
https://www.supermicro.com/products/system/4U/6048/SSG-6048R-E1CR60L.cfm

We bought one of these last year to expand our storage infrastructure and I was impressed at how quiet it is. We like it so well that we are working a contract right now to buy a second system. The first is equipped with 60 of the Seagate Exos 10TB drives and 10Gb network interface. I can tell you from when we did the initial data load, it was able to fully fill the pipe on that 10Gb network connection and the server was not struggling at all.
The next one is planned to have 12TB drives instead.
The vendor describes them as, "12.0TB SATA 6.0GB/s 7200RPM - 3.5" - Seagate Exos X12 Series (Helium)".

The vendor we have been working with says that Supermicro configures most of the system at their facility before sending it to the reseller and it is also a requirement that it be sold with some drives already installed. You might want to talk to a local reseller to see what they can do for you. It is a larger upfront cost to buy a large chassis like this, but it can last for a very long time.

When configuring the server, go for higher clock speed, not more cores, it is a dual socket system, so two quad core processors gets you eight cores and 16 threads. That is plenty for FreeNAS in my experience.

 

[Chris Moore]

The server would be next to me in a largish office. So I want to do everything possible to make a machine as quiet as possible.

If you find that the server is making too much noise, I don't think it is loud, but you might be more sensitive, you can use some sound absorbing panels to block the sound, just don't block the airflow.
Here is a suggestion one of the other moderators made:

ghetto acoustic by @jgreco
https://forums.freenas.org/index.ph...-mb-cpu-hds-recommendations.24453/post-255614

 

[journich]

I would recommend looking into using this https://www.ixsystems.com/community/threads/pid-fan-controller-perl-script.50908/ since you will be using a supermicro board (assuming you will also have the IPMI chip).

That script does the best job (which you can adjust as you see fit) of balancing noise and cool drives.

Thanks for that, I wasn't aware of it and it looks pretty useful.

 

[journich]

I have used Chelsio 10Gb cards in my FreeNAS systems, but I recently was told that these worked well for another forum member:
https://www.ebay.com/itm/Solarflare...rt-10G-Ethernet-10GbE-PCIe-w-SFP/113540380624
I ordered two of them myself to do some testing with, but I have not had a chance to install them yet.
If you can get Chelsio cards at a good price, like these:
https://www.ebay.com/itm/CC2-N320E-...LP-Server-Adapter-Card-w-SFP-QTY/352569766559
It might be more about personal preference.

Thanks for that.

These cards appear to have SFP connections. Any idea about ones with RJ45 connectors? I guess I can just buy an adaptor to convert RJ45 to SFP if I had to do it, but it would be nice if they came with RJ45 connectors.

 

[journich]

I don't think it will be able to do that. I am not an expert on the subject, but most of the commercial products I have seen employ video cards and that is simply not going to work inside FreeNAS.
Wendel at Level1News did a two part video series on the render system they setup. You might want to take a look:
Part 1:
https://www.youtube.com/watch?v=96e9grnOTZE
Part 2:
https://www.youtube.com/watch?v=60QDV_fHQuE

Good to know, I have a video rendering machine now, but not using network rendering, plus it's not on a 10GB connection, so that will no doubt improve things doing it that way rather than trying to cram everything into this new server. Thanks

 

[jgreco]

No, it was another forum member: @dvc9 ... Although that member mistakenly called it a "Solarstorm"

Solarstorm is a chipset product line of Solarflare, but its use is somewhat more esoteric than Intel's chipset naming schemes.

 

[jgreco]

Thanks for that.

These cards appear to have SFP connections. Any idea about ones with RJ45 connectors? I guess I can just buy an adaptor to convert RJ45 to SFP if I had to do it, but it would be nice if they came with RJ45 connectors.

No, you generally can't do that. I'm going to make a suggestion here that you go off and read the 10G Networking Primer because it contains a lot of useful information that you seem to not be aware of. Then you should probably head over to

https://www.ixsystems.com/community/threads/sfp-to-rj45-connection.73717/page-2

 

[Chris Moore]

These cards appear to have SFP connections. Any idea about ones with RJ45 connectors? I guess I can just buy an adaptor to convert RJ45 to SFP if I had to do it, but it would be nice if they came with RJ45 connectors.

I am not endorsing a specific item of hardware, but SFP+ 10Gb hardware is much less costly than the BASE-T (RJ-45) gear and it has lower latency. If you want a fast network, there is absolutely nothing wrong with this older hardware. It is only being sold because the place that was using it probably upgraded to something even faster. Here is an example:
https://www.ebay.com/itm/Brocade-BR...-Switch-24x-10-Gb-SFP-Uplink-JMW/232231406925

 

[journich]

No, you generally can't do that. I'm going to make a suggestion here that you go off and read the 10G Networking Primer because it contains a lot of useful information that you seem to not be aware of. Then you should probably head over to

https://www.ixsystems.com/community/threads/sfp-to-rj45-connection.73717/page-2

Thanks, for these, great reading. Makes more sense for me to look for 10G RJ45 LAN Cards given I have the CAT6A Cabling already done.

 

[journich]

I am not endorsing a specific item of hardware, but SFP+ 10Gb hardware is much less costly than the BASE-T (RJ-45) gear and it has lower latency. If you want a fast network, there is absolutely nothing wrong with this older hardware. It is only being sold because the place that was using it probably upgraded to something even faster. Here is an example:
https://www.ebay.com/itm/Brocade-BR...-Switch-24x-10-Gb-SFP-Uplink-JMW/232231406925

I should have mentioned I have CAT6A cabling already done, and the Netgear 708T already installed and working.

I've got three of these NetGear GS110EMX switches around the place connected to that switch for legacy devices, connected via the 10G port over CAT6A. Each of these switches will run its own VLAN.

My iMac PRO is the only device with a 10G LAN connection at present. My main video rendering machine will get a 10G LAN Card and the iMac I use for recording video will get a 10G LAN card as well, plus the new FreeNAS box will have a single or dual 10G LAN Card.

So yeah, I need 10G RJ-45 LAN cards. Do you have a recommendation? Is there a specific chipset I should look for?

Thanks so much for helping.

 

[ginobee2000]

Hi all

 

[Chris Moore]

So yeah, I need 10G RJ-45 LAN cards. Do you have a recommendation? Is there a specific chipset I should look for?

I have used the Intel 10 Gb card in systems at work successfully. I don't use them at home because of the price, but I have over a dozen of them at work. Reliable for me connected to a Cisco switch.

Intel Ethernet Converged Network Adapter X540-T1 - network adapter

Mfg.Part: X540T1BLK

CDW Part: 3439377

UNSPSC: 43201404

$330.45

https://www.cdwg.com/product/Intel-...work-Adapter-X540-T1-network-adapter/3439377?