Help with building a new TrueNAS Server for VMs

[Ericloewe]

those 48 ports 10G switches, i see power consumptions are very high. Is the power consumption based on number of ports used? I hope so, else won't make sense getting those if one is not planning on using lots of the ports

Ports used, transceivers used (for SFP), traffic... But expect a fairly high base power consumption for most such things.

If you've not yet bought into 10GBase-T, SFP+ power consumption is noticeably lower than twister pair. You can also get cheaper switches, particularly if you don't need much/any L3 functionality (e.g. The Mikrotik CRS326-24S+2Q+RM or any of the popular older models of enterprise switches the folks over at the STH forums rave on about, similar to @jgreco's suggestions above).
Even if you have a handful of twisted-pair devices, SFP+ modules for 10GBaseT are easily available these days for well under 100 bucks apiece (think 60-70).

 

[jgreco]

Yeah, the Dell switches are server room or data center gear, and since the ones we're discussing are older, they consume a little more power than recent models. For the ones we have running here in production:

The 8132F/N4032F's:

Code:

sw0#show power-usage-history 1

Sampling Interval (sec)........................ 3600
Total No. of Samples to Keep................... 168
Current Power Consumption (Watts).............. 119.5

Sample Time Since           Power        Power
No.    The Sample           Consumption  Consumption
       Was Recorded         On This Unit Per Stack
                            (Watts)      (Watts)
------ -------------------- ------------ -----------
2220   00:00:15:01          117.7        117.7
2219   00:01:15:02          119.5        119.5
2218   00:02:15:03          117.7        117.7
2217   00:03:15:04          119.5        119.5
2216   00:04:15:05          119.5        119.5
2215   00:05:15:06          119.5        119.5
2214   00:06:15:07          119.5        119.5
2213   00:07:15:08          119.5        119.5
2212   00:08:15:09          117.7        117.7
2211   00:09:15:10          119.5        119.5
2210   00:10:15:11          119.5        119.5
2209   00:11:15:12          119.5        119.5
2208   00:12:15:13          117.7        117.7

Code:

sw1#show power-usage-history 1

Sampling Interval (sec)........................ 3600
Total No. of Samples to Keep................... 168
Current Power Consumption (Watts).............. 121.4

Sample Time Since           Power        Power
No.    The Sample           Consumption  Consumption
       Was Recorded         On This Unit Per Stack
                            (Watts)      (Watts)
------ -------------------- ------------ -----------
2457   00:00:37:42          121.4        121.4
2456   00:01:37:43          123.2        123.2
2455   00:02:37:44          121.4        121.4
2454   00:03:37:45          121.4        121.4
2453   00:04:37:46          121.4        121.4
2452   00:05:37:47          121.4        121.4
2451   00:06:37:48          121.4        121.4
2450   00:07:37:49          123.2        123.2
2449   00:08:37:50          121.4        121.4
2448   00:09:37:51          121.4        121.4
2447   00:10:37:52          121.4        121.4
2446   00:11:37:53          123.2        123.2
2445   00:12:37:54          121.4        121.4

The units are about 80% port-full.

A Dell 8024F:

Code:

sw0#show system power

Power Supplies:

Unit  Description    Status     Average     Current          Since
                                 Power       Power         Date/Time
                                (Watts)     (Watts)
----  -----------  -----------  ----------  --------  -------------------
1     System       OK            0.1        97.8
1     Internal(R)  OK           56.2        56.1      12/14/2020 05:07:53
1     Internal(L)  OK           39.9        41.6      06/15/2021 02:05:45

This is going to be typical of datacenter switchgear. I bet a hefty percentage of the watts is just the fans to cool the rest.

By way of comparison, the little Mikrotik 8 port SFP+ switch uses about 20W, the spec sheet says max is 23W.

 

[uberwebguru]

Yeah, the Dell switches are server room or data center gear, and since the ones we're discussing are older, they consume a little more power than recent models. For the ones we have running here in production:

So the Dell 8132F at 80% port full is around 120 watts?

Yup the fans, but by now the fans should be smart enough to know when not to go full blast

 

[uberwebguru]

@Ericloewe @jgreco

what router are you guys using to connect to the switches?
I am planning using pfsense as i am tired of using branded router that will ask to get license to enable stuffs and also EOL software version till one buys new model

I initially found this very costly chassis https://www.ebay.com/itm/401194404938 but not sure if others have better recommendation for 1U chassis to use

that was when i planned to get 10G RJ45 switch, but now with plan for 10G SFP+ switch, wondering what others are using

 

[NugentS]

I am planning building a new TrueNAS server that will be used mostly for NFS share for VMs in production
Mentioned production because this will not be some home lab stuffs, will be running mission critical stuffs

I am planning for this, so haven't bought anything yet, so based on responses from this thread then i will go ahead and buy what are recommended
Here is hardware am planning to get:

latest TrueNAS scale(or core?)
1 x Dell PowerEdge R730xd 12 LFF
1 x Intel Xeon E5-2699 v4 22 cores (better option in E5-2600 v4 series?)
4 x 32GB DDR4
4 x 20TB seagate Exos X20 HDD for RAID 10
4 x 10G RJ45 on Dell R730xd connected to Netgear 10G RJ45 switch

to utilize the 4 x 10G on TrueNAS server, plan to use all 4 NICs for VMs, come connect for NIC1 and others to NIC2 etc


so will start with just 4 x 20TB for RAID 10 for a single pool and later add more drives either to the pool or for another pool

Anything i need to start to keep in mind?
What performance do i expect from this?

This will be my very first TrueNAS build. I need a shared storage for VMs, and i have chosen TrueNAS as storage OS of choice

Have you considered, rather than a build your own, asking IX for a NAS instead. That way you get support

 

[uberwebguru]

Have you considered, rather than a build your own, asking IX for a NAS instead. That way you get support

I can build myself, just gathering some helpful details from folks with more years of experience with TrueNAS/storage tech
I am a systems engineer with almost 2 decades experience, so not sure that route fit my needs

 

[uberwebguru]

@Ericloewe @jgreco

what router are you guys using to connect to the switches?
I am planning using pfsense as i am tired of using branded router that will ask to get license to enable stuffs and also EOL software version till one buys new model

I initially found this very costly chassis https://www.ebay.com/itm/401194404938 but not sure if others have better recommendation for 1U chassis to use

that was when i planned to get 10G RJ45 switch, but now with plan for 10G SFP+ switch, wondering what others are using

@Ericloewe @jgreco

 

[jgreco]

What's wrong with getting a cheap Dell R210-ii for $130 and a Chelsio T520-CR for $100 and then just running bsdrp on it? I'm heavy on vlan usage since it's the only way to get dozens of networks, so the one-interface-per-LAN thing would quickly become onerous.

Examples, labs and benchmarks [BSD Router Project]

bsdrp.net

Multiple real interfaces become useful in the data center though. I was just talking with a client last night about realistic alternatives to the Ubiquiti Infinity router, their theoretically gorgeous $2000 hardware-assisted router that can theoretically handle full-table routing, but in practice its bgpd falls over or locks up, so it's useless in anything more than the most basic simplistic use case. I couldn't even get it to just deliver routes to our route reflectors correctly from a single peering session. Pathetic. Other options like Mikrotik's CCR1072 line are great at packet forwarding and basic routing protocols but suck at BGP convergence; a flap shouldn't require ten minutes of convergence.

So I was spitballing about what might be a lower cost but still performant router with dual power and four interfaces, and came up with:

a Dell PowerEdge R330 is maybe $400

Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz, 16gb ecc ddr4 2133mhz | eBay

For sale is a Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz.

www.ebay.com

and a Chelsio T540 is like $250

Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW | eBay

Find many great new & used options and get the best deals for Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW at the best online prices at eBay! Free shipping for many products!

www.ebay.com

but then most people really don't need multiple upstream and peering on separate interfaces.

 

[uberwebguru]

What's wrong with getting a cheap Dell R210-ii for $130 and a Chelsio T520-CR for $100 and then just running bsdrp on it? I'm heavy on vlan usage since it's the only way to get dozens of networks, so the one-interface-per-LAN thing would quickly become onerous.

Examples, labs and benchmarks [BSD Router Project]

bsdrp.net

Multiple real interfaces become useful in the data center though. I was just talking with a client last night about realistic alternatives to the Ubiquiti Infinity router, their theoretically gorgeous $2000 hardware-assisted router that can theoretically handle full-table routing, but in practice its bgpd falls over or locks up, so it's useless in anything more than the most basic simplistic use case. I couldn't even get it to just deliver routes to our route reflectors correctly from a single peering session. Pathetic. Other options like Mikrotik's CCR1072 line are great at packet forwarding and basic routing protocols but suck at BGP convergence; a flap shouldn't require ten minutes of convergence.

So I was spitballing about what might be a lower cost but still performant router with dual power and four interfaces, and came up with:

a Dell PowerEdge R330 is maybe $400

Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz, 16gb ecc ddr4 2133mhz | eBay

For sale is a Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz.

www.ebay.com

and a Chelsio T540 is like $250

Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW | eBay

Find many great new & used options and get the best deals for Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW at the best online prices at eBay! Free shipping for many products!

www.ebay.com

but then most people really don't need multiple upstream and peering on separate interfaces.

First thanks a loot for the info; taking notes
i will look into the recommendations
i am looking for pfsense with GUI to manage
another brand am looking at very closely is mikrotik routerOS
I like their licensing better than the other brands
I can keep upgrading perpetually, which removed one major cons for me already

what i am after really is to have 1 device to do routing and switching for me
I can go 10G SFP+ or RJ45, so depends, I am tending towards 10G RJ45 because of the wiring
SFP+ is cool and all but the wiring cost more and also just not as easy to manage compared to those for RJ45
Anyways like i said i can go 10G SFP+ or RJ45

but what i want is 1 device to do everything because my needs are small

here is post on this https://www.truenas.com/community/t...rect-routing-and-switching-for-servers.99363/

 

[uberwebguru]

So I was spitballing about what might be a lower cost but still performant router with dual power and four interfaces, and came up with:

a Dell PowerEdge R330 is maybe $400

Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz, 16gb ecc ddr4 2133mhz | eBay

For sale is a Dell PowerEdge R330 2.5" 1U Server Xeon E3-1240 v5 3.5ghz.

www.ebay.com

and a Chelsio T540 is like $250

Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW | eBay

Find many great new & used options and get the best deals for Dell Chelsio T540-CR 10Gb SFP+ Quad Port Full Height Network Card - PMXRW at the best online prices at eBay! Free shipping for many products!

www.ebay.com

but then most people really don't need multiple upstream and peering on separate interfaces.

I was kinda looking for front IO 1U servers but they are very rare and that is why that one cost a arm and leg
so now with your suggestions, i am looking at the following

Dell R230 (similar but shorter depth than R330 at 20inch deep)
Intel E3-12600 v5/v6 of choice

Dell PowerEdge R230 1-Socket Rack Mountable 1U Server for sale online | eBay

Find many great new & used options and get the best deals for Dell PowerEdge R230 1-Socket Rack Mountable 1U Server at the best online prices at eBay! Free shipping for many products!

www.ebay.com

10G RJ45 route
Dell QLogic X1TD1
$100 x 2 for total 8 x 10G ports

Dell QLogic FastLinQ 41164 Quad Port 10 Base-T rNDC X1TD1 | eBay

Find many great new & used options and get the best deals for Dell QLogic FastLinQ 41164 Quad Port 10 Base-T rNDC X1TD1 at the best online prices at eBay! Free shipping for many products!

www.ebay.com

or

10G SFP+ route
Dell Chelsio T540-CR
$160 x 2 for total 8 x 10G ports

https://www.ebay.com/itm/184888422729

Issue now is if RouterOS v7 and Intel E3-1200 v5/v6 CPU is good enough for routing and switching

 

[jgreco]

Dell QLogic X1TD1

<doubtfulLook />

Be aware that high performance routing is not just a matter of throwing random cards into a PC and it magically works. Take some time to look at the available resources on the bsdrp site about design and tuning. It has a LOT to do with the selection of card.

 

[uberwebguru]

<doubtfulLook />

Be aware that high performance routing is not just a matter of throwing random cards into a PC and it magically works. Take some time to look at the available resources on the bsdrp site about design and tuning. It has a LOT to do with the selection of card.

Does bsdrp offer GUI for management? Also i have never heard of that project, so support will be limited compared to pfsense
I am also considering buying RouterOS v7 $250 perpetual license for unlimited everything; not bad for me to do

Issue am after is if the CPU can do routing/switching combo, as that is the issue at hand. Doing routing is not issue, as routing is CPU based, but if it is more than enough for my switching needs. I do not move that much honestly, 1G is good enough, only just want the 10G for faster times when migrating VMs between servers

 

[jgreco]

Does bsdrp offer GUI for management? Also i have never heard of that project, so support will be limited compared to pfsense

I have no idea if it offers a GUI as most of us serious datacenter geeks would not configure a router from a GUI. I think that the issue of "support" for pfSense is probably nothing to write home about with respect to serious issues. Most people using pfSense are just building relatively trite networks.

 

[uberwebguru]

I have no idea if it offers a GUI as most of us serious datacenter geeks would not configure a router from a GUI. I think that the issue of "support" for pfSense is probably nothing to write home about with respect to serious issues. Most people using pfSense are just building relatively trite networks.

=> I am also considering buying RouterOS v7 $250 perpetual license for unlimited everything; not bad for me to do

=> Issue am after is if the CPU can do routing/switching combo, as that is the issue at hand. Doing routing is not issue, as routing is CPU based, but if it is more than enough for my switching needs. I do not move that much honestly, 1G is good enough, only just want the 10G for faster times when migrating VMs between servers

 

[uberwebguru]

@jgreco and anyone else

I have a quick question as i am new to this infiniband thingy
So latency seems to be the major thing am worried and am hearing that using infiniband networking will help with that

So lets say i have a switch like mikrotik https://mikrotik.com/product/crs326_24s_2q_rm with 2 x 40G QSFP+ ports
So what am thinking is get Dell R730xd and get 40GB infiniband card, connect that to the 40 QSFP+ port on the switch and then have TrueNAS be consumed servers connected to the same switch via the 10G SFP+ ports. Is this setup good to have reduced latency to consume storage on TrueNAS core setup via infiniband?

Is that an accurate assumption/expectation setup? or what am i missing?

 

[jgreco]

TrueNAS doesn't support Infiniband. See response from mav in

TrueNAS-12.0-U1.1 Infiniband support?

I am testing out TrueNAS Core 12.0-U1.1 and trying to set up a NFSoRDMA share running on top of ZFS on my system.Relevant Relevant hardware is as follows: CPU: Intel Core i7-3930K Motherboard: Asus X79 Sabertooth RAM: 32 GB (4x 8 GB DDR3-1600 Crucial Ballistix Sport I think. I actually don't...

www.truenas.com

Latency is always an issue with storage networks, but at a certain point you get so much stuff stacked on top of each other that you cannot make a lot of improvements just by fixing the network. Infiniband can get you some network-layer improvements, yes, but they're relatively modest in the grand scheme of things. Definitely could be relevant if you had a large NVMe all-flash array and 40G networking and needed the best possible single-client support, but really in most other setups where parallelism is giving you massive amounts of concurrent I/O, it's not that relevant.

 

[uberwebguru]

TrueNAS doesn't support Infiniband. See response from mav in

TrueNAS-12.0-U1.1 Infiniband support?

I am testing out TrueNAS Core 12.0-U1.1 and trying to set up a NFSoRDMA share running on top of ZFS on my system.Relevant Relevant hardware is as follows: CPU: Intel Core i7-3930K Motherboard: Asus X79 Sabertooth RAM: 32 GB (4x 8 GB DDR3-1600 Crucial Ballistix Sport I think. I actually don't...

www.truenas.com

Latency is always an issue with storage networks, but at a certain point you get so much stuff stacked on top of each other that you cannot make a lot of improvements just by fixing the network. Infiniband can get you some network-layer improvements, yes, but they're relatively modest in the grand scheme of things. Definitely could be relevant if you had a large NVMe all-flash array and 40G networking and needed the best possible single-client support, but really in most other setups where parallelism is giving you massive amounts of concurrent I/O, it's not that relevant.

I see no support with TrueNAS yet
Yeah i was only going to use to lower latency
That is the one last thing am thinking of now

But if TrueNAS were to support, is the setup i described the approach to go? I will be able to use the 40G QSFP+ port to connect to the 40G infiniband card? like this one for example https://www.ebay.com/itm/175043214207

 

[jgreco]

Both Infiniband and FC are specialty technologies that have limited use cases, but haven't really found a home in TrueNAS. In the case of FC, your average disk shelf can flood the FC uplink to the HBA pretty easily, making it generally undesirable. Infiniband and FC both share another quirk, in that they are technologies that involve a significant amount of host-side jiggery and specialized knowledge of how to set up the gear. It's not more difficult than IP and ethernet, but it IS another damn thing to have to know arcane bits about.

In both the case of FC and Infiniband, the limited resale market for these things tends to depress used gear prices. For Infiniband, there's a tendency for HPC sites to do "forklift upgrades" and this often leaves lots of not-horribly-old really fast gear available at really attractive prices. If you know what you're doing and how you can use the gear, then you can definitely get some really awesome deals. But Infiniband is not ethernet, and so this comes with some downsides as well.

 

[uberwebguru]

thank to @jgreco
incorporated some of his advice/recommendations and here I am now

Ok decided to get an awesome switch

Switch got 2 x 40G QSFP+ ports

Plan on getting Dell R730xd with 24 + 2 2.5" drive bays, so i can use SSD for much better IOPs
Getting this 2 x 40 QSFP+ card (cant find this on ebay, if you find please reply with link on ebay)

Electronics & Accessories | Dell Australia

Dell electronics & accessories page has monitors, gaming accessories, webcams and popular categories to help quickly locate the right products.

www.dell.com

Getting 870 QVO since it got 8TB and has better endurance of 2880TB (TDW) than 870 EVO which maxes at 4TB


and then with TrueNAS Core

 

[NugentS]

Err, the QVO are crap SSD's and definately NOT what you want to be using for a VMStore