New NAS with C3000 Series CPU

Status
Not open for further replies.

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Hello Guys,

its a long time ago that I was really active in this forum, mainly in the german threads.
But now its time to get back into it.
My current FreeNAS System is running for over 5 years in 24/7 Mode and I had zero issues so far. Even tho I made some major mistakes in the build process like not buying server hardware. But back in the day I didn`t want to spend so much money.

Now I wanna solve these mistakes once and for all. So its time to get proper hardware and to upgrade to the newest version of FreeNAS.

Usecase now and in the future for my NAS will be the following:

safe Backupstorage
ownCloud
Media Server probably with Plex for the new NAS atm its miniDLNA because of an old FreeNAS 8
maybe Transcoding for media content to watch on the go

I'm using it also for storing pictures, videos and other files for my website. So multiple users need fast access at the same time.
That´s why I want to give LACP a try - I already have a smart managed Switch from D-Link for that. DGS 1210-20. And you never know what else you do along the way. Maybe it´ll get more jobs to do in the future. Virtualisation brings interesting features.

So the plan is the following hardware vise:

Mainboard one of these or 4 core versions:
Supermicro A2SDi-8C+-HLN4F
ASRock Rack C3758D4I
ASRock Rack C3758D4U

8 cores is probably overkill, but better to have them now, than there is no need to upgrade in the future. Or what do you think?
The Marvell LAN controller on the ASRock D4I is a problem?

RAM:
Crucial 64GB Kit (4 x 16GB) DDR4 2400 RDIMM CT4K16G4RFD824A

Should also be safe for the future.


OS Storage:
2x SanDisk Ultra Fit V2 16GB


HDD:

I don´t know yet... in the first place I'll use my old drives. So RAID Z1 5x 2TB WD Red. I already tried an import over a VM. So I created a VM with my old FreeNAS 8, created a pool and imported it in a new FreeNAS 11 VM machine. Should work.
Later maybe RAID Z2 with only 4, but larger drives. Or 3 drive RAID Z1 with larger WD Red`s... IDK.


PSU:
350 Watt 80+ gold Cert. Super Flower from my current setup.

The case is the last thing, but as it just houses everything... it doesn't really matter. I`m also looking into an USV this time.


What do you think of this setup. I like to here your thoughts. Thanks in advance.
Am I missing something - should work flawless in my opinion :D

Update: The case will be a Fractal Design Forcus G mini. It will be modified to hold more HDDs. So additional HDD Cages are coming in and it`ll get soundproofed with king mod mats. Noctua FANs will take care of the airflow.
In my current setup are Noiseblocker eLoops which are starting to make noises after 5 years 24/7. We`ll see if the Noctua FANs hold longer. Probalby use 3 NF-S12A.


Which USV should I get? USB connection for safe shutdown, min. 10 min runtime is important. It doesn`t need to smooth out the power.

Experience with this unit: APC Back-UPS ES 550VA
 
Last edited:

wywywywy

Dabbler
Joined
Jun 1, 2014
Messages
32
Hi.

One thing I can say is that I have the Supermicro A1SRi 8-core and the D-Link DSG 1210, and LACP was super easy no problem at all, but it needs to be done via the console (IPMI).

But Bhyve still seems immature, and I think you might be disappointed if you want to use FreeNAS as a hypervisor. If you want virtualisation, probably look into using ESXi as a host and FreeNAS as a VM, but you will need to get an HBA card to passthrough.

And holy shit memory is so expensive right now.
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Thanks for the info, sounds great. I was a little worried with LACP, but that sounds promising.

Full VM functions maybe down the road. The small and normal virtualisation for ownCloud etc. is handled quite good from FreeNAS.

And yes... RAM... oh boy :D
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
FWIW, I would consider a slightly different supermicro board, the A2SDi-H-TP4F. It is a bit more future-proof with two on-board 10GBe copper interfaces as well as two SFP+ cages. Processor is a 16-core but RAM, etc. will be similar. Room for a M2 interface, powered SATADOM port (orange), 12 SATA interfaces and 1 PCIe slot. Serve the Home had a pretty good review of the thing, including benchmarks.

I am not a huge fan of the C3758D4I because it does not include 10GBe nor a PCIe interface so you could retrofit it later. If I were to go ASRock, I'd choose the C3758D4U 2TP2OP8R for that reason - more expansion options for the future, plenty of interfaces.
 
Last edited:

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Thanks for the Input, I'll have a look into it.
True, 10 GBe would be nice. I just don´t know if I want to spend 400€ more on the double core count and 10 GBe.
I thought 8 cores is already overkill. I mean atm I`ve a 2,6GHz dual core.

So you prefer SATA DOMs for OS rather then two USB 3.0 devices? I mean these small thumbdrives are not even sticking out of the USB port.
 
Last edited:

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
1,828
The consensus here seems to be that SATADOM and SATA SSDs have much better reliability than thumb drives. Having never used a thumb drive, I wouldn't be the best person to opine! My SATADOM on the other hand (sourced along with the rest of the Mini XL from iXsystems) has been holding up fine.

As for the boards, it's just my opinion. I like boards that will last me a long time. If I were you, I would wait 2 years while setting aside the extra cash for a great board. By then, the current RAM shortage (and associated sky-high prices) should be history. Similarly, there will be more cheap switches on the market by then to tie the network together.
 
Last edited:

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Ah ok. I checked out some sata Doms, but for 50€ each I can get a lot of USB Sticks which I planned to use in a RAID config anyway. So if I wanna do that with a sata Dom, than its 100€ straight away. My USB stick is working for 5 years now, but the older FreeNAS Versions are not writing so much on them. So we'll see how they hold up on V 11.

As for SSDs I've to dive in again about caching, l2arc, log etc. But I guess for the current setup and current storage capacity I don´t need any of those.

The network switch is already here, for RAM I still could go for half of the kit and upgrade later down the road. It`s not like I don`t have the money for the 16 core, 10GBe Board - I just don`t want to spend that much on it :D
8 Core and 10GBe would be nice, I`ve to check out more mainboards.
The ASRock D4I has one PCIe interface, so you could add a 10GBe Card - I just can´t find a shop to buy them.
But atm I'm heading more to supermicro anyway....
A2SDi-H-TF this one looks good, too.
 
Last edited:

Nick2253

Wizard
Joined
Apr 21, 2014
Messages
1,633
safe Backupstorage
ownCloud
Media Server probably with Plex for the new NAS atm its miniDLNA because of an old FreeNAS 8
maybe Transcoding for media content to watch on the go

How many users are you looking at for these services?

I'm using it also for storing pictures, videos and other files for my website. So multiple users need fast access at the same time.
That´s why I want to give LACP a try - I already have a smart managed Switch from D-Link for that. DGS 1210-20. And you never know what else you do along the way. Maybe it´ll get more jobs to do in the future. Virtualisation brings interesting features.

You really buried the lead with adding virtualization to the mix. I suggest that if you're serious about virtualizing things, you should look at a dedicated server for that, or you're going to need a much more powerful server.

LACP in practice is far inferior to LACP in theory, especially when the number of simultaneous connections is small. LACP doesn't work by actually balancing loads. It works by statistically balancing loads. So for 100 simultaneous connections and two links, you're pretty likely to get 50 users on one, and 50 users on the other. With two users and two links, you're just as likely to have both users on one link as having them balanced across both links. Unless you're looking at 10 or more simultaneous connections, you're probably better of putting the two links on separate VLANs, one dedicated for web-facing services and one for internal use.

Mainboard one of these or 4 core versions:
I'm not sure why you want an Atom. Based on the price, I would think it would be more cost effective to get a Xeon setup, and that will give you a bunch of processing power for your already planned transcode use, and will more than future proof you for additional uses.

OS Storage:
2x SanDisk Ultra Fit V2 16GB

I would recommend SSD(s) for your boot device. Multiple USB drives works (it's what I'm using now), especially if you are SATA port limited, but a USB drive's reliability is going to be peanuts compared to an SSD, and you're going to find yourself replacing USB drives much faster than you'd expect (I've killed four USB drives in the 2+ years I've been using FreeNAS at home).
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
You really buried the lead with adding virtualization to the mix. I suggest that if you're serious about virtualizing things, you should look at a dedicated server for that, or you're going to need a much more powerful server.
I was thinking along those lines, but you phrased it nicely. @PrincePaul , the board you are looking at is not robust enough (my opinion) for running virtualization.
LACP in practice is far inferior to LACP in theory, especially when the number of simultaneous connections is small. LACP doesn't work by actually balancing loads. It works by statistically balancing loads. So for 100 simultaneous connections and two links, you're pretty likely to get 50 users on one, and 50 users on the other. With two users and two links, you're just as likely to have both users on one link as having them balanced across both links.
This is why I stopped trying with LACP in my home environment. It just wasn't worth the trouble.
I would recommend SSD(s) for your boot device. Multiple USB drives works (it's what I'm using now), especially if you are SATA port limited, but a USB drive's reliability is going to be peanuts compared to an SSD, and you're going to find yourself replacing USB drives much faster than you'd expect (I've killed four USB drives in the 2+ years I've been using FreeNAS at home).
After the first two USB drives died on me, within 6 months of each other, I switched to a mirrored pair of spinning disks (laptop drives) for my boot pool and have had zero trouble with it in the last two years.
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Thanks for the answers.
So normal Backups for 4 Users.
ownCloud for 3-5 Users - but the upload is only 7mbits so that doesn`t create load at all.
MediaServer for 4 TVs (maybe 2 at a time), multiple Music Boxes, Laptops and every so often tablets/phones.
2 workstations are connected to it for website content creation, mostly for pictures and other documents.
Realtime transcoding only for one user at a time.

It could get more but thats how it is right now.

I`ve never tried LACP, so I was hoping that if I´ve 3x 1GBit for LAN that three users could get full or nearly full gigabit speed each. The fourth network interface would than be dedicated for online services like the ownCloud. I`ll definitely give it a try just to see how it is.


Virtualization is not planned from the get go, was just an idea later on, you never know whats coming in the next 5 years.
I`ve chosen the ATOM manly for its performance/efficiency - 25 W TDP is pretty nice. So it would either get the same overall power consumption or a slightly better one with way more power than my current NAS.

Regarding the OS devices I can`t say anything bad about USB devices. Atm I`m running a generic 8 GB USB stick and it runs since the beginning over 5 years ago. I`ll definitely use two in RAID1 or more in a ZFS RAIDZ1, so I can easily fix any problems. But for under 10€ each I can`t see the benefits of using an SSD. But as you said, you can easily change that later down the road if it really creates major problems.

What do you consider a proper virtualization ready cpu?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I`ve never tried LACP, so I was hoping that if I´ve 3x 1GBit for LAN that three users could get full or nearly full gigabit speed each. The fourth network interface would than be dedicated for online services like the ownCloud. I`ll definitely give it a try just to see how it is.
The only way to ensure that each user was directed to a different interface would be to use vlans to force those systems to specific interfaces. In use, any transaction is assigned to any interface in the aggregation group and my experience was that as often as not, only one interface on the NAS was being used while the other was idle.
I`ll definitely use two in RAID1 or more in a ZFS RAIDZ1, so I can easily fix any problems.
FreeNAS will allow you to add more than one device into the boot pool to create a mirror set and a mirror set can have more than two drives, but the GUI only allows it to be a mirror. When I moved from a mirrored pair of USB drives to spinning disk drives I added a HDD, bringing the number of drives in the mirror to three and then added the second hard drive, bringing the number of drives in the mirror to four, then I started removing the USB drives. It was mostly painless.
You can do some virtualization with a small system, but it does limit you. Here is a great build log that you might want to review. It will give you some idea of what can be done.
https://forums.freenas.org/index.ph...sdv-tln4f-esxi-freenas-aio.57116/#post-401275
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Uh thats an interesting thread - that might take a while ;)

I think Mainboard wise it will be one of these two, the price difference ist just 40€:
Supermicro A2SDi-H-TF (2x 10GBe on RJ45)
Supermicro A2SDi-8C+-HLN4F (4x 1GBit)

I can`t find prices for the ASRock Modelles, so... and they also don`t use Intel NICs.

Back in the FreeNAS 8 days running it in an VM was a no go... did that change with the newer versions?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Uh thats an interesting thread - that might take a while ;)

I think Mainboard wise it will be one of these two, the price difference ist just 40€:
Supermicro A2SDi-H-TF (2x 10GBe on RJ45)
Supermicro A2SDi-8C+-HLN4F (4x 1GBit)

I can`t find prices for the ASRock Modelles, so... and they also don`t use Intel NICs.

Back in the FreeNAS 8 days running it in an VM was a no go... did that change with the newer versions?
It does work and I think it is even documented.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
It was working before too, but just not recommended on a production system.

As Samba is single threaded do you think the Atom with its 2,2 GHz is slower than my current 2,6 GHz Celeron on a one user file copy? At the moment the Raid Z1 is capable of 450 MB/s if I remember right - its a long time a go I benchmarked the NAS. So at around 100-110 MB/s copy speed it was the maximum the onboard realtek nic can handle. That shouldn´t be a problem for the Atom, right? Multiple user file copy means multiple instances of samba, too - right? So it would be spread over the cores.

I cant`t decide whether I should go 2x 10 GBe or 4x 1 GBit, because now I just got the DGS 1210-20 Switch without a 10 GBe uplink and was intended to use 3 NICs for LACP and one in a VLAN for ownCloud to separate them. Question is anyway when or if the rest of the network will be upgraded to 10 GBe.
The 1 Gbit solution would probably be fine for over 5-8 years and if I decided to go more in the direction of virtualization than I may have to upgrade anyway...
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
If I were making the call, I would say the 2 x 10GB port option, it gives you more options.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Mhh :D

Another quick question - its better/faster to run owncloud and plex from a jail than over a docker rancher vm, right?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Mhh :D

Another quick question - its better/faster to run owncloud and plex from a jail than over a docker rancher vm, right?
I have not tried it yet, but I understand there are some difficulties sharing the data from the FreeNAS volume to the docker VM. If you search for, 'docker vm' in the forum search tool, you will see over 1500 results where people have already solved some of the issues. I am hanging with the warden jails for now because I already have them configured and they work fine the way they are. I will transition once some more of the bugs are worked out.
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Yeah and the new iocage jails which are coming in 11.2 are not bugfree either.
I played a bit around on the weekend with plex. But to get the network working properly with the bridges and interfaces it needs a bit more console work than with the well automated process of the warden jails.

I have to say the step up vom miniDLNA over to Plex is huuuuge.
 

PrincePaul

Patron
Joined
Feb 26, 2012
Messages
225
Does anyone know a good supermicro reseller in germany who actually sells to private persons?
 
Status
Not open for further replies.
Top