Building a new system...no budget...

[kspare]

With the new features coming out in TrueNAS, we're looking to build a new nas.

The only restriction is we want to stick with 4tb or 8tb spinning disks and needs to be min 10gb max 40gb.

If you had no budget, how would you build out a system?

 

[joeschmuck]

If I had no budget then I'd call iXsystems and tell them that I wanted their flagship server and I wanted it configured with the maximum RAM and I required 40TB of heavily redundant media, and for good measure, I wanted HA support which means, buying two systems.

See, your question is basically unbound from any reason. If you are looking to purchase a system then you must come up with a list of things such as:
1) How much bandwidth do you need 10Gb/1Gb/other for maybe streaming?
2) What type of use will the server get? Database operations, Live data manipulation, Video Editing, simple word processing and file storage?
3) How many people will be using it?
4) Are you planing to run any VM's, if yes, which ones?

The list goes on. I would expect iXsystems to ask you several questions before customizing a system to your needs. While it's nice to dream, I wouldn't want a system like that. I have no place for a loud and hot machine, nor would I want to pay the electrical bill.

 

[kspare]

Lol...thanks for being a buzz kill. I was use trying to have some fun. A lot of people dont want to just call IX and want to build it themselves....just try and have some fun ok..it’s good for ya.

In my case, I’d be using it for hosting virtual machines. File servers and terminal servers.

I already stated it needs to be min 10gb.

NFS only in my case.

But again, what if *you* were building a machine, and IX wasn’t an option to call. How would *you* do it with the features of the new TrueNAS.

What system board, ssd‘s, nvme, disks, hba’s etc etc etc...the list goes on......

 

[Yorick]

In my case, I’d be using it for hosting virtual machines

bhyve is notoriously shit at that.

So. No budget. Sure I'll play.

Proper ESXi server. At the very least 2x44-core Xeon/EPYC, 512GiB or more RAM, 2x10Gb/s NICs. Let's keep in mind though that a SMALL virtual CDNAC takes 44 cores and 512GiB RAM all by itself, so size WAY ABOVE THAT. (Even sillier side note: We have limitless budget. Bribing a Cisco SA to give us a CDNAC virtual ISO and thus jeopardizing their livelihood is totes fair game. We can afford it.)

FreeNAS server with 2x10Gb/s NICs, proper SLOG (mirrored x2), NVMe for ESXi (multiple mirror x3 vdevs as well), multi-target iSCSI, 256+ GiB of RAM.

Can we do 2x40Gb/s tho?

Budget-less enough for you?

Sillyness aside: If you want reasonable ball-park ideas, you gotta answer @joeschmuck's questions. Number of VMs? IOPS per VM? How much network connectivity? Expected TiB of storage for the VMs? We'll want to be in the 10-25% ZFS ball-park, and since we have limitless budget, we'll keep it under 5% for growth.

 

[ornias]

I think 40Gb/s is a bit thin and outdated.
Lets be real if you go 2x44core epyc, you can expect a minimum of SFP28, so lets pick 2x qsfp28...
Even 2xqsfp28 is just about 2-3Gb/s per core... not that high...

 

[Yorick]

I think 40Gb/s is a bit thin and outdated.

That's the spirit!

 

[kspare]

Definitly wouldnb’t be hosting the actual vm’s on the storage server...we use FreeNAS simple as an NFS Datastore.

In my case,, I’d stick with 40gb, no plans to upgrade our switch atm.

 

[kspare]

I think 40Gb/s is a bit thin and outdated.
Lets be real if you go 2x44core epyc, you can expect a minimum of SFP28, so lets pick 2x qsfp28...
Even 2xqsfp28 is just about 2-3Gb/s per core... not that high...

Lol fair enough...trying to set some boundaries!

 

[joeschmuck]

Well if this was a system just for me and I have a one time free gift, I'd want 20TB of RaidZ3 high quality SSDs with 3 spare SSDs, 10Gb/e and a network switch to handle it. I don't need 10Gb/e but it would be nice to play with it. I'd also want the latest Intel Xeon CPU and 512GB ECC RAM. For the OS storage I'd want a pair of Rocket Q 8TB NVME PCIe SSDs because I use ESXi and run everything on top of that. I'm not sure what motherboard could handle all of that, since it's a pipe dream, it's not worth the effort to search for something. One major thing that must occur is the server must be silent. I don't want to hear a single fan running. I have a silent server already except for I can hear the hard drives when they are under heavy load, hence the SSDs in my dream system. And I'd like an unlimited license for all of the Vmware products.

BTW, I don't need 20TB of storage, 10TB is more than enough, and 64GB RAM is fine, all I really need for my current system is a nice new SSD as my datastore for ESXi. Someday soon I'll buy one but I do have another server I need to power on. I might sell it. We will see if the movers broke it first.

 

[ornias]

Well if this was a system just for me and I have a one time free gift, I'd want 20TB of RaidZ3 high quality SSDs with 3 spare SSDs, 10Gb/e and a network switch to handle it. I don't need 10Gb/e but it would be nice to play with it. I'd also want the latest Intel Xeon CPU and 512GB ECC RAM. For

You might want to add 1 or 2 Optane SSD'd for SLOG and/or L2ARC.

 

[Evertb1]

Here I am thinking about the specs for my new ESXi rig (while saving all the money I can). Before you know it this thread gives me al the information I need to spec it out and build it.

 

[HoneyBadger]

In my case, I’d be using it for hosting virtual machines. File servers and terminal servers.

I already stated it needs to be min 10gb.

NFS only in my case.

But again, what if *you* were building a machine, and IX wasn’t an option to call. How would *you* do it with the features of the new TrueNAS.

What system board, ssd‘s, nvme, disks, hba’s etc etc etc...the list goes on......

No budget, NFS only, 4T/8T spinners, primary purpose is hosting VMs?

Well, first of all, I'd ask if we could go all flash instead. "No budget" after all. And you can realistically get away with packing your data in more tightly on NAND because fragmentation is an order of magnitude less impactful to your performance. I'd argue pretty strongly in favour of decent endurance TLC SSDs as a huge step up to give consistent low-latency performance. Next up, how hard a limitation "NFS only" was - the VAAI extensions are still better in the CTL iSCSI world.

But from the physical hardware end, I'd go with a Supermicro prebuilt server on the X11 family (prebuilt so that I can get a pre-validated NVDIMM config) with at least four NVMe U.2 bays up front, preferably more. Start with a pair of Optane P4800X's for metadata, maybe 3-way mirror if I have the slots. Max out the included RAM - preference towards density-per-slot if I can't max it out. L2ARC is Optanes if my back-end is flash, mixed-use or better Intel/HGST/Toshiba SAS if back-end is spinners. (Could also go NVMe for L2ARC but not if it means losing slots or expansion potential on the metadata.)

Obviously there's a sliding scale of where you wind up based on what the actual budget is - but you did say "sky's the limit" after all.