vektorprime
Cadet
- Joined
- Oct 20, 2020
- Messages
- 2
Hi all, I wanted to get some feedback and share my experience with FreeNAS.
Thought I'd share my latest build, but first I'll review the previous one. Note these are all ebay, used items nothing but the ooptane drives were new.
My old build:
I house mostly linux and windows VMs using proxmox and a bunch of customized scripts for automation. Originally I used VMWARE but I needed more automation with less effort so I went with proxmox/KVM.
When I tune my TXG settings, with sync on/forced, compression on, I can easily saturate the 10G nics. Unfortunately I did increase my TXG timeout so give me more (insane) iops. My ZIL sits at around 20-50GB of TXG data, which I know can be bad if power is lost but I'll take the risk, at least it also sits on the Optane drive for reply if needed. My hope is that with the new build I can have this down to under 5GB and not mess with the TXG settings.
For the new build:
I haven't ran any bench marks yet or created the pool on the new build, I'll be doing that today and updating the post.
Thought I'd share my latest build, but first I'll review the previous one. Note these are all ebay, used items nothing but the ooptane drives were new.
My old build:
- Dell r720xd (12 SFF 3.5)
- For HBA, I flashed a H310 mini as I wanted to save PCIE slots. Thanks to this forum, this was possible.
- For CPU, originally I had some E5-26xxL CPUs but after some benchmarking and reading I found those might be a bottle neck, so I got 2x E5-2650 v2 @ 2.60GHz .
- For RAM, DDR3 376GB 1333Mhz
- For SLOG I used an Intel Optane 280GB AIX NVME PCIE card. (LOVE IT)
- For L2ARC I used 2 900GB Samsung 9xxP M2 NVME drives. (pretty good, would've been good SLOGs too but the Optane performed way better)
- For the drives I bought 12 used Seagate 4TB on ebay for about $200-300.
- I configured these in a 3 way mirror (3 drives in a mirror instead of 2) because I was worried about all the SMART errors and wasn't sure how long they'd last.
- Unfortunately I didn't use the 2x SSD slots in the back, I wish I had put 2 small SSDs in a raid 1 for the OS since this is a prod storage server.
- For NICs I originally had 2x 2 10G intel NICs, but the drivers weren't that great and I really had to tweak it a lot.
- I ended up trading into a 4x 10G Chelsio NIC which just worked very well.
- Currently storage is served to as just 4x separate networks with NFS3 as proxmox's NFS4 support isn't great.
I house mostly linux and windows VMs using proxmox and a bunch of customized scripts for automation. Originally I used VMWARE but I needed more automation with less effort so I went with proxmox/KVM.
When I tune my TXG settings, with sync on/forced, compression on, I can easily saturate the 10G nics. Unfortunately I did increase my TXG timeout so give me more (insane) iops. My ZIL sits at around 20-50GB of TXG data, which I know can be bad if power is lost but I'll take the risk, at least it also sits on the Optane drive for reply if needed. My hope is that with the new build I can have this down to under 5GB and not mess with the TXG settings.
For the new build:
- Dell r720xd (12 SFF 3.5)
- For HBA, I used a flashed H710 Mini, again thanks to this forum, and this guy (https://fohdeesha.com/docs)
- For CPU, I have 2x 35-2680 v2.
- For RAM, 768GB DDR3 at 1333Mhz (24x32gb)
- For SLOG I got another Optane 280GB as it just worked so well.
- For L2ARC I have 2x 2TB Samsung M2 NVME Drives.
- For the drives, this time I found a REALLY good deal on 6 new 4TB HGST drives, and the other 6 I bought at normal price. But at least they were all new.
- The goal here was to not do 3 way mirror so I can get more storage and more write speed/IOPS for the TXG flushing.
- For the NICS, I went to 2x 10Gbps Broadcom NICs as I wanted at least some NIC redundancy.
- Storage is served via 4x 10G LACP (VPC (cisco mlag)), which is dirty to say but I just needed the redundancy and I have enough hosts where I can load balance across the links fine. NFS 4 would've been nice to use but again proxmox doesn't play to nice with it.
I haven't ran any bench marks yet or created the pool on the new build, I'll be doing that today and updating the post.