Will it FreeNAS - a "Forced" Upgrade during a rebuild.

[omcn7]

Greetings community thank you for clicking on the post :)

My SuperMicro Board X10SLL-F just fried. Not quite sure why. So now I am doing a "Forced" upgrade. My current FreeNAS build will need a new mainboard,CPU, and RAM.

My use case is, a Raidz2 pool pool A w/ ZIL SSD for an iSCSI target as backup of large datasets (VMDKs) for Type 1 hypervisor snapshots (testing ProxMox, ESXi and XCP-NG).
Hypervisor Server is running 11+ VMs on a Dell R710 with 11TB in a raid10 w/ 6x 4TB SAS drives . A second pool pool B for NFS raidz2 w/ ZIL, A third pool pool Craidz mirror as another NFS share.
All the pools use encryption keys

The new hardware needs to be compatible with the following.
Pool-A Raidz2 w/ ZIL
1x 4TB WD blue
1x 4TB seagate desktop HDD
3x 4TB WD red
1x 59GB SATA SSD - ZIL

Pool B RaidZ2 w/ ZIL
4x 2TB WD red
3x 2TB Seagate barracuda
1x 59GB SATA SSD - ZIL

Pool C
2x 3TB Dell Constellation SAS drives

All drives are SATA 5400 RPM except for the two SAS drives which are 7.2k

raidz mirror for boot 16GB usb and supermicro sataDOM

To connect all these drives I have
1x 9220-8i LSI
1x 9210-8i LSI

After spending the better part of two days I have seen that many have suggested:

Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O Suggested mother board
and
Intel Xeon E3-1230 Processors BX80677E31230V6 Suggested CPU

Although it is hard for me to tell if these parts are best for my use case. My old supermicro only had 8 PCI lanes to work with.
Before answering my questions it is important to note the following
1.The motherboard failed while I was upgrading to add the second LSI card.
2. My previous FreeNAS ran for 2 years. I just backed up all the data locally, pool A and Pool B grew in the upgrade and Pool C has yet to be added.
3. This is also when I will be adding ZIL to pool A and Pool B
4. Pool B is encrypted and has all my files and NFS shares set. I have a local backup but it is not organized into the shares and datasets. To reload the pool would take a considerable amount of time.
5. Pool A is encrypted but I have not set up the iSCSI target yet
6. I will not be growing the pools for some time. I hope this system not be EOL before 5 years
7. I just bought 2 quad port NICs for iSCSI, I have not tested that setup yet. I went with quad port because on the 8x PCI limit of the intel C222 chipset with C236 I would have 20 PCI lanes.
8. I only run linux/unix on bare metal, I would like IPMI to work. Supermicro had some outdated java stuff that only ran on windows and you had to turn off every security feature in IE to get it to work, even then it was buggy. I am weary of trusting supermicro after this experience. I have never built with asrock before...

Questions:
Q1. For my use case how much ram do you suggest I go with? What speed? I have 32 GB of DDR3
Q2. After watching a recent video I learned that in a future release encryption before sending to cloud for off-site backup will be available I want to be sure that I have the correct CPU and crypto hardware that supports this. youtube video with @ChrisMoore of ixSystems What CPU do you recommend? https://youtu.be/brfncQ5jPoA
Q3. What mobo do you recommend? Asrock or supermicro?
Q4. For my purposes do you suggest quad NICs or Chelsio 10GB cards? (FreeNAS and my hypervisor server are in the same rack).
Q5. I already have the SSDs, for my use case does ZIL make sense, will it speed up transfers?
Q6. What is the most economical way to meet my use case requirements?

I have done a lot of research and googling. I wanted to reach out and get involved in the community. I would really appreciate your valuable input. Thank you. I read the forum rulez but I am a N00b to this forum (although I have read a lot going back to 2012 when I first started using FreeNAS) if anything is formatted incorrectly, kindly lmk.

 

[joeinaz]

While your motherboard was fried, were you able to test your CPU and memory? If your original CPU and RAM are still good that would be the way I would proceed. If it were me replacing things now, I would stick with SuperMicro and find something that uses DDR3 memory as it is generally more cost effective than DDR4. I currently have an x8, x9 and x10 system. Right now the x9 is my favorite but I am only now testing the x8. I am wrestling with which of my 3 systems to keep...

 

[omcn7]

Right now the x9 is my favorite but I am only now testing the x8

Thanks for your reply :)
What is your favourite aspect of the x9 system?

While your motherboard was fried, were you able to test your CPU and memory?

I wasn't sure "will it FreeNAS" was the correct forum to bring up a hardware fault.. However, since you asked.
When I started the system there was a spark and that smell you never want to smell. Something fried.
I have tested the PSU by swapping that out. On both PSUs the solid amber LED4 (power is supplied) light and the green blinking LED1 light read normal operation according to the owners manual. However they read "normal" even if I remove the cpu and ram from the board, which I find odd. There are no beep codes, although I see there is a small piezo on the board, I can only assume it is there for this purpose? With CPU and ram in place the system will not POST. CPU fan does not spin. I examined the board under a strong (10x) magnifier. The caps look fine, no scorch marks, the only thing I do see is that some of the CPU socket pins are bent.. which is odd because I have not reseated that CPU since I originally installed it.
The RAM I am currently testing in my Dell R710.
The CPU, sadly I do not own another board with an LGA 1150 socket to test.
I started an RMA for the x10 Whenever I have a failure like this, I like to explore upgrade possibility. Right now DDR4 is crazy expensive (like you mentioned) and I would like to reuse the DDR3 if I can.

@joeinaz what is your experience with the IPMI on these collection of supermicro boards? I have a rev 1.02 x10 The IPMI ver I am not sure, it's a nightmare interface and I have to roll a crash cart or spin up bloody XP in a VM to view the bios etc.

Can anyone weigh in on their experience with asrock IPMI and a (this is the current RAM I have)>> DDR3 solution? Mainboard and CPU suggestions welcome. ;)

Re: the PCI x8 lanes of the C222 chipset I would appreciate it. It looks to me that with 2x HBA cards (see OP) and a 10GBE card those buses would be over saturated. Perhaps due to the slow speed (5400 RPM in my case for most + ZIL in RaidZ2) the bottleneck is the drives and not the PCIe bus lanes?
Re: ZIL I have read (but not fully Grokked) the ZFS primer and it's info on ZIL. Right now those SSDs are too small for me to put in a laptop so I might as well use them for ZIL. I just do wonder the benefit? I see posts here saying it will speed up iSCSI transfer and is generally good, I don't have a battery solution for them, but the rack is on a UPS so.. hopefully I can shut down the systems in the event of an outage.

Any other advice I would love to hear from yous' guys and gals. Thank you. and thanks again @joeinaz

 

[joeinaz]

Thanks for your reply :)
What is your favourite aspect of the x9 system?


I wasn't sure "will it FreeNAS" was the correct forum to bring up a hardware fault.. However, since you asked.
When I started the system there was a spark and that smell you never want to smell. Something fried.
I have tested the PSU by swapping that out. On both PSUs the solid amber LED4 (power is supplied) light and the green blinking LED1 light read normal operation according to the owners manual. However they read "normal" even if I remove the cpu and ram from the board, which I find odd. There are no beep codes, although I see there is a small piezo on the board, I can only assume it is there for this purpose? With CPU and ram in place the system will not POST. CPU fan does not spin. I examined the board under a strong (10x) magnifier. The caps look fine, no scorch marks, the only thing I do see is that some of the CPU socket pins are bent.. which is odd because I have not reseated that CPU since I originally installed it.
The RAM I am currently testing in my Dell R710.
The CPU, sadly I do not own another board with an LGA 1150 socket to test.
I started an RMA for the x10 Whenever I have a failure like this, I like to explore upgrade possibility. Right now DDR4 is crazy expensive (like you mentioned) and I would like to reuse the DDR3 if I can.

@joeinaz what is your experience with the IPMI on these collection of supermicro boards? I have a rev 1.02 x10 The IPMI ver I am not sure, it's a nightmare interface and I have to roll a crash cart or spin up bloody XP in a VM to view the bios etc.

Can anyone weigh in on their experience with asrock IPMI and a (this is the current RAM I have)>> DDR3 solution? Mainboard and CPU suggestions welcome. ;)

Re: the PCI x8 lanes of the C222 chipset I would appreciate it. It looks to me that with 2x HBA cards (see OP) and a 10GBE card those buses would be over saturated. Perhaps due to the slow speed (5400 RPM in my case for most + ZIL in RaidZ2) the bottleneck is the drives and not the PCIe bus lanes?
Re: ZIL I have read (but not fully Grokked) the ZFS primer and it's info on ZIL. Right now those SSDs are too small for me to put in a laptop so I might as well use them for ZIL. I just do wonder the benefit? I see posts here saying it will speed up iSCSI transfer and is generally good, I don't have a battery solution for them, but the rack is on a UPS so.. hopefully I can shut down the systems in the event of an outage.

Any other advice I would love to hear from yous' guys and gals. Thank you. and thanks again @joeinaz

"What is your favourite aspect of the x9 system?"

The my X9SRL-F system has the following benefits over my X10SAE system:

1. It uses relatively inexpensive DDR3 RDIMMs. The x10 uses DDR3 UDIMMs which almost as expensive as DDR4 DIMMs. I bought 32GB of RDIMM for about $13.
0.
2. Speaking of memory, my x10 has 4 DIMM slots is limited to 32GB of RAM while my x9 has 8 DIMM slots and can go to 512GB. More virtualization options.
3. The older CPUs in the x9 system are also a better value. My socket 2011, 8 core E5-2650 CPU cost about $70. I would spend 3 times that for similar performance in a socket 1150 CPU.
4. My x10 has only 2 PCIe slots bigger than x2. My M1015 and the expander leave me with only x2 and PCI slots limiting options.

my x10 has the following advantages:

1. Digital video; much easier to hookup to my KVM. (this benefit is only useful during the initial setup of the FreeNAS system)
2. Eight on board 6Gb SATA ports; (this is nice if you don't want to purchase a RAID controller) and have 8 or less 6Gb disks. The x9 has 10 but most are 3Gb
3. Supports USB3. My case has USB3 ports I cannot use.
4. Lower power usage; the newer CPUs use less power.

The only bad news is the X9SRL-F is becoming very hard to find.

I hope this helps...

 

[omcn7]

he x10 uses DDR3 UDIMMs which almost as expensive as DDR4 DIMMs

Thanks again for your reply.
I already have 4x8GB=32GB UDIMMS they tested out good! ...Phhheww! I also do not need more than 32GB, my VMs live on my Hypervisor, FreeNAS is a backup target I don't plan on using for VMs or any heavy lifting (video trans coding etc) I only need it to run ZFS iSCSI and NFS stable and secure. The only "heavy lifting" to consider is hardware encryption for the drives. i.e. AES-NI. Unless there is other hardware to consider for the new encryption feature that will hit stable perhaps next year.
Thanks @joeinaz I think for me the X10SAE might be a better choice.

Any other thoughts @joeinaz.. or others?