1. You said you are not keen on peer to peer connection from NAS to workstation..why?
Some of it is network guy snottiness (if I am completely honest). It is certainly lower cost to do it with Twinax cables (big clunky cables with SFP+ ends soldered on). I guess most of it is thinking that it would be a pain to deal with the static IP on two point point networks. If that doesn't phase you, then it probably isn't a big deal. You could also have the FreeNAS box bridge the two connections together and make it a single IP network. This are addresses #2 & #3.
3. I am thinking 8x 8TB or 4x 10TB or 8x 6TB something like that in raidZ2. Would one be faster than the other due to the configuration if I stick with 7200rpm drives?
If you are going to do RAIDZ2 which uses two drives for parity, 6 or 8 physical drives is the right number. Spreading the data across more platters gives you some better speed. RAM that FreeNAS will allocate for the read buffer is also a big win. It will take everything it can get. Rotational speed on the drives is kind of hard to say. Higher RPM drives definitely generate more heat. I can tell you in my system I have two RAIDZ2 vdevs with 8 x 1TB 7.2k SATA drives, and I can pretty consistently get around 8Gb read performance. I do have 128GB RAM which also helps.
4. Taking your RAM advice I will bump that up to 32GB of RAM. Would also installing a NVME m.2 cache drive benefit me, or a couple of SSDs in raid?
The use case for an SLOG (dedicated ZFS intent log) is when you are using something that does synchronous writes like ESXi accessing FreeNAS via NFS. That is how my ESXi hosts use FreeNAS and adding an NVME SLOG was a major jolt in write performance. It depends how you are accessing FreeNAS, and what you are doing with it. A dedicated L2ARC is a major performance jolt for some very specific cases like reading the same file to image computers. I have never done it because that isn't how I how use mine.
5. Here is the build list so far: correct me if you think I will have problems
Chelsio is definitely10G NIC of choice in FreeNAS-land, but I can't remember if the S320 is still supported in the latest versions. I know ESXi dropped support for it which caused me some grief.
6. "You should probably consider a hot spare just in case" whats a hot spare?
7. "A extra HDD as a CYA wouldn't be too much of an extra investment" what do you mean?
You have an extra drive installed in your system, and you add it to the pool as a spare. That just gives you more protection in the case of multiple drive failures.
I plan on backing up to redundantly to backblaze home vis iscsi method someone recommend if possible to have the drive mounted as a physical drive
I don't know if FreeNAS can mount an iSCSI target, but I know it can mount a remote NFS file system. You can also do rsynch to another system/NAS that supports it. I am just getting started on this myself, so I can't provide too much more insight.
SPECS I am thinking:
HGST 10TB or 8TB Hard Drives 7200 RPM Ultrastar He10 SATA 6Gs/s 256MB 3.5
HBA: LSI SAS 9207-8i
PC Workstation: MNPA19-XTR MELLANOX 10GB ETHERNET NETWORK INTERFACE CARD W/CABLES
HACKINTOSH Workstation: Solarflare SFN5122F Nic card
freeNAS Machine: Chelsio s320 Dual Port NIC card
Intel Xeon E5-1428L v2 6-Cores 2.20 GHz 15M Cache 0 GT/s Processor SR1B9 LGA1356 or something used and cheap but powerful enough.
RAM 32GB ECC DDR3
Supermicro motherboard (not sure yet the model)
The LSI SAS controller is a good choice. I have a 9207-8E for an external enclosure and works very well. That also seems to be the SAS controller of choice here. ECC on the memory is a must IMHO, so good there. Supermicro boards seem to be very popular here, so that seems good as well. Unless you are running jails (using FreeNAS like a kind of Hypervisor), it isn't IMHO terrible processor intensive. Good drive controller, drives, NIC, and enough memory are the most important.
