jgreco, first of all, thank you very much for taking the time to make such a detailed study and reply of the hardware. I really appreciate that.
Something like an S8236-IL? Um, that just seems to me like the start of a bad set of choices. Three gigabit ethernets for such a large system could turn frustrating, especially with few options for expansion when you're limited to two PCIe slots. You probably don't need the two dozen cores that the board was built for. You're short on everything else: poor networking, limited PCIe, limited memory. I don't know that the SP5100 SAS is supported by FreeBSD, which means you might be stuck needing to get the LSI SAS2008 variant. And of course you have to populate both sockets to get access to the full 256GB. All that fun for $600.
You're not planning to use ZFS, then? Because you really shouldn't put a dinky little hardware RAID controller in front of your massive ZFS RAID system. It defeats the purpose. You should be picking out HBA hardware if possible.
I haven't extensively checked the HBAs against FreeBSD 8 HCL. I'll be doing that soon.
I only picked that SAS controller because it's 3ware (likely to be supported in FreeBSD 8) and with two cards, on the limited two PCIe slots of the mobo, I would have enough SAS ports to attach all of the drives (12x 8087 connectors). I had intended to use the controller in JBOD/HBA mode, not in RAID mode of course. ZFS is an absolute requirement.
For $500, you could get the Supermicro X9SRH-7TF. Requires higher density memory to hit 256GB, but sports 10GbE and three usable PCIe. Onboard LSI SAS2308 for basic RAID functionality, compatible with FreeBSD.
For $800, you can get the super-crazy-fun Supermicro X9DR7-TF+, dual 2011, up to 768GB, six PCIe slots, dual 10GbE, and LSI 2208. There are other variants that substitute quad gigE or other features for a lower price. Beautiful board, we've had one here in shop for several months to play with and it is very nice.
3x GbE for the environment in which this system will be used should be more than adequate. However, I was a little apprehensive about only 2x PCIe. I haven't had time to look for more motherboards but I will look at your suggestions. All of my previous experience with server-class motherboards has been with ASUS, Tyan, and OEM (Dell, HP, etc.). I know of SuperMicro but I have never used their products. They seem to be held in high regard here on the forums, so I'm good with that.
As for CPU, yes this is a big question. 16 cores seems stupid to me for this system. I'm not using dedupe or high-compression on the ZFS. I question whether the way this server will be used will be able to effectively take advantage of more than 4 or 6 cores.
What is your opinion on the number of cores? Given no dedupe, fast-compression, and the relatively slow drives, how many cores do you feel would be appropriate?
RAM. Yes, the mobo is limited to 256 GB of RAM, but the chassis is also limited to 50 SAS bays, too. (I'll get to the chassis next). I was questioning whether I even needed 256 GB ... of course I could easily attached another external JBOD under the same system domain. More RAM could potentially be advantageous in that case, but I think chances are slim that this system will ever be expanded.
Conspicuously missing from this discussion is what you're going to throw everything in. And I'm going to suggest something.
An archival system, by definition, probably does not need instant highest possible speed access to every drive at every second. Consider a chassis like the SC846BE16-R920B. This unit will house your mainboard and will connect all 24 front bays to your HBA of choice via a single SFF8087 cable. And here's the point: those ST4000DM000's are only capable of about 150MB/sec, times 24 is 3600MB/sec, and a SFF-8087 is four 6Gbps lanes for 24Gbps, which is something around 125MB/sec per drive for 24 drives. You buy two of these chassis, one for your mainboard and drives, the other as an expansion chassis. You hook them up to something like an IBM ServeRAID M1015, or maybe pair of M1015's. You end up with a substantially lower expense for unnecessary RAID cards, and you're also ending up on hardware known to work well with FreeNAS.
I forgot to mention the chassis. I'll edit the original post with that info next. I'm planning to use this as the main chassis for this initial set of disks and mobo. I don't even know if the system will need expanded to more disks in the future, but if it does, I can use the external SAS connector to a JBOD with a SAS expander.
Chenbro RM91250: 9RU, 50x 3.5" hot-swap SAS bay, 2x 3.5" hot-swap SATA bay, 12x SAS backplanes (8087), 1620W 3+1 redundant PSU, enough fans to give me a splitting headache is less than a second 
Again, thank you for your commentary and suggestions. I'll be checking SuperMicro mobos tonight, and will be scrubbing the HCL. If I can get more PCIe slots, I'd much rather use simpler HBAs with a proven track record.
What about using 2x HBA with only 2x 8087 connections to a couple SAS expanders I install in the chassis? Given these are slow drives, might this be a good solution since I don't need massively parallel I/O?
