BTW, I just want to make sure it is said, I really appreciate the info, this is a very informative post.
For direct file storage the "best practice" would be to store it directly on your array. The challenge with putting them on a Windows system consuming an iSCSI ZVOL instead is adding the layer of abstraction that is NTFS - ZFS no longer has the visibility into the files themselves, they're just a chunk of blocks. But as mentioned, if you're doing fancy things with Windows ACLs (or just really prefer to feed Windows to Windows) then it will work - just possibly with a little more space used and a little less performance overall.
You know, I think this here is getting to the brass tacks of my original question and it is a key consideration.
If I do it this way, am I basically negating much of the benefits of ZFS by not giving it access to the file data?
Not sure if NFS would get around this problem, some trade-offs but maybe.
This more than anything else would sway me to just living with Samba and moving on.
I feel lucky to find someone with the depth of knowledge to explore this point.
Glad to hear you saved a bunch from the shredder.
Well, you can see I was pretty selective. There were a whole lot of drives (100's) <4TB and most of them SATA.
We have one of those hard disk crusher things, they are fun for the first 50-60 drives, but then it quickly becomes work especially because you have to take them out of the drive carriers first. There are still two decent sized dumpsters full of drives that me and other people have been chipping away at over months. If I had the energy I would harvest the neodymium magnets, my kids would love it but it is a bear to get them out.
No worries. A group of disks joins together to make a vdev - this is where your redundancy level comes in, eg: mirrors, RAIDZ1, Z2, etc. A group of vdevs join together (in a "JBOD" style arrangement) to make a pool. Different types of vdevs tolerate a different number of disks lost, but losing an entire vdev means the whole pool is offline.
ZVOLs can be thought of as "partitions" - block devices that consume space from the pool.
I will get it down eventually, what happens is that we have many different products at work that all use different terminology for the same thing, although ZFS has a bunch of different stuff going on.
No support for vVols I don't even think on commercial TrueNAS. You can go for in-guest iSCSI mounting, you'll just want to use initiator groups either by IP or IQN to make sure that your Windows guest doesn't try to mount your VMware LUNs, and vice versa.
Yeah, they probably never will, because I think VMware charges a fee.
If FreeNAS can do it, I could separate the traffic on different port groups or even separate vSwitches, and never the twain shall meet.
No need for an expensive flash-backed one if you're just going to boot vSphere and load the FreeNAS VMX; if it's supported under 7.x I'd go with your last option (switch onboard back to IR and boot from that) especially if you intend to use FreeNAS to provide the VMFS datastores via loopback iSCSI/NFS. You do have that second (or third?) 32GB system as well; that would likely be fine if you were going to use it for the iSCSI file-store, but wouldn't do justice to the rest of that hardware.
Well, I get into this analysis paralysis: find something for a good price, that is not counterfeit, and I tend to go with things that will last me a little while, so maybe not have flash-backed, but with the capability to add it. And it needs to be new enough to support ESXi 7.x. That last one seems to be the kicker, otherwise I could use previous-gen 6Gig parts. Currently looking at this one here:
LSI Logic MegaRAID SAS 9341-8i Looks like you don't even need the cache if you are all-flash anyways, and as you say just RAID 1 or 10.
Yeah, the 3rd system is old but decent, Xeon E3-1280 on a good old X9SCM-F. Definitely a good fit if I don't bother with iSCSI and just go bare metal. I may be doing that anyways just for backup otherwise just make it a pFsense box.
I ended up with the first two that are Supermicro X10srh-cf because I got an amazing price, and I had the right CPUs and RAM needed in the junk pile. I am actually thinking of keeping the 2nd one in case the first one dies or something, I have a spare.
On the other hand, if I *did* go bare metal with one of these, they have 10x SATA3 ports in addition to the SAS, which would work fine with the 10TB SATA disks.
And speaking of hardware, let's have a look at all that awesomeness you salvaged.
I have to say it is helpful to get this assessment of this hardware from someone who is familiar with these drives.
Damned fine devices, although the U.2 form factor is harder to pack densely into a system. I believe IcyDock makes a 4-into-5.25" bay adapter but you'd need the NVMe HBA anyways; not cheap.
This is how I am doing it:
Those SM 2x M.2's are cheap, and the cables aren't too bad either.
The card behind it is a regular single NVMe with a Samsung 512Gig drive on it.
I
can plug them into the LSI 9400 but the cables are actually expensive and it burns allot of ports.
Starting to get crowded on PCIe slots here though, because you have to turn on bifurcation to use this card, and that setting changes it for each pair of slots. I may end up just using them in my desktop or something, I dunno.
Vanilla SM1625s are eMLC - these might be SLC though due to the OEM source. Good for L2ARC, possibly too dated for SLOG, maybe too small for regular data use, especially given that:
Yeah, too small but the high endurance should make for great cache devices. How does the fact that these are older affect the SLOG?
Most people would be happy to have a couple for SLOG devices - you can build entire pools from these stupidly well overprovisioned (576G of raw NAND inside these I believe) SSDs and have them soak up writes for petabytes.
Yeah, these are arguably the real gem of the salvage operation.
Especially since they are just big enough to start adding up, and the amount of VM storage I need is small, even 1TB would probably be plenty. They have been in use for awhile, not sure if there is an easy way to tell how much endurance they have left. The SAN spread the load across so many they are probably fine.
Constellation ES.3's, quality SAS spindles, I've got some of the 0043 self-encrypting ones myself. Easy to slap together into RAIDZ2 and store capacity.
That is good to hear, I wasn't sure if these were junk or not.
Probably dreams of having an all-flash array for your bulk file storage, "because science isn't about 'why' it's about 'why not?'"
Hehe, I probably shouldn't tell you this, but the expander in the JBOD box is a nice Intel with 36x 12Gig ports, and the LSI 9400 is a 16i card. That many SSDs would likely saturate the uplink even if you did wide, but that is still 8x12Gig ports. Really the only thing stopping me from even trying it would be buying that many SAS cables. But it would be interesting. Although with that much cables my lab would look like Darth Vader's bathroom....
But if you're having second thoughts, I'll be more than happy to take any of those extra, unnecessary SSDs off your hands. ;)
Honestly I would give some of them away if I was allowed to, but they have serial numbers and stuff. But even if I thought I could get away with it a promise is a promise to my boss.
Ultimately let's look at this in terms of solving your needs.
The challenge here is that your data currently resides in Windows format on those 7x 10T drives; and those would be the best candidates for a 7-drive RAIDZ2 yielding about 50T of usable space. Failing that, if you put 10x4T into the JBOD as a single RAIDZ2 vdev, that would give you ~32T usable out of the gate. You could then use that to transfer all of your data off the 7x10T drives, then format them and load them into 7 out of the 16 main bays in your case, copy/zfs send the data over to them, and clear out your JBOD. It can then be a cold/backup copy via replication. Set up a pair of the HGST SSDs for mirrored SLOG and you can turn on sync writes to protect in-flight writes without wreaking havoc on your latency; you can even keep three more ready for when TrueNAS 12 drops and assign them as metadata-only devices in a three-way mirror for extra safety.
Yeah, it's not the end of the world if I effectively have to build a temporary setup so that I can shift the data off of those 10TB drives, because I do like your idea here, it might be the way to go. Just a bit time consuming.
You really like using those HGST drives for the SLOG, will it even use that much space?
If you're going to serve VMs, the path forward with that much SSD is obvious - a big pool made up of mirrored HGST HUSMM drives. It should be fast even with sync writes. But that depends on how much space you need for the performance pool, or if you need that performance at all. Mirrors all the way for the VMs though.
I think right now I am leaning toward dropping the iSCSI idea completely. Just get a good RAID card for VMFS to host all the VMs.
Then do the swap to get the 10TB drives for the data pool going with your suggestion. Keep it simple. By the way, there are 7x of those drives because that was the number needed for dual-parity under storage spaces. I could add one if 8x works better for FreeNAS. But otherwise 50TB is more than enough space to last me quite awhile.
Technically I could just map a persistent network drive from Windows to the FreeNAS, using SMB3 at 10Gig the performance will probably be OK. The latest versions of Samba in FreeNAS are approaching feature parity with Windows, and it is much more mature than that QNAP I had. I might even be able to use junction points and mount to folders.
I could always play with the iSCSI idea in a lab scenario not using real data. This is one of the reasons to have multiple servers around anyways.
Then I just use the other drives to create another server to be the backup. I am leaning toward just using something like Veeam because it is just a backup. Replication is fine but generally that is more for HA than backup, I know ZFS can do the snapshots and stuff but I am not that experienced with it yet.
Bah, you said "Seagate 400G SAS" in your initial post, "HGST 400G SAS" would have been a dead giveaway for the HUSMM-series.
Yeah, my bad, I was confusing them with the 4TB drives.....
Great conversation, I think I am homing in on a good solution here.
-JCL
www.servethehome.com
