GELI re-key before resilver finished?

[RobL]

Hi everyone,

Setup looks like this:

ESXi 6.5U1 fully patched
Dell R730XD
PERC H730 mini
16 x 8TB SAS drives
2 x 300GB 15K SAS drives
2 x 8c 2.4GHz CPU
128GB RAM
2 x X710 Intel 10GE NICs
1 x X540 Intyel NIC
Dell 2x10GE + 2x1GE mixed NIC
FreeNAS-9.10.2-U6 (561f0d7a1)

So, I've just had my second drive fail, and have been experimenting with simple ways to replace the drive. In short, there aren't any :-( even given the horrendous price of this server from Dell. There is no hot swap option for a passthrough SAS hard drive using the H730 mini in ESXi 6.5U1. You need to shutdown any VMs using the drive and rebuild the RDM mappings and reassign to the VM. A real PITA.

More interestingly, I am currently re-silvering my principle zpool and it's going to take about 3 hours.

Here's my question ... as I have to re-key the GELI config across the volume, can I start this prior to the re-silver completing, to avoid forgetting if I go to sleep? If not, why not?

That's it in a nutshell. BTW - Dell didn't care that I can't hot replace an H730 drive in ESXi 6.5 without shutting down the VM that consumes the drive. "I only have basic support" said the droid. I pointed out the firmware in the H730 mini and on the SAS drives was covered, and if that was breaking the AHCI behaviour it is very much Dell's problem to solve. We'll see.

Many thanks,

--RobL.

 

[Chris Moore]

That's it in a nutshell. BTW - Dell didn't care that I can't hot replace an H730 drive in ESXi 6.5 without shutting down the VM that consumes the drive. "I only have basic support" said the droid. I pointed out the firmware in the H730 mini and on the SAS drives was covered, and if that was breaking the AHCI behaviour it is very much Dell's problem to solve. We'll see.

You are using hardware that is not correct for FreeNAS and using it in a way that is specifically not the way to do it, so I don't think it is Dell's fault that you are using the hardware wrong or that you are using the wrong hardware for what you are doing...

It is not that you can't virtualize FreeNAS successfully, but you need to use a straight SAS HBA and pass the whole HBA through to the virtual machine so the FreeNAS has direct access to the disk. The way you are doing it is probably causing the failed disks and the disks may not actually be bad, it may just be FreeNAS rejecting them because of some latency in the RAID controller.

 

[RobL]

You are using hardware that is not correct for FreeNAS and using it in a way that is specifically not the way to do it, so I don't think it is Dell's fault that you are using the hardware wrong or that you are using the wrong hardware for what you are doing...

It is not that you can't virtualize FreeNAS successfully, but you need to use a straight SAS HBA and pass the whole HBA through to the virtual machine so the FreeNAS has direct access to the disk. The way you are doing it is probably causing the failed disks and the disks may not actually be bad, it may just be FreeNAS rejecting them because of some latency in the RAID controller.

Chris,
Everyone is entitled to opinions. It's what makes democracy great. Can I suggest we separate the two issues here, as I'm only really interested in understanding whether or not FreeNAS 9.10.2-U6 can handle re-keying of GELI encrypted drives while a re-silver is in progress.

As for which hardware FreeNAS does and does not support, and which HBA passthrough configurations Dell EMC does and does not support to the ESXi 6.5 hypervisor... those are interesting conversations, driven by each individuals risk appetite. I've gone for months without issue with FreeNAS 9.10 on ESXi 6.5 with H730 mini passthrough of all ZFS pool devices. Backups, and replication ... yes - I have both of those.

My question is now academic, but still useful to understand, as the job is done:

Code:

[user@zfs1-FreeNAS ~]$ zpool status vol1
  pool: vol1
 state: ONLINE
  scan: resilvered 549G in 2h19m with 0 errors on Tue Oct 10 04:09:35 2017
config:

		NAME												STATE	 READ WRITE CKSUM
		vol1												ONLINE	   0	 0	 0
		  raidz2-0										  ONLINE	   0	 0	 0
<deleted output>

Thanks,
--RobL.

 

[Chris Moore]

Not an opinion.

 

[SweetAndLow]

Not an opinion.

I totally agree with you but this guy is not going to listen and doesn't care.

 

[RobL]

Gentlemen, I appreciate your observations on my server specification however I'm only looking for an answer to the re-key question, and I do care. I've found this forum very helpful in the past and I really like the ZFS implementation in FreeNAS. The H730 bit was fodder for humerous chortles, not to start a flame war.

@SweetAndLow - I am reading the forum (!) and I'm not sure why you have taken such an aggressive stance towards my question. From your "Recommended reading" list in your footer, I can assure you that the Dell R730XD upon which I have installed ESXi and FreeNAS 9.10 meets and exceeds all the recommendations within it (Hardware 2016 R1e.pdf) except for the H730 mini. The H730 mini is problematic, but isn't addressed directly. My drive failure was real, hardware, and not related to the H730 mini or its firmware. I would consider a Dell H310 HBA (reducing SAS bus speed by 50%) crossflashed with LSI firmware to tick 100% of the recommendation boxes before asking a question related to data integrity. But I'm not asking such a question.

Thanks in advance. I hope your day is going well with happy thoughts.