iSCSI adding device context

[Squirreljester]

I have a disk volume set up already, and FreeNAS can see it, but when I go to set up a device context as part of the iSCSI initiator setup, nothing is in the drop down box. Did I miss a step somewhere?

I'm testing this out on an old Dell PowerEdge 2450, 1.5GB ram, 4 73GB hard drives (1 has the FreeNAS install, the other 3 are the UFS volume I created within FreeNAS after installation.

Pretty basic, I just want to see how stable this is before testing it out on a larger scale.

Also, if someone is feeling adventurous... it seemed to not boot to the USB drive I installed it to when I created a hardware RAID volume first, and then rebooted, but leaving the disks not in a RAID let FreeNAS boot to the USB drive. Kind of odd behavior, and it doesn't make sense, because the 2 shouldn't be connected.

Thanks for any help anyone can give me. I've not very Linux savvy, but I'm learning more and more every day.

 

[jgreco]

A device extent? If you have a disk volume set up, you can make a file on that and export it as an extent (NOT "device extent"). If you have a device that you want exported as an extent, it cannot be a member of an existing volume.

For example, on our storage server,

storagex 2726314172 1345 2726312827 0% /mnt/storagex
storagex/iscsi 2744102422 17789595 2726312827 1% /mnt/storagex/iscsi

# ls -l /mnt/storagex/iscsi/
total 17790591
drwxr-xr-x 2 root wheel 4 Sep 11 12:32 ./
drwxr-xr-x 10 root wheel 15 Sep 11 07:31 ../
-rw-r--r-- 1 root wheel 1099511627776 Sep 16 15:51 extent0

Appears to our ESXi cluster as a 1TB iSCSI device, but in reality is just a rather smaller sparse file on a ZFS RAIDZ2 filesystem. This is a really sweet way to go unless you have a specific reason you need a device extent.

 

[Squirreljester]

So....

I just want to test out presenting a vmware server the space I've created on the FreeNAS box without doing anything special. And I have no idea what all that is that you posted, remember that I have very light Linux knowledge, especially the commands. I grew up working on DOS, so command line isn't something completely foreign to me, but I really want something that I can configure and admin through the GUI.

Do I even need an extent set up at all, if all I want to do is share the entire volume I've created on the FreeNAS box via iSCSI to the vmware server?

 

[jgreco]

You cannot "share the entire volume" "via iSCSI".

Follow along and we'll get you sorted out.

You appear to have a loose idea of what these things are; that's fine, nobody does a good job explaining it, it seems.

iSCSI is a SAN storage protocol. "Storage Area Network". SAN storage is block-oriented, and can be based on many different technologies to connect drives. But fundamentally you have a host adapter on your server that asks storage device N for disk block X. VMware's vmfs is an example of a filesystem that is designed to run on top of a shared SAN network. The SAN network has no filesystem of its own; it just stores blocks of data.

FreeNAS is a "Network Attached Storage" server. NAS storage is file-oriented, and is typically offered via protocols such as NFS, CIFS/SMB, AFP, or FTP. The remote clients have no clue what filesystem the NAS unit uses, and they treat the NAS server as a black box for storing and retrieving files. FreeNAS usually uses ZFS. You're using UFS. That's fine and doesn't matter much.

So when you do "ls /foo/bar" on a NAS, your client machine says "give me the directory listing of /foo/bar" to the NAS box, which then feeds that to its local filesystem code and gets you the answer. But when you do "ls /foo/bar" on a SAN, your client machine feeds that to its local filesystem code, which then turns around and issues an appropriate set of disk block I/O requests to the SAN disk. Note that the filesystem on a NAS is embedded in the NAS; the filesystem on a SAN is the client's problem. This is KEY to understanding the difference.

Now FreeNAS is very capable; it *also* offers some basic iSCSI SAN capabilities. But you can go awry very quickly if you don't keep this all straight.

So what you're trying to do is to export a volume (the NAS's filesystem) through iSCSI. Doesn't work. The filesystem holds files. iSCSI speaks blocks. You won't find a way to do this sanely.

You have some choices, now.

1) You can use FreeNAS as a NAS. With vmware, you will be using NFS-based NAS. We can get you sorted out if you want to go that route. Otherwise just turn on NFS, export your filesystem via NFS, and configure vmware to speak NFS to your NAS. This is *fairly* easy.

2) You can use FreeNAS as a SAN with device extents. This means your vmware box will access a physical disk on the FreeNAS box using iSCSI and a device extent. This is very similar to attaching it directly to the vmware box, except that it can be shared by more than one vmware box if you're putting a vmfs filesystem on it. On the other hand, there is no easy way to do disk redundancy (RAID1, RAID5, etc). This is *hard* to set up, mostly because you have to cross all your i's and dot all your t's (and that's usually how it all goes wrong).

3) You can use FreeNAS as a SAN with iSCSI file extents. Doing this, you create a *file* on the FreeNAS filesystem that represents a disk for your vmware server, virtually. Think of it as a hard disk image file. FreeNAS exports that as a virtual iSCSI disk to your vmware boxes. The advantage to this is that it can take natural advantage of things like RAIDZ2 protection offered by ZFS, but you have to be carefully aware that you are not exporting the ZFS filesystem, you are just exporting a file ON the ZFS filesystem that's an image of a hard disk. This is also a bit of a pain to set up; NFS is easier. But some of us like iSCSI.

Now I suggest you might want to read this over a few times, because it's confusing until it isn't. You're probably going to end up going with NFS or iSCSI file extents. Those are the usual (and safest) choices.

 

[Squirreljester]

Most of that I knew already, but thanks for the additional information. I have worked with Linux admins setting this up in the past, but I've never done it myself. I should have thought of the file extents as virtual hard drives, I'm not sure why I didn't, but I think I was fixated on setting up a device extent to make the entire volume available to the client connecting to it.

Answers to your questions:
1) I like easy, and I like things that I can set up and manage via a GUI pretty robustly. I'm testing this as a home grown shared storage solution for some VMWare servers as an alternative to purchasing $20,000 worth of dedicated SAN equipment. This is one option I could use, but I would like to use iSCSI. In my experience iSCSI is quite a bit faster than NFS, and speed is something I need with the setup.

2) This option is the one I was thinking. I would like to have this setup be capable of VMotion, shared storage of vm stores between multiple VMWare hosts, etc. Is this option more stable than the other options like going NFS or using file extents?

3) I wouldn't be opposed to doing this, but I would like to maximize the space I'm using on the FreeNAS server, and I'll only have 1 shared place where the VMWare hosts store their VM's.
Actually, can I have different shares set up for the different disk *files*? Like, have one formatted with VMFS for the VMWare shared storage, and one set up as a SAMBA share for the windows machines in the office? If so, this may be an option that I could be open to using.

Here's some questions:

4) If I create a file extent, can I back it up, and if anything ever happens to the FreeNAS server, I can build another one, and restore that extent to the new installation, and be ok? Or is it more complicated than that?

5) Deduplication. Which is "the best way to do it" to optimize deduplication?

6) Replication. I want to replicate whatever shares and/or virtual disks I set up to a remote site, so I'd like to set this up with that in mind for something to set up in the future, for some level of DR.

 

[jgreco]

1) I don't know if iSCSI is quite a bit faster than NFS. I don't know that it's not, either. There are tunables on both sides for vmware and FreeNAS, though.

2) No. Basically it means that if your disk goes toast, so does your data. If you're using NFS or file extents, you at least have the option to store them on RAIDZ-, RAIDZ2-, etc., filesystems. That strikes me as safer.

3) With a virtual disk, FreeNAS sees it as just a file - except that iSCSI offers the file up as an iSCSI device to anything smart enough to talk to it. You do not need separate filesystems or separate disks. You can have one big filesystem "/mnt/mystuff" and then make an iSCSI file extent on it "/mnt/mystuff/vmware0" of 1TB of sparse space, that will fill in as things are stored on the disk. You can still share out the remainder of that filesystem with other protocols too, but those are shared as files. If you were to share "/mnt/mystuff" via Samba, for example, you would see "vmware0" as a file sitting within that share.

4) Yeah, you can do that. It may or may not make sense to do it. For example, if you have a very busy vmware VM that's writing all sorts of stuff to the disk, you can grab a snapshot of the file extent but the VM's disk may not be left in a completely rational state. You could certainly shut down all your VM's and then grab a snap. Basically FreeNAS knows nothing about vmfs; it's "just data".

5) Current FreeNAS doesn't do dedup. One of the releases in the next year probably will, in my opinion. Newer version of ZFS required.

6) Yes, FreeNAS and ZFS can do snapshot based replication to a remote site.

Point #4 is probably the most significant; NFS offers FreeNAS some better visibility into what is going on. You may have better options for more fine-grained operations if you are able to go into a VM's directory and copy just that one VM or something like that.

 

[Squirreljester]

2) I think file extents are the way I want to go then. I've been thinking more about it, and I like the control of setting up a big disk space, and then using file extents as different "share locations", one for the VM datastores, one for the SAMBA/NFS share(s). If I use RAIDZ or RAIDZ2, I can't set up a hardware raid before hand right? And if I use either of those, I can't use ZFS as my filesystem can I?

5) I thought FreeNAS 8 had dedupe because it has the newest version of ZFS which has it.

I went through a walk-through config from youtube, and combining that with our discussion, I think I understand a little better how things are connected, but I'm having a strange issue now. I'm currently just trying to connect my Windows 7 laptop to ANYTHING on the FreeNAS box.

I first had it half set up, I basically created everything, but I didn't attach the targets to the extents. I then went back, changed the iqn default name to iqn.2011-03.local.freenas, set up 2 file extents, added the targets, associated the targets, and set authorized initiator to ALL/ALL.
On the XP side, I clicked on configuration and put in my iqn.2011-03.local.freenas in the initiator name, but left the CHAP secret blank. Then I went to discovery, clicked on discovery portal, put in the IP address of the FreeNAS box, and I got no errors. Here's where the problem is, I go to the targets tab, and it lists my 2 targets, but they have the old default iqn name as their name, not iqn.2011-03.local.freenas, but the targets are both named correctly. They are also Inactive, and I get Authentication Failure when I try to connect them, but when I click connect, I can't change the target name from the example.org one to the right one... and I can't find where it's picking that iqn up anywhere on the FreeNas box.

7) Will it create the files when I create the file extents, or do the files have to already be there? Basically what I'm asking is, when you click extents, add extent, give it a name, browse to my volume which is /mnt/DA123UFS and just add a /target1 at the end to make a file extent named target1 on /mnt/DA123UFS? And when I delete the extent in the GUI, does it delete the actual extent file too?

Thanks again for all your help, I really really appreciate it.

 

[jgreco]

2) You don't use a file extent for SAMBA/NFS, those protocols just share files that are on the NAS.

2a) You can set up hardware RAID if you like. If it breaks, you are then left mostly to your own devices to recover your data if there's a problem.

2b) You can put ZFS on a hardware RAID. It doesn't care. It's just a fancy filesystem. But most people let ZFS do the RAID because ZFS does it very well on many levels.

5) FreeNAS 8 has ZFS v15 because FreeBSD 8.2-RELEASE has ZFS v15. A more recent ZFS is coming to FreeBSD 8.{3? 4?}.

I'm not sure on the discovery thing.

7) I don't recall. I "truncate"'d the files manually. Mostly because I'm an old-timer and I don't trust all this newfangled automation. But I would be pretty shocked if it deleted an extent file when you delete it in the GUI, so I'm going to say it doesn't do that, because that would be a disaster.