Will it Openstack

[Donny Davis]

It surely will

https://github.com/iXsystems/cinder

Just awesome...

I have been waiting for a long time for Openstack awesomeness with Freenas, and I couldn't be happier to see this finally make it upstream to the Freenas Community.

Thanks to all those who put in the hard work getting the driver operational.

 

[Chris Moore]

I have been waiting for a long time for Openstack awesomeness with Freenas, and I couldn't be happier to see this finally make it upstream to the Freenas Community.

Thanks to all those who put in the hard work getting the driver operational.

You appear to be familiar with OpenStack, but this will probably create more questions. Would you be willing to do a write-up on how you got it working?
It might help others that would like to give it a try but may not be as familiar with the material.

 

[Donny Davis]

Oh I surely can.
I plan to write an ansible playbook that takes care of it for you. I base my Openstack deployments on upstream RDO, but it should work with others too. All you really have to do is put the driver in the right place, configure the driver, and enable it.

In my setup the controller is running separate from the FreeNAS machine (I can assume this is how its intended to run given the architecture of Openstack).

I used this to get up and running with Openstack. Its simple, quick and easy. Do not put production workloads on a packstack deployment, you will be disappointed. Its built for stuff like this (POC/Dev env)

http://rdoproject.org/install/packstack/

Your Openstack machine will need to be able to reach your FreeNAS machine. If you use cinder as the backend for provisioning VM's the image store will download and clone the disk to the created FreeNAS volume.

This took me about 10 minutes to get up and running.

Grab the driver from github

Code:

git clone --depth=1 https://github.com/iXsystems/cinder
cd cinder/driver
cp -R ixsystems /usr/lib/python2.7/site-packages/cinder/volume/drivers/
restorecon -Rv /usr/lib/python2.7/site-packages/cinder/volume/drivers/

Open up /etc/cinder/cinder.conf
edit this line to look as such (unless you have multiple backends, like the default lvm (*which can safely be disabled*))

Code:

enabled_backends=freenas

and add this at the bottom

Code:

[freenas]
iscsi_helper = tgtadm
volume_dd_blocksize = 512
volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver
ixsystems_login = root
ixsystems_password = "nowayIamtellingyou"
ixsystems_server_hostname = your.freenas.ip.address
ixsystems_volume_backend_name = freenas
ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl
ixsystems_datastore_pool = tank
ixsystems_vendor_name = iXsystems
ixsystems_storage_protocol = iscsi

ixsystems_volume_backend_name Is the parameter for what name you want to appear in Openstack
ixsystems_datastore_pool is the pool you will provision volumes to

Code:

systemctl restart openstack-cinder-*

If you have issues turn on debug mode

Code:

#debug=False
debug=True

And it just works

I will hopefully have all this wrapped up nicely into an ansible playbook shortly so you don't have to even do this much.

Let me know how it works for you... I am testing all of the cinder features now, but it looks like we have a winner!

 

[konetzed]

What version of openstack and version of freenas were you testing with? Any special setup needed on your freenas box?

 

[Donny Davis]

I'm on the latest freenas and openstack ocata.

Still working great

 

[konetzed]

Thanks ill build out an Ocata setup and see if my issues resolve. Using Queens on centos7 ATM and no joy. :D

 

[Donny Davis]

What is the issue are you running into @konetzed

 

[konetzed]

I keep meaning to dig up the exact error but if i remember right freenas api errors out because it cannot check the path to the volume it just created. I remember digging around and finding an error in the api on the freenas box.

 

[Donny Davis]

Hopefully I will have some time to automate this for people soon. I have just been slammed, so no time to write any code.

 

[konetzed]

I don't know if setup needs to be automated it's pretty simple. I have this running in the deployment I run for my employer with two other backends. I have to dig into it, maybe next week I will have time. One thing I noticed on my fresh install of freenas to test this out is that zvols no longer show up in a path and I am ~90% sure on my installs from a while ago that i have upgraded zvols do show up in as a path. From the logs it seemed as the failure occurs when something tries to verify the path exists. If I had captured debug logs this would make way more sense but I wont be back to my setup until wednesday.

 

[Donny Davis]

Turn on cinder debugging, should give you all the data you need. I know that I was not able to use anything other than the root pool. IE no dataset that is nested.

So the volumes go under tank in my case, and the volumes get created directly under that.
tank > freenas-$cindervolid

I wanted to use a nested pool, something like
tank > openstack > cinder > freenas-$cindervolid

But it doesn't seem to like that, and I single purpose boxes... so it wasn't a big deal for me.

 

[konetzed]

Here is what I get and I am thinking this is a bug in the freenas api. I have installed a 100% fresh freenas system from latest bits to test this.


I try to create cinder volume 'mytest'

cinder logs

Code:

		   |__Flow 'volume_create_manager': FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating iscsi target: 409:Conflict
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager Traceback (most recent call last):
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 result = task.execute(**arguments)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 model_update = self._create_raw_volume(volume, **volume_spec)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 ret = self.driver.create_volume(volume)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/iscsi.py", line 96, in create_volume
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 freenas_volume['name'])
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 224, in _create_iscsitarget
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 tgt_id = self._create_target(name)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 141, in _create_target
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 self._create_target_group(target_id)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 123, in _create_target_group
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 raise FreeNASApiError('Unexpected error', msg)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating iscsi target: 409:Conflict

Logs from freenas server

Code:

ct  2 23:06:17 freenas uwsgi: [middleware.notifier:2279] stat /mnt/tank/volume-c12f062a: [Errno 2] No such file or directory: '/mnt/tank/volume-c12f062a'
Oct  2 23:06:17 freenas uwsgi: [middleware.notifier:185] Popen()ing: zfs destroy 'tank/volume-c12f062a'
Oct  2 23:06:17 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "./freenasUI/api/utils.py", line 247, in dispatch
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 request_type, request, *args, **kwargs
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 482, in dispatch
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 response = method(request, **kwargs)
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 1524, in delete_detail
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 self.obj_delete(bundle=bundle, **self.remove_api_resource_names(kwargs))
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "./freenasUI/api/resources.py", line 651, in obj_delete
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 response=self.error_response(bundle.request, retval)
Oct  2 23:06:17 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse

Not sure what exactly is calling the stat yet if that is from the process freenas goes through to create it or on the driver side.

 

[konetzed]

Never mind that last post its all wrong. I figured out the issue and will be making a pr for the driver to fix. Need to dig through things more but the api looks like it has changed and driver is incorrect. I have it all working at the moment just want to be sure the code changes I made are not in some branch.

 

[Beholder101]

I'm preparing/testing a new Openstack Rocky installation using Freenas 11.2. I know it is still BETA, but i guess that version will be (close to) full release by the time we hit production. The driver is giving me the same error:

Reason - Unexpected error:Error while creating iscsi target: 409:Conflict

I hope you can find the time to update the driver. If you need to verify or test, please let me know. My setup is still very much test.

Thank you for your time and effort!

 

[konetzed]

Finally tracked down my issue and was able to recreate it. I was 1000% wrong it is not a driver issue but a config issue in freenas. You need to setup a portal and an initiator for it to use. Look under shareing -> block (iscsi) -> portal or initiators.

you can just add both as defaults but here are mine

LMK if this doesn't fix your problem

 

[Beholder101]

That does seem to get me a bit further. After creating the Portals and Initiators entries, i can see that the driver now creates Targets and Extents entries, but fails at this point:

FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating relation between target and extent: 409:Conflict

In the config there is a relation between the target and extent is made.

The Freenas logs will show me:

Oct 25 09:20:10 ms-stor-01 ctld[2164]: no LUNs defined for target "iqn.2005-10.org.freenas.ctl:target-1462689d"
Oct 25 09:20:10 ms-stor-01 ctld[2164]: portal-group "pg1" not assigned to any target
Oct 25 09:20:11 ms-stor-01 ctld[2164]: no LUNs defined for target "iqn.2005-10.org.freenas.ctl:target-1462689d"

I will try and see of more preconfigured entries are required for this to work.

 

[Beholder101]

OK. I have put in a few hours of work and i'm no closer. If you can help me fill in the blanks, it will be much appreciated.

A little update on top of above post: the extent, target and zvol only get created when i move a volume from NFS to iSCSI, but still end in failure and a missing association/relation. When creating a new iSCSI volume, i can hear the disk trying, but no target, extent or zvol get created on the Freenas.

 

[konetzed]

I cannot test beta ATM but I have told it to DL so maybe later tonight i can test my setup against that. One other thing I thought of is making sure your iscsi service is started and running. You should still get volumes created but they wont attach if the iscsi service isn't running.

Worse case move all the passwords and ips from your config and post in what you have in your "[freenas]" section

 

[Beholder101]

The service is running:

And i have created the Portals and Initiators config:

Finally the only other item that has configuration data is the Target Global configuration:

My Openstack Cinder config for the driver is as follow:

[DEFAULT]
.......
default_volume_type = ixsystems-iscsi
enabled_backends = nfs,lvm,ixsystems-iscsi
.......

[ixsystems-iscsi]
iscsi_helper = tgtadm
volume_dd_blocksize = 512
volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver
ixsystems_login = myuser
ixsystems_password = mypassword
ixsystems_server_hostname = 10.10.10.60
ixsystems_volume_backend_name = ixsystems-iscsi
#ixsystems_volume_backend_name = iXsystems_FREENAS_Storage
ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl
ixsystems_datastore_pool = MS-DATA-HDD
ixsystems_vendor_name = iXsystems
ixsystems_storage_protocol = iscsi
image_volume_cache_enabled = False
volume_backend_name = ixsystems-iscsi

[lvm]
.....

[nfs]
.....

Side notes: I have a config that serves NFS, LVM and iSCSI for testing purposes. In openstack each of the items is directly coupled to a Volume Type. So they are not grouped. The account used to connect to the Freenas server is the root account. I know this is a big NO in production, but i used it to rule out any restrictions on the account side.

Openstack Volume Types:

If i create a volume in Openstack, it creates the volume on Freenas as a zvol. Example: volume-29e32973
Something is doing what it is supposed to, like you also expected it to.

On the Freenas unit, the /etc/ctl.conf reads:
portal-group default {
}

portal-group pg1 {
tag 0x0001
discovery-filter portal-name
discovery-auth-group no-authentication
listen 0.0.0.0:3260
option ha_shared on
}

 

[konetzed]

Beholder101 I have good news and bad news. Good news is you are not crazy. The bad news is the driver does not seem to work with 11.2 beta2.

I install the beta i could not create volumes but just got the 409 conflict as you do. I switched back to a freenas 11 stable install and everything works fine. Both freenas environments were fresh installs into kvm vms.

I think if you switch to using the 11 stable build all your issues will go away and openstack cinder will work as you expected.