Will it Openstack

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
It surely will

https://github.com/iXsystems/cinder

Just awesome...

I have been waiting for a long time for Openstack awesomeness with Freenas, and I couldn't be happier to see this finally make it upstream to the Freenas Community.

Thanks to all those who put in the hard work getting the driver operational.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I have been waiting for a long time for Openstack awesomeness with Freenas, and I couldn't be happier to see this finally make it upstream to the Freenas Community.

Thanks to all those who put in the hard work getting the driver operational.
You appear to be familiar with OpenStack, but this will probably create more questions. Would you be willing to do a write-up on how you got it working?
It might help others that would like to give it a try but may not be as familiar with the material.
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
Oh I surely can.
I plan to write an ansible playbook that takes care of it for you. I base my Openstack deployments on upstream RDO, but it should work with others too. All you really have to do is put the driver in the right place, configure the driver, and enable it.

In my setup the controller is running separate from the FreeNAS machine (I can assume this is how its intended to run given the architecture of Openstack).

I used this to get up and running with Openstack. Its simple, quick and easy. Do not put production workloads on a packstack deployment, you will be disappointed. Its built for stuff like this (POC/Dev env)

http://rdoproject.org/install/packstack/

Your Openstack machine will need to be able to reach your FreeNAS machine. If you use cinder as the backend for provisioning VM's the image store will download and clone the disk to the created FreeNAS volume.

This took me about 10 minutes to get up and running.

Grab the driver from github
Code:
git clone --depth=1 https://github.com/iXsystems/cinder
cd cinder/driver
cp -R ixsystems /usr/lib/python2.7/site-packages/cinder/volume/drivers/
restorecon -Rv /usr/lib/python2.7/site-packages/cinder/volume/drivers/

Open up /etc/cinder/cinder.conf
edit this line to look as such (unless you have multiple backends, like the default lvm (*which can safely be disabled*))
Code:
enabled_backends=freenas

and add this at the bottom
Code:
[freenas]
iscsi_helper = tgtadm
volume_dd_blocksize = 512
volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver
ixsystems_login = root
ixsystems_password = "nowayIamtellingyou"
ixsystems_server_hostname = your.freenas.ip.address
ixsystems_volume_backend_name = freenas
ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl
ixsystems_datastore_pool = tank
ixsystems_vendor_name = iXsystems
ixsystems_storage_protocol = iscsi


ixsystems_volume_backend_name Is the parameter for what name you want to appear in Openstack
ixsystems_datastore_pool is the pool you will provision volumes to
Code:
systemctl restart openstack-cinder-*


If you have issues turn on debug mode
Code:
#debug=False
debug=True


And it just works

I will hopefully have all this wrapped up nicely into an ansible playbook shortly so you don't have to even do this much.

Let me know how it works for you... I am testing all of the cinder features now, but it looks like we have a winner!
 
Last edited:

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
What version of openstack and version of freenas were you testing with? Any special setup needed on your freenas box?
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
I'm on the latest freenas and openstack ocata.

Still working great
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
Thanks ill build out an Ocata setup and see if my issues resolve. Using Queens on centos7 ATM and no joy. :D
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
What is the issue are you running into @konetzed
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
I keep meaning to dig up the exact error but if i remember right freenas api errors out because it cannot check the path to the volume it just created. I remember digging around and finding an error in the api on the freenas box.
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
Hopefully I will have some time to automate this for people soon. I have just been slammed, so no time to write any code.
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
I don't know if setup needs to be automated it's pretty simple. I have this running in the deployment I run for my employer with two other backends. I have to dig into it, maybe next week I will have time. One thing I noticed on my fresh install of freenas to test this out is that zvols no longer show up in a path and I am ~90% sure on my installs from a while ago that i have upgraded zvols do show up in as a path. From the logs it seemed as the failure occurs when something tries to verify the path exists. If I had captured debug logs this would make way more sense but I wont be back to my setup until wednesday.
 

Donny Davis

Contributor
Joined
Jul 31, 2015
Messages
139
Turn on cinder debugging, should give you all the data you need. I know that I was not able to use anything other than the root pool. IE no dataset that is nested.

So the volumes go under tank in my case, and the volumes get created directly under that.
tank > freenas-$cindervolid

I wanted to use a nested pool, something like
tank > openstack > cinder > freenas-$cindervolid

But it doesn't seem to like that, and I single purpose boxes... so it wasn't a big deal for me.
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
Here is what I get and I am thinking this is a bug in the freenas api. I have installed a 100% fresh freenas system from latest bits to test this.


I try to create cinder volume 'mytest'

cinder logs
Code:

		   |__Flow 'volume_create_manager': FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating iscsi target: 409:Conflict
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager Traceback (most recent call last):
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 result = task.execute(**arguments)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 1011, in execute
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 model_update = self._create_raw_volume(volume, **volume_spec)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_raw_volume
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 ret = self.driver.create_volume(volume)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/iscsi.py", line 96, in create_volume
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 freenas_volume['name'])
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 224, in _create_iscsitarget
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 tgt_id = self._create_target(name)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 141, in _create_target
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 self._create_target_group(target_id)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/ixsystems/common.py", line 123, in _create_target_group
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager	 raise FreeNASApiError('Unexpected error', msg)
2018-10-02 23:06:14.195 2153 ERROR cinder.volume.manager FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating iscsi target: 409:Conflict


Logs from freenas server
Code:
ct  2 23:06:17 freenas uwsgi: [middleware.notifier:2279] stat /mnt/tank/volume-c12f062a: [Errno 2] No such file or directory: '/mnt/tank/volume-c12f062a'
Oct  2 23:06:17 freenas uwsgi: [middleware.notifier:185] Popen()ing: zfs destroy 'tank/volume-c12f062a'
Oct  2 23:06:17 freenas uwsgi: [api.utils:12] Traceback (most recent call last):
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "./freenasUI/api/utils.py", line 247, in dispatch
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 request_type, request, *args, **kwargs
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 482, in dispatch
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 response = method(request, **kwargs)
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "/usr/local/lib/python3.6/site-packages/tastypie/resources.py", line 1524, in delete_detail
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 self.obj_delete(bundle=bundle, **self.remove_api_resource_names(kwargs))
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]   File "./freenasUI/api/resources.py", line 651, in obj_delete
Oct  2 23:06:17 freenas uwsgi: [api.utils:12]	 response=self.error_response(bundle.request, retval)
Oct  2 23:06:17 freenas uwsgi: [api.utils:12] tastypie.exceptions.ImmediateHttpResponse


Not sure what exactly is calling the stat yet if that is from the process freenas goes through to create it or on the driver side.
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
Never mind that last post its all wrong. I figured out the issue and will be making a pr for the driver to fix. Need to dig through things more but the api looks like it has changed and driver is incorrect. I have it all working at the moment just want to be sure the code changes I made are not in some branch.
 

Beholder101

Dabbler
Joined
Feb 21, 2016
Messages
14
I'm preparing/testing a new Openstack Rocky installation using Freenas 11.2. I know it is still BETA, but i guess that version will be (close to) full release by the time we hit production. The driver is giving me the same error:

Reason - Unexpected error:Error while creating iscsi target: 409:Conflict

I hope you can find the time to update the driver. If you need to verify or test, please let me know. My setup is still very much test.

Thank you for your time and effort!
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
Finally tracked down my issue and was able to recreate it. I was 1000% wrong it is not a driver issue but a config issue in freenas. You need to setup a portal and an initiator for it to use. Look under shareing -> block (iscsi) -> portal or initiators.

you can just add both as defaults but here are mine
upload_2018-10-24_15-19-37.png



upload_2018-10-24_15-19-57.png


LMK if this doesn't fix your problem
 

Beholder101

Dabbler
Joined
Feb 21, 2016
Messages
14
That does seem to get me a bit further. After creating the Portals and Initiators entries, i can see that the driver now creates Targets and Extents entries, but fails at this point:

FreeNASApiError: FREENAS api failed. Reason - Unexpected error:Error while creating relation between target and extent: 409:Conflict

In the config there is a relation between the target and extent is made.

The Freenas logs will show me:

Oct 25 09:20:10 ms-stor-01 ctld[2164]: no LUNs defined for target "iqn.2005-10.org.freenas.ctl:target-1462689d"
Oct 25 09:20:10 ms-stor-01 ctld[2164]: portal-group "pg1" not assigned to any target
Oct 25 09:20:11 ms-stor-01 ctld[2164]: no LUNs defined for target "iqn.2005-10.org.freenas.ctl:target-1462689d"

I will try and see of more preconfigured entries are required for this to work.
 

Beholder101

Dabbler
Joined
Feb 21, 2016
Messages
14
OK. I have put in a few hours of work and i'm no closer. If you can help me fill in the blanks, it will be much appreciated.

A little update on top of above post: the extent, target and zvol only get created when i move a volume from NFS to iSCSI, but still end in failure and a missing association/relation. When creating a new iSCSI volume, i can hear the disk trying, but no target, extent or zvol get created on the Freenas.
 

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
I cannot test beta ATM but I have told it to DL so maybe later tonight i can test my setup against that. One other thing I thought of is making sure your iscsi service is started and running. You should still get volumes created but they wont attach if the iscsi service isn't running.

upload_2018-10-25_16-44-26.png


Worse case move all the passwords and ips from your config and post in what you have in your "[freenas]" section
 

Beholder101

Dabbler
Joined
Feb 21, 2016
Messages
14
The service is running:
25859-865b445b741d9dfea47834f2a7e71259.jpg


And i have created the Portals and Initiators config:
25858-aed352cd5759287c7fd47556a1c38ddc.jpg
25857-257c8344ba9a28d271fa54f03e01ad59.jpg


Finally the only other item that has configuration data is the Target Global configuration:
25856-e18fa661993748c42e81739baab7aa96.jpg


My Openstack Cinder config for the driver is as follow:

[DEFAULT]
.......
default_volume_type = ixsystems-iscsi
enabled_backends = nfs,lvm,ixsystems-iscsi
.......

[ixsystems-iscsi]
iscsi_helper = tgtadm
volume_dd_blocksize = 512
volume_driver = cinder.volume.drivers.ixsystems.iscsi.FreeNASISCSIDriver
ixsystems_login = myuser
ixsystems_password = mypassword
ixsystems_server_hostname = 10.10.10.60
ixsystems_volume_backend_name = ixsystems-iscsi
#ixsystems_volume_backend_name = iXsystems_FREENAS_Storage
ixsystems_iqn_prefix = iqn.2005-10.org.freenas.ctl
ixsystems_datastore_pool = MS-DATA-HDD
ixsystems_vendor_name = iXsystems
ixsystems_storage_protocol = iscsi
image_volume_cache_enabled = False
volume_backend_name = ixsystems-iscsi

[lvm]
.....

[nfs]
.....


Side notes: I have a config that serves NFS, LVM and iSCSI for testing purposes. In openstack each of the items is directly coupled to a Volume Type. So they are not grouped. The account used to connect to the Freenas server is the root account. I know this is a big NO in production, but i used it to rule out any restrictions on the account side.

Openstack Volume Types:
25860-25b9a0d41d8661ca5e5b30e2e11524b4.jpg


If i create a volume in Openstack, it creates the volume on Freenas as a zvol. Example: volume-29e32973
Something is doing what it is supposed to, like you also expected it to.

On the Freenas unit, the /etc/ctl.conf reads:
portal-group default {
}

portal-group pg1 {
tag 0x0001
discovery-filter portal-name
discovery-auth-group no-authentication
listen 0.0.0.0:3260
option ha_shared on
}
 

Attachments

  • global.JPG
    global.JPG
    28.9 KB · Views: 504
  • initiators.JPG
    initiators.JPG
    35.9 KB · Views: 477
  • portals.JPG
    portals.JPG
    41.4 KB · Views: 539
  • service.JPG
    service.JPG
    36.1 KB · Views: 488
  • volumetypes.JPG
    volumetypes.JPG
    52.9 KB · Views: 490
Last edited:

konetzed

Dabbler
Joined
Aug 16, 2018
Messages
20
Beholder101 I have good news and bad news. Good news is you are not crazy. The bad news is the driver does not seem to work with 11.2 beta2.

I install the beta i could not create volumes but just got the 409 conflict as you do. I switched back to a freenas 11 stable install and everything works fine. Both freenas environments were fresh installs into kvm vms.

I think if you switch to using the 11 stable build all your issues will go away and openstack cinder will work as you expected.
 
Top