A couple of questions regarding owncloud plugin

Status
Not open for further replies.

zephyrus00jp

Cadet
Joined
Feb 25, 2016
Messages
9
Thank you for sharing this great piece of software, FreeNAS and
providing the prepackaged plugin of owncloud.

I have a couple of questions regarding owncloud plugin.
I have run an installation of owncloud on Debian GNU/Linux on my main
desktop machine at home.
But now, I obtained an old HP Microserver which I intend to use as
24x365 home NAS, I thought it is a good idea to run owncloud there.

But when I tried to use owncloud plugin and learn how it fares, I realized I
am facing a few issues. So here are the questions.

Major Question 1: the version of bundled php in owncloud plugin is too old.
It is 5.4 and 5.6 seems to be the preferred choice today.

Is there a plan to update the version of the PHP in the prepared
plugin of owncloud?

Or is there a way for a third-party contributor like me to create
such a plugin EASILY? (I wonder if there is a shell script or
something to automate the creation of the prepared PLUGIN with updated PHP 5.6 and share such plugin?)

Major Question 2: can we switch to MySQL database from SQLite in the
owncloud plugin ? (It seems that the prepared PLUGIN is is pre-configured to
use SQLite if I am not mistaken. A fresh install of "raw" owncloud
usually asks us to select sqlite or mysql, but that questoin is not shown on
the first web login to owncloud plugin.)

Being able to convert SQlite to MySQL during run-time is very nice.
The suggested "php occ ..." command in owncloud manual for database conversion does not seem
to work with the prepackaged PLUGIN.
(The command needs to be issued under "www" user which owns owncloud's
config.php. Otherwise the above command refuses to run at all. Currently
config.php is owned by www. Usually, I tinker with the configuration
files under root account. I found out "su www ..." does not work.
"www" user is not usable from "su".
I see the error message: www user is not currently available, etc.
Also, the non-standard location of the PHP scripts, etc. [under /var/pbi/...]
may be interfering with the conversion process. But I am not sure abou this.
At least, on Debian GNU/linux installation of owncloud, when I
temporarily changed the ownership of config.php to "root" and tried
the conversion of database from the command-line, it worked there.
Debian uses "www-data" user for owncloud and thus config.php is owned by it.
But running such a command under root does not sound nice. I may change the permission of
files in unexpected ways.)

Anyway, I could not convert the SQlite db to mysql db easily.

Sub-question 2-b:

What is the recommended manner to update the database (from SQLite to
MySQL) from the viewpoint of the developer of the plugin who created it:
the choice of users such as www, the particular file layout chosen
and other factors obviously affects it.
This is NOT a general owncloud question, but very particular to the way the packaging of PLUGIN was
done. That is why I am asking this question No 2-b here.

Sub-question 2-c: plugin of owncloud that works with mysql

If it is not easy to change the database in the prepared PLUGIN, is
there a plan to offer owncloud plugin that pre-configures MySQL?
(Well, the user can create the MySQL database, user, and its password
from the Jail's shell feature once, and can proceed to setup owncloud
from its own web interface as is documented in owncloud manual after
the jail is populated with the prepackaged owncloud plugin..)
So I think it is just a matter preparing plugin of owncloud with necessary
mysql binaries, etc. in a desired directory hierarchy in avance.
Again, a standard script to install the desired packages to the
desired places in the selected file layout would be very handy.

BTW, the following is the performance situation gleaned by the use of
DTrace and WHY I want to switch to mysql.

Background: why I want to switch to MySQL: SQLite slows down owncloud.

Using SQLite is a performance hog.

I CAN NOT describe the performance problems below WITHOUT DTrace.
Kudos to FreeBSD developers and FreeNAS developers to incorporate
DTrace in the OS kernel and distribution.

With owncloud server that uses SQLite,
uploading takes a looooong time after many small files are added to
local directory that is managed by owncloud sync app under windows
(and other devices from the horror stories I read) .

Using DTrace on FreeNAS, I found that owncloud with SQLite on the server side
OPENs the database for each upload of a file, and SCANs (READs) a
VERY LARGE portion of database, THEN CLOSEs it at the end.
And the server REPEATs this. You can see the inefficiency immediately.

Real-world Example:
I have about 40 GB worth of files, among which are small data files
that I wanted to share using owncloud. (Now I count 63K+ files, but
this number contains a triplicate of latest data files that triggered the
slow performance issue that led me to post this inquiry.
These data sets contain 44921 files (strange the number is not multiple of three.) totaling 919MB.
(So actually, I began to experience problem when the number of files
exceeded about 18K+ and the size of the data is about 39GB. These were
mostly text files, MS-Office doc/ppt/xls files, JPEG photos and such.
The current figures contain the triplicates of a particular data set,
two of which have been added for performance benchmarks.)

For this dataset, the created SQLite owncloud.db on the server was almost 64MB.

Using DTrace, I found out that during the slow upload phase (a couple
of uploads or so every 10 seconds) after many small files are added to
local Windows PC, I saw owncloud.db is opened on the server, and then a lot
of reads of 1K block totalling a few dozen MB (close to 30-50MB
each 10 sec (!) of this db file, and then the file is closed and the
process repeats for each upload. No wonder the upload of a small file
is so slow. I mean, for uploading a small file of a few KB,
the server reads and scans and answer SQL queries against 64MB
database. Simply reading 30-40 MB takes close to half a second. And interpreting the database metastructure for the data adds overhead, of course.
This process is insane. Obviously, the way owncloud uses SQlite, it recreates the in-core data structure
by reading that many data and handles the SQL queries and start all
over for each uploaded file (!).

Converting sqlite database to mysql seems to correct the situation
SOMEWHAT since now mysql daemon sits on the server with the database
already read from the disk. It does not have to open and read the
database for each upload. This improvement may be a deal killer for
many users, and thus I am posting this inquiry.

Re the improvement by using mysql: Not perfect IMHO.

I tested the owncloud, nginx, mysql, php-fpm combination under another
jail to see the speed improvement. There, the large amount of read of DB for each upload obviously
disappears and, now, according to DTrace, too many calls of "lstat" by
PHP code becomes a problem although the speed measured by elapsed wall
time is probably is 2-3 x faster. But there is a room for improvement:
I mean "so what if one day upload becomes 8 hour upload" (!).
I think the throughput of small file upload is still not something on a par with Dropbox, its main contender.
If you are interested in "lstat" issues , see, for example,
php.webtutor.pl/en/2011/06/02/running-php-on-nfs-huge-performance-problems-and-one-simple-solution/
)

People who complained on the slow owncloud performance can see the innards of the issue using DTrace very easily. I recommend those who interested to try that themselves.

Again, thank you for sharing this great piece of software, FreeNAS and
plugins.

Although I have installed owncloud with mysql under another jail,
I prefer to use prepackaged plugin if the issues of outdated PHP and
database selection(MySQL) are solved.
This is because the more people use the prepackaged PLUGINs, the more
kinks are ironed out in the configuration, and security issues will be
scrutinized by more eyeballs IMHO.
(Debian's packages and their configurations are well thought out IMHO
thanks to such user feedback and contributions.)

TIA
 

Joshua Parker Ruehlig

Hall of Famer
Joined
Dec 5, 2011
Messages
5,949
1) No plan currently. There are issues with building plugins for FreeNAS/BSD that make it fairly difficult to update dependencies. It's possible but I'm not going to put the effort in if FreeNAS 10 is coming around fairly soon.
It's not easy to build plugins currently as I outline here.
https://forums.freenas.org/index.php?threads/more-plugins.14626/page-2#post-175364

2) You probably could, you'd need to install the mysql-server, and related php extension, then do the migration.
Though, in my opinion it's cleaner to just run owncloud in a standard jail until there's a version of the plugin that is designed to use mysql.

'su www' doesn't work because its default shell is nologin. change it first
Code:
chsh -s /bin/sh www
su www
cd /usr/pbi/owncloud-amd64/www/owncloud
/usr/pbi/owncloud-amd64/bin/php ./occ


2-b) Kinda covered what I would do above, but like I said I personally would just run in a standard jail or wait till the plugin is designed differently.

2-c) Yes, I plan on changing this with the release of FreeNAS10. If I change this now, current owncloud users installs would probably be broken when they try to update the plugin.
 

Joshua Parker Ruehlig

Hall of Famer
Joined
Dec 5, 2011
Messages
5,949
Rather than changing the shell for www, you could do 'sudo -u www'
the jail doesn't have sudo installed by default
 

zephyrus00jp

Cadet
Joined
Feb 25, 2016
Messages
9
Hi, thank you for the answers.
I think I will run in a jail environment for now. (I can always copy / remount my data if FreeNAS comes out and you can create a better owncloud plugin.)

I read
https://forums.freenas.org/index.php?threads/more-plugins.14626/page-2#post-175364
> ##MAJOR CAVEATS##
>
> you can't build PBIs with the currents ports tree. the current ports tree dropped support for building the old pkg format. But, the
>pbi-tools that work on FreeBSD 9 don't support the new package format.

This is tough. (Not exactly a chicken and egg problem or catch-22 sitaution, but sitll tough!).

> My solution is to use a ports tree from September 20, 2014.
>I manually update specific ports/dependencies.
>
> If you do update a port it may no longer build because the old pkg system doesn't understand "@dir" in the pkg-plist. you need to
> manually delete these.

I am not familiar with FreeBSD packaging system and so I better not try this.

> I think these issues will be resolved when FreeNAS 10 is out.

I hope so. For now, the jailed owncloud seems to work fine.

Thank you again for your sharing owncloud plugin.
 

Joshua Parker Ruehlig

Hall of Famer
Joined
Dec 5, 2011
Messages
5,949
I updated the owncloud plugin to 9.0.0 and was able to update the bundled php to 5.5
 

zephyrus00jp

Cadet
Joined
Feb 25, 2016
Messages
9
Thank you for the update.
Any reason not going to 5.6 all the way (maybe incompatible with the main FreeNAS script suite?).
Also, have you solved the issue of the incomplete package support? It seems to me that solving the package utility issue
will go a long way for better maintenance experience, so to speak.
But I suspect FreeNAS 10 is not far behind.

Thank you again for the maintenance of the package.
(Since my last post, I have installed UPS and a daemon to go with it so that second-hand HP microserver that runs FreeNAS will
shutdown cleanly when UPS is cut off from power grid. So I am all set.
But I have to ask the developer of the UPS software to make the daemon hang on when the USB cable is accidentally pulled off: the upsd daemon will quit if the communication is cut off after spewing out a couple of lines of message to syslog.
This is not good for a home use, especially where an earthquake can shake the machine, floor, etc. [I am living in Japan.])
I want the daemon to continue so that when someone notices the loose cable and plug it in, upsd daemon will operate normally... But I digress.
 

Joshua Parker Ruehlig

Hall of Famer
Joined
Dec 5, 2011
Messages
5,949
Thank you for the update.
Any reason not going to 5.6 all the way (maybe incompatible with the main FreeNAS script suite?).
Also, have you solved the issue of the incomplete package support? It seems to me that solving the package utility issue
will go a long way for better maintenance experience, so to speak.
But I suspect FreeNAS 10 is not far behind.

Thank you again for the maintenance of the package.
(Since my last post, I have installed UPS and a daemon to go with it so that second-hand HP microserver that runs FreeNAS will
shutdown cleanly when UPS is cut off from power grid. So I am all set.
But I have to ask the developer of the UPS software to make the daemon hang on when the USB cable is accidentally pulled off: the upsd daemon will quit if the communication is cut off after spewing out a couple of lines of message to syslog.
This is not good for a home use, especially where an earthquake can shake the machine, floor, etc. [I am living in Japan.])
I want the daemon to continue so that when someone notices the loose cable and plug it in, upsd daemon will operate normally... But I digress.
php5.6 wasn't in the ports tree I am using.

It's not a problem to solve. FreeBSD purposely depreciated the old pkg system and caused this problem.
FreeNAS plugins use PBI, which aren't supported on ports tree's after September 2014 on FreeBSD 9.
 
Status
Not open for further replies.
Top