Middleware schema migration

[carpus]

Hey everyone,

I wanted to contribute something to the cloud sync middleware, which requires changing the database schema (replacing a column, more specifically the boolean "follow_symlink" with the text column "symlink"). I have already found other migration stuff, and I have a migration for the actual data:

Code:

async def migrate(middleware):
    tasks = await middleware.call('datastore.query', 'tasks.cloudsync')
    for task in tasks:
        await middleware.call("datastore.update", "tasks.cloudsync", task['id'], {
                'symlinks': 'FOLLOW' if task['follow_symlinks'] == True else 'IGNORE'
            })

However, this does not migrate the actual database schema. I cannot find any existing migrations doing this, nor can I find any documentation on this. The files in middlewared/plugins/datastore, which seems to be the source for e.g. the 'datastore.query' call, does contain a schema.py file. But this file only contains internal functions.

Is there a way to migrate the schema of a table in a datastore or does it get migrated automatically somehow?

Thanks!

 

[anodos]

Hey everyone,

I wanted to contribute something to the cloud sync middleware, which requires changing the database schema (replacing a column, more specifically the boolean "follow_symlink" with the text column "symlink"). I have already found other migration stuff, and I have a migration for the actual data:

Code:

async def migrate(middleware):
    tasks = await middleware.call('datastore.query', 'tasks.cloudsync')
    for task in tasks:
        await middleware.call("datastore.update", "tasks.cloudsync", task['id'], {
                'symlinks': 'FOLLOW' if task['follow_symlinks'] == True else 'IGNORE'
            })

However, this does not migrate the actual database schema. I cannot find any existing migrations doing this, nor can I find any documentation on this. The files in middlewared/plugins/datastore, which seems to be the source for e.g. the 'datastore.query' call, does contain a schema.py file. But this file only contains internal functions.

Is there a way to migrate the schema of a table in a datastore or does it get migrated automatically somehow?

Thanks!

I think following symlinks in a cloudsync task is inherently problematic. For instance, if your dataset contains a backup of user home directory that contains a wine profile (which often has symlinks back to `/`) you can create an infinite loop.

 

[carpus]

You are right, which is why I would like to extend the symlink option beyond the current option to follow symlinks (rclone -L) with the additional option to copy symlinks as a reference (rclone -l), where the symlink is converted to a text file with an .rclonelink file extension. This would mitigate the problem you mentioned while still being able to copy symlinks.

 

[anodos]

With schema changes, you will need to create an alembic revision. C.F. https://github.com/truenas/middleware/commit/eea0f3f6a24598bfb3acd093e8897b4ebf88fdc3

see documentation here: https://github.com/truenas/middleware/blob/master/docs/source/database/migrations.rst