Provide developer documentation on migrations #2742

DeepDiver1975 · 2016-12-01T14:30:50Z

@PVince81 mind reviewing? THX

mention-bot · 2016-12-01T14:30:51Z

@DeepDiver1975, thanks for your PR! By analyzing the history of the files in this pull request, we identified @RealRancor to be a potential reviewer.

PVince81 · 2016-12-01T14:32:31Z

developer_manual/app/schema.rst

+
+
+As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
+Instead all migration step which not yet have been executed will be executed.


"migration steps" (plural)

PVince81 · 2016-12-01T14:32:48Z

developer_manual/app/schema.rst

+As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
+Instead all migration step which not yet have been executed will be executed.
+
+How to develop a migration?


Remove question mark

PVince81 · 2016-12-01T14:33:14Z

developer_manual/app/schema.rst

+
+  <?php
+
+  namespace OCA\dav\Migrations;


Replace "dav" with "MyAppName" ?

PVince81 · 2016-12-01T14:34:01Z

developer_manual/app/schema.rst

+  }
+
+Within the up() method the schema changes are to be added by working on the `Class Schema <http://www.doctrine-project.org/api/dbal/2.5/class-Doctrine.DBAL.Schema.Schema.html>`_.
+Within the down() method the schema changes are to be reverted.


is it mandatory to implement down() ? What if the changes are not reversible like some repairs or deleting a column ?

It is only mandatory if we ever allow doangrading 🙈

PVince81 · 2016-12-01T14:34:34Z

developer_manual/app/schema.rst

+Within the up() method the schema changes are to be added by working on the `Class Schema <http://www.doctrine-project.org/api/dbal/2.5/class-Doctrine.DBAL.Schema.Schema.html>`_.
+Within the down() method the schema changes are to be reverted.
+
+3. Test your migration step by executing


Unit tests for migrations, does that make sense / is it possible / wanted ?

because all steps are executed upon installation there is no real need from my pov

PVince81 · 2016-12-01T14:34:42Z

developer_manual/app/schema.rst

+
+   ./occ migrations:execute --up dav 20161130090952
+
+In order to revert the migration you an run


PVince81 · 2016-12-01T14:34:57Z

developer_manual/app/schema.rst

+
+   ./occ migrations:execute --down dav 20161130090952
+
+By executing up and down can explicitly apply the changes and roll them back.


s/can/you can/

PVince81 · 2016-12-01T14:35:54Z

developer_manual/app/schema.rst

+4. Bring the migration live
+
+For the real app upgrade process the app version has to be increased to trigger the migrations.
+This will apply all steps which have not yet been executed.


How does Doctrine know which migrations to run for which versions ?
How does Doctrine find the migration classes, do they all have to be in a specific package ?

How does Doctrine know which migrations to run for which versions ?

there is a table for each app holding the list of all already executed steps - no idea if we should put this in the documentation

Might be a good idea to add it, along with the relevant documentation, so that if anyone is new to Doctrine, they can quickly get up to speed.

Well - this is an implementation detail of doctrine migrations from my pov.
There are also not docs available upstream about these internals.

PVince81 · 2016-12-01T14:36:29Z

Thanks for the docs, that clarifies a bit. See comments.

DeepDiver1975 · 2016-12-01T14:59:35Z

How does Doctrine find the migration classes, do they all have to be in a specific package ?

regular php classloader mechanism

PVince81 · 2016-12-01T15:02:03Z

do they all have to be in a specific package

@DeepDiver1975 ^

You said there is a table to track which migrations have run already. But is there a table of the ones that haven't ? Or is the classloader trying to find all migration classes within the app's namespace in any namespace and sort them by version ?

DeepDiver1975 · 2016-12-01T15:04:19Z

Or is the classloader trying to find all migration classes within the app's namespace in any namespace and sort them by version ?

There is one namespace for each app \OCA\MyAppName\Migrations.
All classes within have the datetime of creation as class name - this defines the sort order.

PVince81 · 2016-12-08T17:55:49Z

@DeepDiver1975 can you add your answers to my questions inside the documentation ? Because other people will also wonder...

DeepDiver1975 · 2016-12-09T08:10:48Z

@PVince81 please have another look - thx

PVince81

Looks better.

A few minor comments and this is good to go!

PVince81 · 2016-12-09T08:52:13Z

developer_manual/app/schema.rst

+
+
+As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
+Instead all migration steps which not yet have been executed will be executed.


which have not yet been executed

PVince81 · 2016-12-09T08:54:34Z

developer_manual/app/schema.rst

+How are migrations detected and executed
+========================================
+
+For each app there will be a table holding the list of migrations which have been executed.


which have been executed during past upgrades (the precision makes it even clearer)

PVince81 · 2016-12-09T08:55:40Z

developer_manual/app/schema.rst

+========================================
+
+For each app there will be a table holding the list of migrations which have been executed.
+This list is compared to all migration classes in the app's migration folder :file:`appinfo/migrations`.


... and the migrations that are not present in the table will be executed in the order of the class names alphabetically (or revision numbers ?)

phil-davis · 2016-12-09T09:01:59Z

developer_manual/app/schema.rst

+
+ownCloud uses `Doctrine Migrations <http://www.doctrine-project.org/projects/migrations.html>`_ to manage any kind of database changes.
+
+Any apps which wants to use this mechanism has to enable it in :file:`appinfo/info.xml` by adding


Any apps which want to use this mechanism have to enable it in
or
Any app which wants to use this mechanism has to enable it in

phil-davis · 2016-12-09T09:04:15Z

developer_manual/app/schema.rst

+
+By executing up and down you can explicitly apply the changes and roll them back - as long as down() is properly implemented.
+
+Because all migration steps will be executed upon installation there is no explicit need on unit tests.


s/on unit tests/for unit tests/

phil-davis

Just a couple of bits of English grammar.

DeepDiver1975 · 2016-12-09T10:20:29Z

Thanks for your feedback @phil-davis @PVince81

PVince81 · 2016-12-09T10:35:18Z

👍

DeepDiver1975 · 2016-12-09T10:37:00Z

@settermjd please review and merge - thx

PVince81 · 2016-12-09T10:43:35Z

Sorry, but I have another question: when creating an app from scratch, does the developer need to develop an initial migration that creates the database ? If yes, it would be good to add that to the docs as well.

DeepDiver1975 · 2016-12-09T10:50:38Z

the initial migration will create the table.

		$prefix = $this->connection->getPrefix();
		$table = $schema->createTable("{$prefix}activity");
		$table->addColumn(...);
		$table->addIndex(...);
		$table->addForeignKeyConstraint(...);

PVince81 · 2016-12-09T10:51:49Z

Thanks, can you add a note in the documentation that we need an initial migration instead of database.xml to create the tables initially ? The reason I was confused is because this was missing.

PVince81 · 2016-12-09T10:52:05Z

(no need for the code details, I think the Doctrine docs have examples)

PVince81 · 2016-12-09T15:41:16Z

Alternative: provide a good example of what a fully implemented migration looks like.

PVince81 · 2016-12-09T15:42:47Z

Ok, found an example here: https://akrabat.com/using-doctrine-migrations-outside-of-doctrine-orm-or-symfony/
Apparently you only set your stuff on the schema and that's it.

PVince81 · 2016-12-09T17:08:28Z

Note for future schema porters: if you see "text" in the MDB2 schema, don't use it! Use "string" instead in the migration definition. Because "text" will explode with Mysql if you set an index on it, see owncloud/customgroups#11 (comment)

settermjd · 2017-01-12T16:15:55Z

developer_manual/app/schema.rst

+
+
+As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
+Instead, all migration steps which have not yet been executed will be executed.


@DeepDiver1975 this doesn't fully make sense. The previous sentence says that they'll be ignored, yet this one says that any outstanding ones will be executed. Can you clarify pls?

This commit revises the content and separates out the larger code examples.

settermjd · 2017-01-12T16:31:13Z

I made a few revisions to the file. If no one objects, then I'm happy to merge the PR.

PVince81 · 2017-01-12T16:37:50Z

developer_manual/app/schema.rst

+   :linenos:
+
+To update the tables used by the application, update database.xml and increase
+the application version number in :file:`appinfo/info.xml` to trigger an update.


I think this is not true any more. Changes in this file are not used any more except for the initial creation.
Or at least that's why I observed when updating from stable9.1 to master, the code would not go there.

@DeepDiver1975 please confirm

PVince81 · 2017-01-12T16:39:52Z

developer_manual/examples/migrations/Version20161130090952.php

+     */
+    public function up(Schema $schema)
+    {
+        // this up() migration is auto-generated, please modify it to your needs


I suggest adding a real example here which would ease the life of many.

Have a look at https://github.com/owncloud/core/pull/26923/files#diff-f853a252a02574b7da2d70ca20e06a6eR18 for table creation (ignore hasTable for now).

Note that we also need to clarify whether we want developers to always explicitly specify the correct collation as it's not automatic. See this issue: owncloud/core#26925

I'll try and clarify this with @DeepDiver1975 next week

I'll hold off on making any changes until you've gotten further clarification.

DeepDiver1975 · 2017-01-26T12:24:50Z

We need to rework this - we had to remove doctrine/migrations ....

settermjd · 2017-01-26T13:01:41Z

@DeepDiver1975 what's being used in place of it?

PVince81 · 2017-01-26T17:34:48Z

@settermjd nothing yet... work in progress

settermjd · 2017-01-27T12:29:33Z

@PVince81 ok. thanks for the update.

PVince81 · 2017-02-01T14:32:46Z

@DeepDiver1975 time to update the docs ?

settermjd · 2017-02-14T13:57:31Z

@PVince81 @DeepDiver1975 can this be closed in deference to #2862?

DeepDiver1975 added this to the 9.2 milestone Dec 1, 2016

PVince81 reviewed Dec 1, 2016

View reviewed changes

developer_manual/app/schema.rst

<?php

namespace OCA\dav\Migrations;

Copy link

Contributor

PVince81 Dec 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace "dav" with "MyAppName" ?

PVince81 reviewed Dec 1, 2016

View reviewed changes

DeepDiver1975 force-pushed the migrations branch from 3b061d3 to f281ebe Compare December 9, 2016 08:09

PVince81 reviewed Dec 9, 2016

View reviewed changes

phil-davis reviewed Dec 9, 2016

View reviewed changes

Provide developer documentation on migrations - fixes #2726

c8f236a

DeepDiver1975 force-pushed the migrations branch from f281ebe to c8f236a Compare December 9, 2016 10:12

PVince81 assigned DeepDiver1975 Dec 12, 2016

settermjd modified the milestones: 10.0, Documentation Backlog Dec 20, 2016

settermjd added the Migration label Dec 20, 2016

settermjd modified the milestones: Documentation Backlog, 10.0 Dec 21, 2016

settermjd reviewed Jan 12, 2017

View reviewed changes

Revise the content in schema.rst

999e46b

This commit revises the content and separates out the larger code examples.

PVince81 reviewed Jan 12, 2017

View reviewed changes

settermjd added the Waiting On Review/Feedback label Jan 16, 2017

DeepDiver1975 added Needs More Work and removed Waiting On Review/Feedback labels Jan 26, 2017

PVince81 mentioned this pull request Feb 13, 2017

New collumn of execution time added in db schema of oc_jobs owncloud/core#27144

Merged

9 tasks

PVince81 closed this Feb 14, 2017

PVince81 deleted the migrations branch February 14, 2017 14:11



		As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
		Instead all migration step which not yet have been executed will be executed.


		./occ migrations:execute --up dav 20161130090952

		In order to revert the migration you an run


		./occ migrations:execute --down dav 20161130090952

		By executing up and down can explicitly apply the changes and roll them back.



		As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
		Instead all migration steps which not yet have been executed will be executed.


		ownCloud uses `Doctrine Migrations <http://www.doctrine-project.org/projects/migrations.html>`_ to manage any kind of database changes.

		Any apps which wants to use this mechanism has to enable it in :file:`appinfo/info.xml` by adding


		By executing up and down you can explicitly apply the changes and roll them back - as long as down() is properly implemented.

		Because all migration steps will be executed upon installation there is no explicit need on unit tests.



		As soon as migrations are enabled any existing :file:`appinfo/database.xml` will be ignored on upgrade of the app.
		Instead, all migration steps which have not yet been executed will be executed.

Provide developer documentation on migrations #2742

Provide developer documentation on migrations #2742

Conversation

DeepDiver1975 commented Dec 1, 2016

mention-bot commented Dec 1, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PVince81 commented Dec 1, 2016

DeepDiver1975 commented Dec 1, 2016

PVince81 commented Dec 1, 2016

DeepDiver1975 commented Dec 1, 2016

PVince81 commented Dec 8, 2016 • edited Loading

DeepDiver1975 commented Dec 9, 2016

PVince81 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phil-davis left a comment

Choose a reason for hiding this comment

DeepDiver1975 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

DeepDiver1975 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

DeepDiver1975 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

PVince81 commented Dec 9, 2016

settermjd Jan 12, 2017 • edited Loading

Choose a reason for hiding this comment

settermjd commented Jan 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DeepDiver1975 commented Jan 26, 2017

settermjd commented Jan 26, 2017

PVince81 commented Jan 26, 2017

settermjd commented Jan 27, 2017

PVince81 commented Feb 1, 2017

settermjd commented Feb 14, 2017

PVince81 commented Dec 8, 2016 •

edited

Loading

settermjd Jan 12, 2017 •

edited

Loading