CakeDC Blog

TIPS, INSIGHTS AND THE LATEST FROM THE EXPERTS BEHIND CAKEPHP

CakePHP Migrations plugin: easily version and deploy whole applications

This article is a quick introduction to the Migrations plugin, open sourced a few weeks ago by our company. You will see how simple it is to use the plugin and what you could do with it. I hope this article will show you the benefits of using migrations in your CakePHP applications and make you give it a try right after the reading!

Here is a one-sentence description of the plugin: the Migrations plugin allows developers to easily version and automate the creation / update process of any database schema and application data from the command line.

For information, CakeDC uses this plugin on its project since several years to make team collaboration and deployment easier. The plugin has been entirely rewritten a few months ago and fully tested (code coverage >95% as always at CakeDC) before being open sourced under the MIT license. It is now available to the community along with its documentation... and it is free!

Why is it useful?

It has been a while since companies integrated Source Code Management in their development process and CVS, SVN, Mercurial or Git are now common tools. Inspired from the open source movement it is also a good practice for single developers to version application source code.

As you might know, an application almost always depends of the database schema it is aimed to use... however it is not easy to version both the source code and database schema with a SCM. Let's take the example of a CakePHP application: until now the only way to do was to version a single file, either a sql dump or a CakePHP schema.php file generated with the cake schema shell. These two approaches are not very convenient to use on a daily basis, the first one forcing the developer to drop and recreate the whole database every time!

Moreover, a web application development is never really finished (there are always new features to add, software updates or bug fixing to do...) and deploying these change on a test or production server is always a delicate task.

Here comes the Migrations plugin! It provides a simple and easy way to version a database... and to perform many other different tasks thanks to its callback system. Here are some features:

  • keep a local database schema up-to-date: you just have to run all non applied migrations to update the local database schema to the latest version
  • make team work easier: when several developers work on the same application it is important that all of them work with the same database schema during all the development cycle. With migrations every commit is tied to the database schema at this precise instant, which makes easy switching branches and resetting a branch to a specific commit.
  • make installation and updates easier: ready to push the new version of your application live? You will only have to push the sources on the server and run all non applied migrations!
  • migrate more than database schema: the callback system allows you to do everything you want before (or after) applying (or reverting) each migration. Here are some examples: creating an initial admin account, add initial or test data to the application (lorem ipsums, categories, content...), update values from the database, send an email if debug > 0... The only limit may be your imagination ;)

Where can I find the code?

Announced a few weeks ago, a packaged version of the plugin can be downloaded from the "Plugins" section of CakeDC.com. This page contains a link to download the 1.0 version, the plugin documentation and the Github project for tickets and direct Git access to the repository.

To make people aware of the need to show their support to the Cake Software Fundation by donating a few bucks (this is unfortunately not done enough), the plugin was first available to donors only. The "Download without donation" button was added later, when the repository was made public! However, if you find this plugin useful please consider making a donation to the CSF... that is the best thing you could do for thanking us.

Click here to lend your support to: cakephp1x and make a donation at www.pledgie.com !

Even better! A sample application was also released for those who want to see how migrations could be used and integrated in an application. To play with it, Download the code or git clone the project using:

git clone git://codaset.com/cakedc/sample-migrations-application.git sample_migrations

You will only need to create a database.php configuration file and update CakePHP's core location to make the application work. Git users, run

git submodule init
git submodule update

to automatically add the migrations plugin as a submodule!

What do I need to use it in my application?

Note: the packaged plugin is for the CakePHP 1.3 version only. You can either download the 1.3-beta package of the framework, or use the 1.2 branch available in the Git repository.

Adding the plugin to an existing application is very simple. If you downloaded the archive containing the plugin code, unzip it in the "/plugins/migrations" folder of your application. Git users can add it as a submodule with the following command:

git submodule add git://codaset.com/cakedc/migrations.git plugins/migrations

To check that it is installed correctly, execute the following command from your application root (it will display the available command to use the plugin):

cake migration help

If you encounter any problem here, please read the official documentation about CakePHP's console usage.

How does it work?

This post is not aimed at providing a comprehensive tutorial on how to use the plugin, thus I will just introduce the most useful commands along with some use cases.

For a complete documentation, please read the official documentation provided on the plugin page. For a simple (but useful for understanding purpose) use case you can take a look at the sample application introduced above. Going through the commit history will allow you to understand how migrations could be used in a development process.

Create a migration

To generate a new migration, type the following command

cake migration generate

The tool will ask you to give a name to the migration and suggest to do a dump of the current database schema. If a "schema.php" file is found in the application, it will ask you if you want to generate a diff between this schema and your current database one.

Generated migration files will be added to the "/config/migrations" application directory.

Apply / Revert migrations

When you pull an application containing migrations, several commands are available to apply or revert migrations. The simplest one is:

cake migration

It will display all the found migrations along with their status (applied or not applied) and id number. Just enter a migration number to update your database to the correct version. Some convenience commands are also available. You can use:

cake migration up, down, all or reset

These commands will respectively:

  • apply the next migration
  • revert the latest applied migration
  • apply all non applied migrations (and thus update the schema to the most recent version)
  • revert all applied migrations (and empty the database)

Migrations for plugins

Adding plugins to an existing application often implies adding new tables to the database or altering existing ones. The Migrations plugin brings a quick and efficient way to automate this installation. On the one hand developers can easily add necessary migrations to their plugin (making upgrades easier), on the other hand users can apply them as easily.

The only difference compared with commands introduced above is the parameter "-plugin pluginname" that needs to be added. Here is how the user will install the database for the newly added / updated plugin "test":

cake migration run all -plugin test

I would like to highlight the fact that callbacks allow the developer to do everything they want before / after each migration. It is convenient for adding initial data, and one can even implement a callback method opening the bootstrap.php file to append plugin's configuration entries there (it is just an example ;)).

... going further

Of course, feel free to add any remark or example of migrations use in the comments.

As this post is not aimed at providing support for the plugin, I recommend you to use the official tools available:

  • If you found a bug or want to suggest enhancements: open a ticket!
  • An installation problem or a question about the plugin usage? Ask your question to the community!
  • You would like a custom version of this plugin, or professional related services... contact us, it is our job ;)

I hope you enjoyed this post, it is now time for you to start playing with the Migrations plugin...

Latest articles

A CakePHP Docker Development Environment

We sponsor a monthly CakePHP training session (register here https://training.cakephp.org ) where we cover different topics about the framework. One of our sessions, the "Getting Started with CakePHP 4" is aimed to help developers starting a new project quickly and following the best practices.   Our previous "recommended" quick setting for a CakePHP development environment was using a vagrant box. See details here:  https://www.cakedc.com/jorge_gonzalez/2018/01/17/using-a-vagrant-box-as-quick-environment-for-the-getting-started-with-cakephp-training-session. However, we've switched internally to use docker as our primary development environment and also we started using docker in our training sessions.   Here's a quick overview of a simple docker based development environment for CakePHP.  

1. Create a new CakePHP project skeleton using 

composer create-project cakephp/app myproject   A new folder "myproject" will be created with a CakePHP project skeleton inside. Go to this new directory and proceed.  

2. Create a new "docker-compose.yaml" file with the following contents

version: '3' services:   mysql8:     image: mysql:8     restart: always     container_name: mysql     environment:         MYSQL_ROOT_PASSWORD: root         MYSQL_DATABASE: my_app         MYSQL_USER: my_app         MYSQL_PASSWORD: secret     volumes:       - ./:/application     ports:       - '9306:3306'     cakephp:     image: webdevops/php-apache:8.0     container_name: cakephp     working_dir: /application/webroot     volumes:       - ./:/application     environment:       - WEB_DOCUMENT_ROOT=/application/webroot       - DATABASE_URL=mysql://my_app:secret@mysql/my_app     ports:       - "8099:80"
 

3. Run "docker-compose up"

You'll create 2 containers named mysql and cakephp -  check the docker-compose configuration to see default database and users created in the mysql container, and the same environment params passed to the cakephp container via DATABASE_URL to allow the cakephp container to connect with the mysql database.   NOTE: the ports exposed are 9306 for mysql and 8099 for cakephp webserver. You can list them using docker-compose ps.  

4. Access your database and cakephp shell

  • To access the database you can use the command:
mysql --port 9306 -umy_app -psecret my_app   To restore a database dump for example, you can use the command: curl -L https://raw.githubusercontent.com/CakeDC/cakephp4-getting-started-session/master/my_app.sql |mysql --port 9306 -umy_app -psecret my_app   You can also configure any database tool to access the database in: localhost:9306  
  • To access the cakephp environment and shell you can use the command:
docker exec -it --user application cakephp bash   You'll go to the webroot folder, so in order to run the cake shell you'll need to: cd .. bin/cake 
  Now you have a working environment to play with the training session contents.   In this previous article, we covered another approach to setting up a local docker environment: https://www.cakedc.com/rochamarcelo/2020/07/20/a-quick-cakephp-local-environment-with-docker    We hope to see you in our next training session! https://training.cakephp.org   

Updating Model Layer

One reason to migrate from CakePHP 2.x to newer versions, is the very powerful ORM system that was introduced in CakePHP 3.x.  

Improved ORM Objects

The CakePHP model layer in CakePHP 3.x uses the Data Mapper pattern. Model classes in CakePHP 3.x ORM are split into two separate objects. Entity represents a single row in the database and it is responsible for keeping record state. Table class provides access to a collection of database records and describe associations, and provides api to work with a database. One notable change is afterFind callback removal. In CakePHP 3.x, it is possible to use Entity level getters to provide calculated fields on entity level.  

Association Upgrade 

In CakePHP 2.x associations are defined as arrays properties like this:     public $belongsTo = [         'Profile' => [             'className' => 'Profile',             'foreignKey' => 'profile_id',         ]     ];   In CakePHP 3.x and 4.x associations are declared in the initialize method. This gives much more flexibility in association configuration.     public function initialize(): void     {         $this->belongsTo('Profile', [                 'className' => 'Profile',                 'foreignKey' => 'profile_id',         ]);              }   Or using setters syntax, it could be done this way: public function initialize(): void     {         $this->belongsTo('Profile')         ->setForeignKey('profile_id')     }  

Behavior Upgrade

In CakePHP 2.x, behaviors are initialized as arrays properties:     public $actsAs = [         'Sluggable' => [             'label' => 'name',         ],     ];   In CakePHP 3.x and 4.x,  behaviors are configured in the initialize method. This gives much more flexibility in configuration, as in params it's possible to pass anonymous functions.     public function initialize(): void     {         $this->addBehavior('Sluggable', [             'label' => 'name',         ]);              }  

Validation Upgrade

In CakePHP 2.x, behaviors are  initialized as arrays properties:     public $validation = [         'title' => [             'notBlank' => [                 'rule' => ['notBlank'],                 'allowEmpty' => false,                 'required' => true,                 'message' => 'Title is required'             ],         ],     ];   In CakePHP 3.x and 4.x, validation is defined in validationDefault method which builds validation rules.     public function validationDefault(Validator $validator): Validator     {         $validator             ->scalar('title')             ->requirePresence('title', 'create')             ->notEmptyString('title');           return $validator;     }   Additionally, CakePHP introduced the buildRules method, which is where  described foreign keys constraints, uniqueness, or business level rules.     public function buildRules(RulesChecker $rules): RulesChecker     {         $rules->add($rules->existsIn(['user_id'], 'Users'));         $rules->add($rules->isUnique(['username'], __('username should be unique')));                            return $rules;     }  

Finder Methods

In CakePHP 2.x, the custom finder method is called twice - before and after fetching data from the database, which is defined by the $state parameter. Parameter $query contains current query state, and in $results passed data returned from database.     protected function _findIndex($state, $query, $results = array()) {         if ($state == 'before') {             $query['contain'] = ['User'];         } else {             // ...         }     }       In CakePHP 3.x, custom finder method accepts query object and some options passed from client code and returns an updated query. This allows for combining multiple finder methods in the same call, and has better grained finder logic.     public function findIndex(Query $query, array $options): Query     {         return $query->contain(['Users']);     }   The afterFind method could be implemented with the Query::formatResults method, which accepts an anonymous function to map each collection item.

Why Database Compression?

Nowadays people are not concerned about how large their database is in terms of MB. Storage is cheap. Even getting cheap SSD storage is not a big deal.    However, this is true if we are talking about hundreds of MB or even several GB, but sometimes we get into a situation where we have massive amounts of data (i.e Several tables with lots of longtext columns). At this point it becomes a concern because we need to increase the hard disk size, and find ourselves checking to see  if the hard disk is full several times per day or week, etc.   Now, if you have faced a situation like this before, it's time to talk about database compression. Compression is a technique, developed theoretically back in the 1940s but actually implemented in the 1970s. For this post we will focus on MySQL compression, which is performed using the open-source ZLib library. This library implements the LZ77 dictionary-based compression algorithm.   Before going into MySQL compression details, lets name some of the main DBMS and their compression techniques:

  • MySQL: ZLib (LZ77) [1]
  • Oracle: Oracle Advanced Compression (Proprietary)[2]
  • Postgres: PGLZ or LZ4 (if added this option at compilation level) [3]
  • DB2: Fixed-length compression or Huffman in some systems [4]
  So, now that we know this useless information, lets learn how to implement this in MySQL.   Firstly, you need to know that you CAN'T enable compression if:
  • Your table lives into `system` tablespace, or
  • Your tablespace was created with the option `innodb_file_per_table` disabled.
  It is important to test if the compression is the best solution for you.  If you have a table with a lot of small columns, you will probably end up with a larger-size table after "compressing" because of the headers and compression information. Compression is always great when you have longtext columns which can be heavily compressed.   Then, to enable compression for a table, you just need to include the following option when your table is created, or execute it as part of an alter statement: ROW_FORMAT=COMPRESSED These are the basics but you may find more useful information in MySQL manual.   You can also take a look at Percona which implements a Column level compression. This is interesting if you have a table with a lot of small fields and one large column, or if you have to optimize your database as much as possible. [6]   Finally, just say that even that storage is cheaper than ever, the amount of information has increased as well and we are now using and processing an incredible amount of data... so it looks like compression will always be a requirement.   I hope you find this information useful and please let me know if you have any questions or suggestions below in the comments section.

  [1]:https://dev.mysql.com/doc/internals/en/zlib-directory.html  [2]:https://www.oracle.com/technetwork/database/options/compression/advanced-compression-wp-12c-1896128.pdf  [3]:https://www.postgresql.org/docs/devel/runtime-config-client.html  [4]:https://www.ibm.com/docs/en/db2-for-zos/12?topic=performance-compressing-your-data  [6]:https://www.percona.com/doc/percona-server/8.0/flexibility/compressed_columns.html

We Bake with CakePHP