CakeDC Blog

TIPS, INSIGHTS AND THE LATEST FROM THE EXPERTS BEHIND CAKEPHP

Quick start with Migrations plugin

In a previous post I gave an overview of the CakePHP Migrations plugin, what it does and why you should use it in your applications. This article will explain how to use it in a practical way. We are going to bake a simple blog application recipe application and see how migrations are integrated in the development process. Since we recently moved all our open source projects on http://cakedc.github.com/, this sample application source code is also available there: Sample Migrations Application - Github (it is a CakePHP 1.3 application). Ready?

Bake a new application and add the migrations plugin

First of all, we need to bake a new CakePHP application. Easy enough to do using cake bake, then configure your database (an empty database is sufficient for now) and check that the home page is all green! If you have not set up your environment to use the CakePHP command line yet, take some time to do so... it worth it! Adding the migrations plugin might also be a straightforward task. You can either download the archive containing the plugin code and unzip it in the "/plugins/migrations" folder of your application, or  add it as a git submodule with the following command: git submodule add git://github.com/CakeDC/Migrations.git plugins/migrations Then check that it is correctly installed by executing the following command from your application root: cake migration help If you see a list of available commands you can move on next step.

Create initial tables and bake the MVC

We now need something to migrate! Let's create some tables in the database. The application will have Users who can publish Recipes, each one having several Ingredients (of course Ingredients can be used in many Recipes). Here is a SQL dump of this simple database schema: CREATE TABLE `ingredients` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; CREATE TABLE `ingredients_recipes` ( `id` int(11) NOT NULL AUTO_INCREMENT, `ingredient_id` int(11) NOT NULL, `recipe_id` int(11) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; CREATE TABLE `recipes` ( `id` int(11) NOT NULL AUTO_INCREMENT, `user_id` int(11) NOT NULL, `name` varchar(100) NOT NULL, `content` text NOT NULL, `created` datetime NOT NULL, `modified` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; CREATE TABLE `users` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, `password` varchar(255) NOT NULL, `created` datetime NOT NULL, `modified` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; As our goal here is not to focus on the application code itself, baked MVC from these tables might be sufficient... just run the command cake bake all for User, Recipe and Ingredient to bake'em all! At this point we must have an application with an initial architecture ready to share. To start from here, one will just have to checkout the related commit... but don't you see a problem with this? How will he create the initial database? Maybe we could send him the SQL dump by email, or better commit it with the application! It is where the Migrations plugin comes in.

Generate the initial migration

"Be kind with your coworkers and include the database schema with your code... along with some sample data." Let's use the migrations shell to generate an agnostic database schema containing our 4 tables, and an initial admin user account. To do so we just need to run the following command: cake migration generate After entering a name for the migration and selected the database dump option, we might have a new "/config/migrations" directory containing two files:
  • map.php representing the different migrations order,
  • name_of_the_migration.php a migration file containing all the necessary information to create your actual database. In the sample application it is named: "001_added_users_recipes_and_ingredients_tables.php". You might have noticed that we added a 001 prefix to the migration name to make it easier to see migrations order, it is a good practice.
We can now open the generated migration file (/config/migrations/001_added_users_recipes_and_ingredients_tables.php) and take a look at it. If you need more information and understand all available migration directives, you can read the plugin documentation. For now we are just going to focus on the empty "after()" callback. This callback is triggered once the migration has been executed, and allow you to do whatever you want, given the direction of the migration: applied (up) or reverted (down). We are going to use this callback to create an initial admin User. Here is the code of the callback (as you are a CakePHP developer you might understand it quite easily): function after($direction) { if ($direction === 'up') { if (!class_exists('Security')) { App::import('Core', 'Security'); } $User = $this->generateModel('User'); $user = array( 'User' => array( 'name' => 'admin', 'password' => Security::hash('unsecurepassword', null, true))); $User->save($user); } return true; } Notice the use of the generateModel() method provided by the Migrations plugin. It is a shorthand allowing you to cleanly load a model in the callback to insert new data or update the existing. We could explain the reason of it more deeply but it is not the goal of this article, so just keep in mind that it is the best way to load a Model from callbacks! Here we are! We can now share the application with anyone. After checked out the application, one will just have to run cake migration all to turn an empty database to a database containing all the needed tables, and an initial admin user to start using the application.

Categorize the recipes!

As the application evolves, we need to sort recipes by categories. This change involves two changes in the current database schema: a new categories table must be created, and a category_id field added to the recipes table. Note: If you later want to use the migrations diff feature to generate a migration containing a diff between your previous database schema and the current one, you have to generate a Cake Schema of your database at this point. Simply run cake schema generate. We can now update the recipes table and create a new categories table. Here is a simple SQL script: CREATE TABLE `categories` ( `id` int(11) NOT NULL AUTO_INCREMENT, `name` varchar(100) NOT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=latin1; ALTER TABLE `recipes` ADD `category_id` INT NOT NULL Bake the MVC for categories and update recipes view pages to display the category so the application reflect these database changes. Before sharing these code changes, we need to generate a second migration describing the above SQL snippet in an agnostic way... and creating initial categories! Nothing different than what we did previously: run cake migration generate, give a name to the migration, and choose between generating a diff from the schema.php file (if one was generated), generating a dump of the database (we will remove unnecessary instructions later) or generating an empty migration file. Once generated, it is always important to check the generated directives for the migration and fix them if needed. The migration must look like this: var $migration = array( 'up' => array( 'create_table' => array( 'categories' => array( 'id' => array('type' => 'integer', 'null' => false, 'default' => NULL, 'key' => 'primary'), 'name' => array('type' => 'string', 'null' => false, 'default' => NULL, 'length' => 100), 'indexes' => array( 'PRIMARY' => array('column' => 'id', 'unique' => 1), ), 'tableParameters' => array('charset' => 'latin1', 'collate' => 'latin1_swedish_ci', 'engine' => 'MyISAM'), ), ), 'create_field' => array( 'recipes' => array( 'category_id' => array('type' => 'integer', 'null' => false, 'default' => NULL) ), ), ), 'down' => array( 'drop_table' => array( 'categories' ), 'drop_field' => array( 'recipes' => array( 'category_id' ), ), ), ); If you understood what we did in the first migration callback to add an initial user you might be able to implement this one. We would like to add initial categories: Starters, Main Dish and Desserts. For lazy people, the code is here: function after($direction) { if ($direction === 'up') { $Category = $this->generateModel('Category'); $categories = array( array('name' => 'Starters'), array('name' => 'Main Dish'), array('name' => 'Desserts')); $Category->saveAll($categories); } return true; } Here we are again! The changes are ready to commit, and the commit will contains both code and database changes. One could update the database after checking out this commit by running: cake migration all.

The end

I hope this very simple use case and the code we built will help you to start using Migrations. As you could see it is very simple to use and will make your life much more easier: you would not have to worry anymore about the state of your database schema. The source code of this tutorial is available on Github. If you found any bug or have any suggestion about the Migrations plugin, please create a ticket on Github. Comment this article if you have any question, and do not hesitate to share it if you found it useful! Finally, if you need help to use the plugin, ask a question on CakeQs.

CakePHP Migrations plugin: easily vers...

This article is a quick introduction to the Migrations plugin, open sourced a few weeks ago by our company. You will see how simple it is to use the plugin and what you could do with it. I hope this article will show you the benefits of using migrations in your CakePHP applications and make you give it a try right after the reading! Here is a one-sentence description of the plugin: the Migrations plugin allows developers to easily version and automate the creation / update process of any database schema and application data from the command line. For information, CakeDC uses this plugin on its project since several years to make team collaboration and deployment easier. The plugin has been entirely rewritten a few months ago and fully tested (code coverage >95% as always at CakeDC) before being open sourced under the MIT license. It is now available to the community along with its documentation... and it is free!

Why is it useful?

It has been a while since companies integrated Source Code Management in their development process and CVS, SVN, Mercurial or Git are now common tools. Inspired from the open source movement it is also a good practice for single developers to version application source code. As you might know, an application almost always depends of the database schema it is aimed to use... however it is not easy to version both the source code and database schema with a SCM. Let's take the example of a CakePHP application: until now the only way to do was to version a single file, either a sql dump or a CakePHP schema.php file generated with the cake schema shell. These two approaches are not very convenient to use on a daily basis, the first one forcing the developer to drop and recreate the whole database every time! Moreover, a web application development is never really finished (there are always new features to add, software updates or bug fixing to do...) and deploying these change on a test or production server is always a delicate task. Here comes the Migrations plugin! It provides a simple and easy way to version a database... and to perform many other different tasks thanks to its callback system. Here are some features:
  • keep a local database schema up-to-date: you just have to run all non applied migrations to update the local database schema to the latest version
  • make team work easier: when several developers work on the same application it is important that all of them work with the same database schema during all the development cycle. With migrations every commit is tied to the database schema at this precise instant, which makes easy switching branches and resetting a branch to a specific commit.
  • make installation and updates easier: ready to push the new version of your application live? You will only have to push the sources on the server and run all non applied migrations!
  • migrate more than database schema: the callback system allows you to do everything you want before (or after) applying (or reverting) each migration. Here are some examples: creating an initial admin account, add initial or test data to the application (lorem ipsums, categories, content...), update values from the database, send an email if debug > 0... The only limit may be your imagination ;)

Where can I find the code?

Announced a few weeks ago, a packaged version of the plugin can be downloaded from the new "Downloads" section of CakeDC.com. This page contains a link to download the 1.0 version, the plugin documentation and the Codaset project url for tickets and direct Git access to the repository. To make people aware of the need to show their support to the Cake Software Fundation by donating a few bucks (this is unfortunately not done enough), the plugin was first available to donors only. The "Download without donation" button was added later, when the repository was made public! However, if you find this plugin useful please consider making a donation to the CSF... that is the best thing you could do for thanking us. Click here to lend your support to: cakephp1x and make a donation at www.pledgie.com ! Even better! A sample application was also released for those who want to see how migrations could be used and integrated in an application. To play with it, Download the code or git clone the project using: git clone git://codaset.com/cakedc/sample-migrations-application.git sample_migrations You will only need to create a database.php configuration file and update CakePHP's core location to make the application work. Git users, run git submodule init git submodule update to automatically add the migrations plugin as a submodule!

What do I need to use it in my application?

Note: the packaged plugin is for the CakePHP 1.3 version only. You can either download the 1.3-beta package of the framework, or use the 1.2 branch available in the Git repository. Adding the plugin to an existing application is very simple. If you downloaded the archive containing the plugin code, unzip it in the "/plugins/migrations" folder of your application. Git users can add it as a submodule with the following command: git submodule add git://codaset.com/cakedc/migrations.git plugins/migrations To check that it is installed correctly, execute the following command from your application root (it will display the available command to use the plugin): cake migration help If you encounter any problem here, please read the official documentation about CakePHP's console usage.

How does it work?

This post is not aimed at providing a comprehensive tutorial on how to use the plugin, thus I will just introduce the most useful commands along with some use cases. For a complete documentation, please read the official documentation provided on the plugin page. For a simple (but useful for understanding purpose) use case you can take a look at the sample application introduced above. Going through the commit history will allow you to understand how migrations could be used in a development process.

Create a migration

To generate a new migration, type the following command cake migration generate The tool will ask you to give a name to the migration and suggest to do a dump of the current database schema. If a "schema.php" file is found in the application, it will ask you if you want to generate a diff between this schema and your current database one. Generated migration files will be added to the "/config/migrations" application directory.

Apply / Revert migrations

When you pull an application containing migrations, several commands are available to apply or revert migrations. The simplest one is: cake migration It will display all the found migrations along with their status (applied or not applied) and id number. Just enter a migration number to update your database to the correct version. Some convenience commands are also available. You can use: cake migration up, down, all or reset These commands will respectively:
  • apply the next migration
  • revert the latest applied migration
  • apply all non applied migrations (and thus update the schema to the most recent version)
  • revert all applied migrations (and empty the database)

Migrations for plugins

Adding plugins to an existing application often implies adding new tables to the database or altering existing ones. The Migrations plugin brings a quick and efficient way to automate this installation. On the one hand developers can easily add necessary migrations to their plugin (making upgrades easier), on the other hand users can apply them as easily. The only difference compared with commands introduced above is the parameter "-plugin pluginname" that needs to be added. Here is how the user will install the database for the newly added / updated plugin "test": cake migration run all -plugin test I would like to highlight the fact that callbacks allow the developer to do everything they want before / after each migration. It is convenient for adding initial data, and one can even implement a callback method opening the bootstrap.php file to append plugin's configuration entries there (it is just an example ;)).

... going further

Of course, feel free to add any remark or example of migrations use in the comments. As this post is not aimed at providing support for the plugin, I recommend you to use the official tools available:
  • If you found a bug or want to suggest enhancements: open a ticket!
  • An installation problem or a question about the plugin usage? Ask your question to the community on CakeQs!
  • You would like a custom version of this plugin, or professional related services... contact us, it is our job ;)
I hope you enjoyed this post, it is now time for you to start playing with the Migrations plugin...

File uploading, file storage and CakeP...

This article includes how to upload and store files, because I've seen a lot of discussion about that too, but if you're just interested in how to use the MediaView class scroll down.

Handling file uploads in CakePHP

First let's start with the required form, to create a file upload form all you have to do is this: echo $form->create('Media', array('action' => 'upload', 'type' => 'file')); echo $form->file('file'); echo $form->submit(__('Upload', true));   The "type" in the options of Form::create() takes post, get or file. To configure the form for file uploading it has to be set to file which will render the form as a multipart/form-data form. When you submit the form now, you'll get data like this in $this->data of your controller: Array
(
[Media] => Array
(
[file] => Array
(
[name] => cake.jpg
[type] => image/jpeg
[tmp_name] => /tmp/hp1083.tmp
[error] => 0
[size] => 24530
)
)
)
Ok, now the big question with a simple answer is where the file data should be processed, guess where. Right – in the model because it's data to deal with and validation to do against it. Because it's a recurring task to upload files I suggest you to write a behaviour for it or convert your existing component to a behaviour. If you keep it generic you can extend it with a CsvUpload, VideoUpload or ImageUpload behaviour to process the file directly after its upload or do special stuff with it, like resizing the image or parsing the csv file and store its data in a (associated) model. We're not going to show you our own code here for obvious reasons, but I'll give you a few hints what you can or should do inside of the behavior:
  1. Validate the uploaded field, the field itself contains already an error code if something was wrong with the upload. Here is a link to the php manual page that shows you the list of the errors that you can get from the form data. http://www.php.net/manual/en/features.file-upload.errors.php
  2. Validate the uploaded file, is it really the kind of file you want and does it really contain the data structure you want?
  3. Check if the target destination of the file is writeable, create directories, whatever is needed and error handling for it, I suggest you to use CakePHP's File and Folder classes for that.
  4. Add a callback like beforeFileSave() and afterFileSave() to allow possible extending behaviors to use them.

Database vs file system storage

Feel free to skip that part if you already store the files in the file system. Storing files in the database is in nearly all cases a bad solution because when you get the file it has to go its way through the database connection, which can, specially on servers that are not in the same network, cause performance problems.

Advantages of storage in the file system:

  1. Easy and direct file access, to parse them (csv, xml...) or manipulate them (images)
  2. You don't need to install any additional software to manage them
  3. Easy to move and mount on other machines
  4. Smaller then stored in a DB
The suggested solution is to store meta data of the file like size, hash, maybe path and other related info in a DB table and save the file in the file system. Some people come up with the security and want to store a file because of that in the database which is wrong. You should not store the file in a public accessible directory like the webroot of the application. Store it in another location like APP/media. You control the access to the file by checking the permissions against the DB records of your meta data and sending it by using the CakePHP MediaView class, I'll explain later how to use it. I don't say that storage of files inside the DB is in general a bad idea but for web based applications it is in nearly every case a bad idea.

File system Performance

A bottleneck in the long run on every file system is a large amount of files in a single directory. Imagine just 10.000 users and each has an individual avatar image. Further ext3 for example is limited to 32000 sub folders, other file systems have maybe similar restrictions. You can find a list of file system limitations here: http://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits To avoid performance problems caused by that you should store your files in a pseudo-random directory structure like APP/media/32/a5/3n/. This will also allow you to easily mount some of the semi-random created directories on another machine in the case you run out of disk space. /** * Builds a semi random path based on the id to avoid having thousands of files * or directories in one directory. This would result in a slowdown on most file systems. * * Works up to 5 level deep * * @see http://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits * @param mixed $string * @param integer $level * @return mixed * @access protected */ protected function _randomPath($string, $level = 3) { if (!$string) { throw new Exception(__('First argument is not a string!', true)); } $string = crc32($string); $decrement = 0; $path = null; for ($i = 0; $i < $level; $i++) { $decrement = $decrement -2; $path .= sprintf("%02d" . DS, substr('000000' . $string, $decrement, 2)); } return $path; } You should also know that php running in safe mode does not allow you to create more then one directory deep in one call. You have to take this in consideration, the above function does not cover that because safe mode is basically deprecated and will be also removed in php6

Sending a file to the client – or the unknown MediaView class

From what I've seen in the ruins of outsourced projects that asked us for rescue and also in the CakePHP googlegroup I think not many people are aware that CakePHP has a view that is thought to be used for downloads and display (images, text...) of files. It's called the MediaView class. http://api.cakephp.org/class/media-view I'll now explain you how to use this class to send files to the client. /** * Sends a file to the client * * @param string $id UUID * @access public */ public function download($id = null) { $this->Media->recursive = -1; $media = $this->Media->read(null, $id); if (empty($media)) { $this->redirect('/', 404, true); } $this->set('cache', '3 days'); $this->set('download', true); $this->set('name', $media['Media']['slug']); $this->set('id', $media['Media']['filename']); $this->set('path', APP . 'media' . DS . $media['Media']['path']); $this->set('modified', $media['Media']['modified']); $this->set('mimeType', $media['Media']['mime_type']); $this->set('extension', $media['Media']['extension']); $this->view = 'Media'; $this->autoLayout = false; if ($this->render() !== false) { $this->Media->updateAll( array('Media.downloads' => 'Media.downloads + 1'), array('Media.id' => $id)); } } You simply have to set autoLayout to false and the view class to media. $this->view = 'Media'; $this->autoLayout = false; There are a few view variables to set to “configure” the file download or display. To control if you want to make the client downloading the file or to display it, in the case of images for example, you simply set 'download' to true or false; $this->set('download', true); You can control the browser caching of the file by setting cache. Please not that you do not have to use caching if download is set to true! Downloads do not need caching. $this->set('cache', '3 days'); The next part might be a little confusing, you have “id” and “name”. Id is the actual file on your server you want to send while name is the filename under which you want to send the file to the client. “path” is the path to the file on the server. $this->set('name', $media['Media']['slug']); $this->set('id', $media['Media']['filename']); $this->set('path', APP . 'media' . DS . $media['Media']['path']); If you want to send a mime type that does not already in the MediaView class you can set it. $this->set('mimeType', $media['Media']['mime_type']); If you don't set it, the class will try to determine the mime type by the extension. $this->set('extension', $media['Media']['extension']); Note that you have to set the extension to make it work and that the extension is attached to the filename! If you store the filename with an extension you have to break it up. When everything is set you can check if render() was successfully and do whatever you want after that, for example count the download. if ($this->render() !== false) { $this->Media->updateAll( array('Media.downloads' => 'Media.downloads + 1'), array('Media.id' => $id)); }

Closing words

I hope you enjoyed reading the article and it helped you improving your knowledge about CakePHP. Feel free to ask further questions by using the comment functionality. Have fun coding!

Felix Geisendörfer - Javascript and Git

Felix gave a demonstration of the production level javascript separation and management that the team at Debuggable use in order to minimise the amount of Javasript that needs to be sent to the client for any specific page view, and to ensure the logic is separated into the pages that it is used for. This creates a better management system for Javascript than using a single file. In addition to this separation, Felix gave an overview of common practices and operations for using Git for version control in a day to day environment. This included: merges, conflict resolution, fast forwarding branches, and managing multiple repositories. Largely this presentation was an interactive one, and to gain the most out of it, you really needed to be there.

Marius Wilms - The CakePHP Media Plugin

If Marius had more than an hour to talk about the Media Plugin, he most certainly would have taken it. To go over the features and functionality of the entire plugin would have been many hours as there is a lot there. A brief touch on the features provided by the plugin was discussed, with some examples. Requirements are in the high end, but considering the state of PHP and the upcoming version of CakePHP, developers should be moving forward in terms of their PHP version and library support anyway. The Media plugin requires CakePHP 1.2.x.x and PHP 5.2.0+. It enables the transfer, manipulation and embedding of files in many varied ways. You can find the media plugin at: http://github.com/davidpersson/media Marius' focus was on doing media manipulation and embedding "properly", and identified that while there are lots of user contributions floating around the net, none of them were meeting his needs and were flexible enough. One of the main points he made here was that if done incorrectly, potential security risks arise due to command line interaction and file saving. Validation was one particular section of the code that made this a tricky plugin to develop, but allowed tests to be implemented to ensure security. Some common points that we hear all the time came through, and they make sense for CakePHP as well as any web application for security reasons:  

  1. Don't trust users supplied filenames
  2. Don't store files in an accessible webroot, rather have them accessible to scripts.
  3. Make the upload location (and local filenames) unguessable (like referencing files by UUIDs)
The media plugin contains about 8 new rules for file validation purposes to ensure that submitted data meets the application needs. Beyond validation, it handles all kinds of uploads, HTTP Post, Remote HTTP and local file inclusion.
A console is included to initialize the default directory structure, and as such, could be included as part of a deployment script with the CakePHP console.examples.
To ensure flexibility of use, a behavior is included to allow attachment to any number of models, and generioc storage and linking provided to ease integration into existing apps.
Marius concluded his talk with a plea for feedback. There are plenty of people using the plugin, but more feedback is required to ensure its the best it can be, and that all bugs  (if any) are squashed. Checkout the code at: http://github.com/davidpersson/media

We Bake with CakePHP