CakeDC Blog

TIPS, INSIGHTS AND THE LATEST FROM THE EXPERTS BEHIND CAKEPHP

Debug & Run Gitlab Pipelines In Your Local Environment

Some time ago, we established https://git.cakedc.com/ as our company workflow. Along with it we created automated tools to support a continuous integration environment, with automated deployments based on develop, qa, stage, master branches and some useful tools to run on specific branches. We used jenkins for a long time, then switched to gitlab around version 6 (more than 5 years ago!) and we've been using it since.

Gitlab provides a very powerful way to configure your pipelines and define specific docker images to be used as your runners. So we defined our own runner image and configured it to provide the typical dependencies needed to run static analysis tools, unit tests and other utilities as part of our build process. For example, one typical build file for a simple CakePHP project could be:

# https://hub.docker.com/r/jorgegonzalezcakedc/cakephp-runner

image: jorgegonzalezcakedc/cakephp-runner:yarn

 

before_script:

  # install ssh-agent

  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

 

  # run ssh-agent

  - eval $(ssh-agent -s)

 

  # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store

  - ssh-add <(echo "$SSH_CI_PRIVATE_KEY")

  - echo "$SSH_CI_PRIVATE_KEY" > /root/.ssh/id_rsa

  - chmod 600 /root/.ssh/id_rsa

 

  # replace git oauth key for composer

  - sed -i "s/__TOKEN__/$GIT_OAUTH_TOKEN/g" ~/.composer/auth.json

 

variables:

  # Configure mysql service (https://hub.docker.com/_/mysql/)

  MYSQL_DATABASE: app_test

  MYSQL_ROOT_PASSWORD: root

 

stages:

  - test

  - deploy

 

test:mysql:

  services:

  - mysql:5.7.22

  script:

  - echo $MYSQL_PORT_3306_TCP_ADDR

  - composer install --verbose --prefer-dist --optimize-autoloader --no-progress --no-interaction

  - ( [ -f vendor/bin/phpunit ] && echo "phpunit already installed";) || composer require phpunit/phpunit

  - mysql -uroot -p$MYSQL_ROOT_PASSWORD -h $MYSQL_PORT_3306_TCP_ADDR -e 'CREATE DATABASE test_myapp_template;';

  - DATABASE_TEST_TEMPLATE_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/test_myapp_template" bin/cake db_test -i

  - DATABASE_TEST_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/app_test" DATABASE_TEST_TEMPLATE_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/test_myapp_template" QUEUE_DEFAULT_URL='null:///?queue=default&timeout=1' vendor/bin/phpunit --verbose --colors=never

  except:

  - tags

 

deploy_develop:

  stage: deploy

  environment:

    name: develop

    url: https://www.cakedc.com

  script:

    - cd deploy && php deployer.phar deploy develop -vvv

  only:

    - develop

  except:

    - tags

 

In this case, on every push to the "develop" branch, we'll run unit tests of the project, then call the specific deploy script to push the project to our CI environment.

This process is usually smooth and clean,  if it's not,  then you need to debug why the runner is failing at some step.

 

One possible answer to this situation would be to dockerize the project and ensure the local docker version matches 100% the runner being used, so you don't have surprises while running your pipelines.  This process is actually done in some projects to ensure we match specific versions and dependencies. But for legacy projects, it's useful to have something more or less generic that just works™ and does not require the effort to dockerize. In this case, and going back to the topic of the article, how can we debug the issues locally without waiting for the pipelines to run? (Note I'm using Ubuntu 16.04 as my dev environment, and possibly aiming to switch to 20.04 LTS soon…)

  • Install docker in your local machine see https://docs.docker.com/get-docker/ 

  • Ensure docker is up and running sudo service docker start

  • Install the gitlab apt repositories curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash

  • Install the gitlab-runner package sudo apt update && sudo apt install -y gitlab-runner

  • Go to your local project, where the .gitlab-ci.yml file is located

  • Run your pipeline locally, note you can pass environment variables via --env and you can name the target you want to build, in this case test:mysql gitlab-runner exec docker test:mysql --env SSH_CI_PRIVATE_KEY="`cat ~/.ssh/id_rsa`" --env GIT_OAUTH_TOKEN="XXX"

  • If there's a problem with the pipeline, add a long sleep time in your .gitlab-ci.yml file to keep the pipeline up and running while you connect to it, for example after the like to run your unit tests, add a new line sleep 1337

  • Run your pipeline again, you'll notice it won't stop…

  • Open a new terminal and check the id of the docker instance using docker ps

  • You'll see a list of the active docker container IDs

  • Finally connect to the container using docker exec -it CONTAINER_ID bash

  • If bash is not available in the container, you'll need another way to connect to it (or another container)

 

Once you get access to the container, you can manually execute commands, check permissions, run shells, and debug db and code contents to hunt down the bug you should have already in a unit test…

This method saved me some time trying to understand a specific issue in a pipeline, I hope it'll save some of your time too!

 

Latest articles

Playing with the new CakePHP Queue

One of the topics discussed in the community is the benefit of a unified, officially supported, CakePHP Queue plugin. Queues are used in most of the projects nowadays and are a central utility plugin. During the CakeFest 2020 event, there were also a couple direct references from speakers: (https://speakerdeck.com/josegonzalez/building-and-releasing-a-cakephp-plugin/?slide=15c) and this comment from Mark Story: https://cakesf.slack.com/archives/C172CS4TE/p1602257791377500.   This motivated me to take a deeper look at the cakephp/queue plugin and write this blog post.   Here at CakeDC we've been using queues for a looooong time. Initially in CakePHP 2, we've used plugins like CakeResque with redis or custom workers tied to Amazon SQS queues. Then in CakePHP 3 & 4 we've been using mostly https://github.com/josegonzalez/cakephp-queuesadilla with redis or mongodb backends. https://github.com/cakephp/queue   First thing would be setting up the plugin in your project, we are going to use the example project we used in CakeFest 2020: https://github.com/cakephp/cakefest2020/#using-docker   So after setting up the project and running it via docker compose, we can proceed to setup the plugin via composer. We will need to add it as a repository and set  

Install via composer

  After the first release it'll be much easier, but for now you'll need to add the package to your composer.json   "repositories": [         {             "type": "vcs",             "url": "https://github.com/cakephp/queue.git"         }     ]   Then do composer require cakephp/queue -W   And install some transport as stated in the official documentation https://book.cakephp.org/queue/1/en/index.html#installation composer require enqueue/redis:^0.9 composer require predis/predis:^1   Ensure your redis server is up and running, you can check the commands sent to your local redis server using redis-cli monitor   Now we are ready to configure the queue, we'll create 1 default queue adding this to the config/app.php file       'Queue' => [         'default' => [             'url' => 'redis:',         ],     ],   Add a Job using `bin/cake bake job Example` and add some code to the execute method       public function execute(Message $message): string     {         $data = $message->getArgument('data');           // do some long operation with data         Debugger::log($data);         sleep(2);           return Processor::ACK;     }   I've added a command utility to enqueue a test message bin/cake bake command addJob       public function execute(Arguments $args, ConsoleIo $io)     {         $callable = [ExampleJob::class, 'execute'];         $arguments = ['id' => 1, 'data' => ['some' => 'data']];           QueueManager::push($callable, $arguments);     }   And finally we can start our worker using bin/cake worker to pull jobs from Redis and process them using the ExampleJob::execute method   Here's all the example code created: https://github.com/cakephp/cakefest2020/tree/feature/cakephp-queue - for your reference.   Please note the plugin is still a work in progress and there is no stable release as of now.  It's looking great so far and we plan to include it in our next projects!  

Best CakePHP Plugins

Members of our team had the privilege of helping with CakeFest 2020 this year. One added virtual feature was the giveaways from CakePHP, these were done in the form of fastest-to-answer, trivia, or participation (random draw).  One of the giveaway games was to share your favorite CakePHP plugin but like, how do we only pick one, right? Anyway… There was a lot of participation in this giveaway! A few people even named our CakeDC Users plugin as their favorite *cue blushing face*. But in all seriousness, I thought it would be a good idea to share with you some of the plugins that were named most useful/helpful by CakeFest attendees this year….   Like I mentioned, the CakeDC users Plugin: https://github.com/CakeDC/users Queue Plugin: https://github.com/dereuromark/cakephp-queue Bake: https://github.com/cakephp/bake DataTables: https://github.com/fheider/cakephp-datatables CakePHP-tools: https://github.com/dereuromark/cakephp-tools Authentication: https://github.com/cakephp/authentication CakePHP-image: https://github.com/josbeir/cakephp-image Fixturize: https://github.com/lorenzo/cakephp-fixturize CakePHP File-Storage: https://github.com/burzum/cakephp-file-storage Crud: https://github.com/FriendsOfCake/crud IDE Helper: https://github.com/dereuromark/cakephp-ide-helper Asset-Compress: https://github.com/markstory/asset_compress CakePHP Debug Kit: https://github.com/cakephp/debug_kit Plum-Search: https://github.com/skie/plum_search CakePHP API: https://github.com/cakedc/cakephp-api/ Bootstrap UI: https://github.com/friendsofcake/bootstrap-ui Trash: https://github.com/usemuffin/trash   You can check out the full list of CakePHP Plugins at Plugins.CakePHP.org.  Have you been utilizing these tools? If not, it may be a good idea to start… while each of these serve their own purpose, using all of them can make your baking process a lot easier.    Perhaps your favorite didn’t make this list? Tell us about it… email HERE. Or, tweet us @CakeDC, @CakePHPThanks for baking!  

CakeFest 2020 recap

Taking a deep breath….. We have made it through another successful CakeFest event.    We didn’t know exactly what to expect with a fully virtual event, as it opens the door for a list of things that can (and most likely will) go wrong. Would the speakers show up? Would the internet connections keep things from running smoothly? Would attendees enjoy the information?   The answer to all of those questions is yes.    The lineup this year was amazing, and we had speakers from 6 different countries! With the ever changing way of life, our team wanted to have a diverse group of speakers, with some talking about some pretty serious subjects - like Dr. Jennifer Akullian and the mental health issues faced in the tech world. Jen allowed for questions and how-to-handle scenarios, and worked through some tough issues with attendees. Workshops from Mark Story, Jorge Gonzalez, Mark Scherer and Jose Lorenzo provided incomparable information while building projects in real time. All of the talks provided useful information that we hope all attendees will take with them as they bake in the future.    Now, does all of this bragging mean we didn’t have any issues? No. As I said in our opening, I think our group is pretty awesome and perfect, but the fact of the matter is, no one is. When bringing people together from all over the world, it is hard to test and check every connection multiple times. We had our fair share of internet lag, connection problems and audio issues. You know what we did? We worked together, fixed it, switched around the schedule and made it happen.   Virtual CakeFest 2020 was a great success, and exceeded our expectations. We were able to gather bakers from all over the world, in real time, and host a live event! I couldn’t believe how interactive the audience was, and everyone is still baking together in our #cakefest chat channel. I hope everyone is as impressed with the turn out as our team is. I know what you’re thinking… when will the recordings be released. We are working on uploading, editing and releasing the videos ASAP. While we tried to find the best universal timezone, we understand that other obligations kept some from attending, and we want to share these videos with you quickly, so know that we are working on it.    To our attendees: THANK YOU. Thank you for joining, supporting our framework and keeping CakePHP alive.   

I would like to thank all of our speakers:

  Remy Bertot  Junichi Okuyama Mark Story Jorge Gonzalez Jose Lorenzo Mark Scherer Jose Gonzalez Cal Evans Ed Barnard Jennifer Akullian Mariano Iglesias Chris Nizzardini Juan Pablo Ramirez
 

A HUGE THANK YOU TO OUR SPONSORS:

   Cake Development Corporation
  Passbolt 
  Marks Software
  RingCentral  


  Here’s to planning next year’s event to be even bigger and better!  

We Bake with CakePHP