Some time ago, we established https://git.cakedc.com/ as our company workflow. Along with it we created automated tools to support a continuous integration environment, with automated deployments based on
master branches and some useful tools to run on specific branches. We used jenkins for a long time, then switched to gitlab around version 6 (more than 5 years ago!) and we've been using it since.
Gitlab provides a very powerful way to configure your pipelines and define specific docker images to be used as your runners. So we defined our own runner image and configured it to provide the typical dependencies needed to run static analysis tools, unit tests and other utilities as part of our build process. For example, one typical build file for a simple CakePHP project could be:
# install ssh-agent
- 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
# run ssh-agent
- eval $(ssh-agent -s)
# add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
- ssh-add <(echo "$SSH_CI_PRIVATE_KEY")
- echo "$SSH_CI_PRIVATE_KEY" > /root/.ssh/id_rsa
- chmod 600 /root/.ssh/id_rsa
# replace git oauth key for composer
- sed -i "s/__TOKEN__/$GIT_OAUTH_TOKEN/g" ~/.composer/auth.json
# Configure mysql service (https://hub.docker.com/_/mysql/)
- echo $MYSQL_PORT_3306_TCP_ADDR
- composer install --verbose --prefer-dist --optimize-autoloader --no-progress --no-interaction
- ( [ -f vendor/bin/phpunit ] && echo "phpunit already installed";) || composer require phpunit/phpunit
- mysql -uroot -p$MYSQL_ROOT_PASSWORD -h $MYSQL_PORT_3306_TCP_ADDR -e 'CREATE DATABASE test_myapp_template;';
- DATABASE_TEST_TEMPLATE_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/test_myapp_template" bin/cake db_test -i
- DATABASE_TEST_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/app_test" DATABASE_TEST_TEMPLATE_URL="mysql://root:$MYSQL_ROOT_PASSWORD@$MYSQL_PORT_3306_TCP_ADDR/test_myapp_template" QUEUE_DEFAULT_URL='null:///?queue=default&timeout=1' vendor/bin/phpunit --verbose --colors=never
- cd deploy && php deployer.phar deploy develop -vvv
In this case, on every push to the "develop" branch, we'll run unit tests of the project, then call the specific deploy script to push the project to our CI environment.
This process is usually smooth and clean, if it's not, then you need to debug why the runner is failing at some step.
One possible answer to this situation would be to dockerize the project and ensure the local docker version matches 100% the runner being used, so you don't have surprises while running your pipelines. This process is actually done in some projects to ensure we match specific versions and dependencies. But for legacy projects, it's useful to have something more or less generic that just works™ and does not require the effort to dockerize. In this case, and going back to the topic of the article, how can we debug the issues locally without waiting for the pipelines to run? (Note I'm using Ubuntu 16.04 as my dev environment, and possibly aiming to switch to 20.04 LTS soon…)
Install docker in your local machine see https://docs.docker.com/get-docker/
Ensure docker is up and running sudo service docker start
Install the gitlab apt repositories curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
Install the gitlab-runner package
sudo apt update&&
sudo apt install -y gitlab-runner
Go to your local project, where the .gitlab-ci.yml file is located
Run your pipeline locally, note you can pass environment variables via
--envand you can name the target you want to build, in this case
test:mysqlgitlab-runner exec docker test:mysql --env SSH_CI_PRIVATE_KEY="`cat ~/.ssh/id_rsa`" --env GIT_OAUTH_TOKEN="XXX"
If there's a problem with the pipeline, add a long sleep time in your .gitlab-ci.yml file to keep the pipeline up and running while you connect to it, for example after the like to run your unit tests, add a new line
Run your pipeline again, you'll notice it won't stop…
Open a new terminal and check the id of the docker instance using docker ps
You'll see a list of the active docker container IDs
Finally connect to the container using docker exec -it CONTAINER_ID bash
If bash is not available in the container, you'll need another way to connect to it (or another container)
Once you get access to the container, you can manually execute commands, check permissions, run shells, and debug db and code contents to hunt down the bug you should have already in a unit test…
This method saved me some time trying to understand a specific issue in a pipeline, I hope it'll save some of your time too!