/ ansible

Gitlab-CI for Ansible part 1


This is the first post in this series and first post on this blog, but it is not an introduction to Ansible or version controlling or anything else we are talking here. It is possible that I will write some intros about it but not this time. It is assumed you already know these parts and are looking for some guide or example how to glue these things together, how to start working with your Ansible code with all the good practices of CI / CD, trying to make it ready to deploy just after the first successful GIT commit, successful means it passed all the tests defined int your CI pipeline

In this post I will explain how to install Gitlab with Gitlab-CI and GitLab Runners and how to configure GitLab Runner to run your build and test your Ansible code inside isolated docker container on GitLab Runner.

Why Gitlab-CI ? I know there are lot of different and maybe better CI tools out there, I just don’t know them. And since you’ve found this guide most probably you already have GitLab for you source control or you think to use it, why not to start looking into its build-in CI functionality ?

In the next paragraphs you will see step-by-step manual guide, however beginning your journey with DovOps and automation you should probably start from automating all your CI/CD tools, Ansible roles for all that is shown below will be included later as well.


Installation of GitLab is very easy with omnibus package, which is GitLab’s recommended installation method. Omnibus is a way to package different services and tools required to run GitLab.

This installation includes also Mattermost – open source Slack alternative, hosted on GitLab and integrated with GitLab and external LDAP integration for authentication. The idea is to be able to install it anywhere, for example on VMs on home PC, without having DNS server configured or PKI available. This will be covered with static entries in hosts files and self-signed certificates. Of course this is not a good example for production environment.

For this lab I am using VMs with latest Centos 7 (7.3.1611) 25GB HD, 1 CPU and 2 GB of RAM. This is most probably not recommended setup, but at least you can easily do the same exercise on your laptop.


yum update -y
yum install curl policycoreutils openssh-server openssh-clients postfix openssl -y
systemctl enable postfix
systemctl start postfix

Install GitLab:

curl -O https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh
chmod +x script.rpm.sh
yum install gitlab-ce -y

Gitlab is ready it is time to create self-signed certificates, the following command will create certificate and key, just modify subj accordingly.
[ C – country, ST – state, L = locality (city), O – organization, CN – common name ( your servers FQDN ) ]

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -subj "/C=PL/ST=Mazov/L=Warsaw/O=Virtoffice/CN=gitlab.example.com" -keyout gitlab.example.com.key -out gitlab.example.com.crt

We will create similar pair for Mattermost frontend and local docker registry used later to keep docker images used for our CI tests

openssl req -new -newkey rsa:2048 -days 365 -nodes -x509  -subj "/C=PL/ST=Mazov/L=Warsaw/O=Virtoffice/CN=mattermost.example.com" -keyout mattermost.example.com.key -out mattermost.example.com.crt
openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -subj "/C=PL/ST=Mazov/L=Warsaw/O=Virtoffice/CN=gitlab.example.com" -keyout registry.gitlab.example.com.key -out registry.gitlab.example.com.crt

Now copy all 6 files to /etc/gitlab/ssl directory, if it doesn’t exist, create it

if [ ! -d /etc/gitlab/ssl ]; then mkdir -p /etc/gitlab/ssl; fi
cp *example.com.crt *example.com.key /etc/gitlab/ssl/

We are ready to configure GitLab now. The main configuration file is /etc/gitlab/gitlab.rb make a copy of it somewhere and use the following configuration instead

external_url "https://gitlab.example.com"
mattermost_external_url "https://mattermost.example.com"
mattermost_nginx['redirect_http_to_https'] = true
mattermost_nginx['ssl_certificate'] = "/etc/gitlab/ssl/mattermost.example.com.crt"
mattermost_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/mattermost.example.com.key"
mattermost['service_use_ssl'] = true
mattermost['service_enable_insecure_outgoing_connections'] = true

nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.example.com.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.example.com.key"

registry_external_url 'https://registry.gitlab.example.com:4567'
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.gitlab.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.gitlab.example.com.key"

gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_host'] = 'ldap.example.com'
gitlab_rails['ldap_port'] = 636
gitlab_rails['ldap_uid'] = 'uid'
gitlab_rails['ldap_method'] = 'ssl' # 'ssl' or 'plain'
gitlab_rails['ldap_bind_dn'] = 'CN=Username,CN=Users,DC=example,DC=com'
gitlab_rails['ldap_password'] = 'password'
gitlab_rails['ldap_allow_username_or_email_login'] = true
gitlab_rails['ldap_base'] = 'o=your_basdn'
gitlab_rails['ldap_block_auto_created_users'] = true

After that run the following command (run it anytime you modify /etc/gitlab/gitlab.rb)

gitlab-ctl reconfigure

Now open your browser and go to http://gitlab.example.com you should be redirected to https, accept security exception and you will see GitLab’s login page]

Congratulations! your GitLab is ready, you can now register your users, create project and work with the remote repositories. However, this is not the post about how to use GitLab in general, but GitLab’s built in Continuous Integration and Deployment features


Few words about architecture.

Two components are needed to start with CI/CD and one is already there, CI/CD it’s a part of GitLab, it is integrated web application with an API exposed. It manages your builds and provides web user interface. Second one is GitLab Runner wich basically processes your builds. It can be deployed on another host and works with GitLab CI/CD through an API.

Install Runner

In this example we will use another Centos7 VM

yum update -y
curl -O https://packages.gitlab.com/install/repositories/runner/gitlab-ci-multi-runner/script.rpm.sh
chmod +x script.rpm.sh
yum install gitlab-ci-multi-runner -y

Since we don’t have any DNS, let’s update the /etc/hosts file again on this VM as well gitlab.example.com mattermost.example.com registry.gitlab.example.com

Register Runner

To register runner you need to get a token, so go to your GitLab page, Admin area –> Runners and grab a registration token from there
We can run command

gitlab-runner register

and answer all the question in interactive mode, but again, we rather prefer non interactive as we can reuse and automate it later.

During the registration few parameters are mandatory, like registration token, gitlab api url or executor. Executor defines how to execute your build. Gitlab runners support few types of executors Shell, Docker, Docker-SSH, VirtualBox, Parallels, SSH or Kubernetes. What we would like to achieve here is to run our Ansible code against something. So the executor here should be able to run and test our Ansible code. The idea is to do it in the docker container. Public docker registry is full of docker images that can fulfill our requirement, however what if we would like to move one step forward, and not only be able to run and test ansible-playbook from that container, but also be able to test it against itself ? Not a bad idea ? This guide is build on Centos7, so let’s assume we are building our test automation for Centos7. We need an image now that behaves like complete Centos7 from were we will start ansible playbooks with target set to localhost . Still sounds great! The most tricky think is to get systemd working there, as most likely we would like to enable some services on the system. I will show you how to prepare docker image that can do all that, and how to use it with special executor type – docker-ssh

Install docker on GitLab Runner

Easiest way is to use script provided by docker

curl -sSL https://get.docker.com/ | sh
systemctl enable docker
systemctl start docker

Now we have to build a docker image. We can do it from GitLab runner, it already has all tools needed. Create some directory cd inside and create file named dockerfile with the following content:

FROM centos/systemd

MAINTAINER "My Name" <email@example.com>

RUN yum -y install epel-release; yum -y install openssh-server; yum -y install ansible; yum -y install git; yum -y install coreutils; yum -y install shadow-utils; yum clean all; systemctl enable sshd.service; echo "root:password" | chpasswd

CMD ["/usr/sbin/init"]

We will rebuild from original centos/systemd image, but we need to install few additional things, like ansible,

docker build --rm --no-cache -t centos7/systemd-ansible .

When it is done you should be able to see two local images

docker image ls
REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
centos7/systemd-ansible  latest              9df5bbd5d35c        20 seconds ago      308MB
centos/systemd           latest              0a3a4b7e94bd        2 weeks ago         193MB

Get GitLab’s certificate

mkdir /etc/gitlab-runner/certs
openssl s_client -connect gitlab.example.com:443 -showcerts < /dev/null | openssl x509 -outform PEM > /etc/gitlab-runner/certs/ca.crt

From here we have two options, we can register runner with default docker image from local repository

 --docker-image centos7/systemd-ansible

to do this we have to give one more parameter, which allows to pull from local repository if image is not available in registry

--docker-pull-policy if-not-present

Or we can push the image into GitLab’s build in private registry, configured previously

The following registration command will point to the default docker image on GitLab’s container registry, In the next step I will explain how to push our image there. Having images in private registry can be useful when we would like to increase our build capacity and install new runners. Configuring default docker image for our runner doesn’t mean we have to use it later, we can specify any image from any place in our pipeline, more about this in next posts

gitlab-runner register --non-interactive \
 --name ansible-runner1 \
 --url https://gitlab.example.com \
 --registration-token "j4tFMxEGhS9x3KSs3ghP" \
 --builds-dir /data/git_build \
 --tls-ca-file /etc/gitlab-runner/certs/ca.crt \
 --tag-list ansible \
 --docker-image registry.gitlab.example.com:4567/doe/docker/centos7/systemd-ansible \
 --docker-extra-hosts gitlab.example.com: \
 --executor docker-ssh \
 --ssh-user root \
 --ssh-password password \
 --docker-privileged false \
 --docker-tlsverify false

That’s it, Runner is registered! check the list of active runners

gitlab-runner list
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
ansible-runner1                                     Executor=docker-ssh Token=c0492e4c389b094e43dbacc958c54c URL=https://gitlab.example.com

Check /etc/gitlab-runner/config.toml

  name = "ansible-runner1"
  url = "https://gitlab.example.com"
  token = "df131cf3cce429efc3b04a2d47cffe"
  tls-ca-file = "/etc/gitlab-runner/certs/ca.crt"
  executor = "docker-ssh"
  builds_dir = "/data/git_build"
    user = "root"
    password = "password"
    tls_verify = false
    image = "registry.gitlab.example.com:4567/doe/docker/centos7/systemd-ansible"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    extra_hosts = ["gitlab.example.com:"]
    shm_size = 0

In case of any problems with registration you can debug gitlab-runner systemd unit

journalctl -u gitlab-runner.service -f

From now on all new projects in GitLab should be configured to use this runner automatically, to check that go to your project Settings –> Pipelines and check if you can see active runner tagged as ansible there, the meaning of this tag will be explained in next post


GitLab’s Conainer Registry

In addition to what we’ve done so far, now we are going to push centos7/systemd-ansible to docker registry configured on GitLab server in previous steps. Because we use self-signed certificate we need to do one more trick. On the gitlab runner create the file /etc/docker/daemon.json and put the following content insde

        "insecure-registries" : [ "registry.gitlab.example.com:4567" ]

Restart the docker deamon

systemctl restart docker

And test if you are able to login with one of your GitLab’s account

docker login https://registry.gitlab.example.com:4567

To be able to work with the registry we have to setup some project for it and set it as visible for public, we are not going to use this project for anything else, only to store our images for CI/CD tests.

Let’s call it docker-images and set visibility to public

Now go to the Registry tab under new project, you will find nice instruction in order to push images there


login to the registry as the previous project owner

docker login https://registry.gitlab.example.com:4567
Username: john.doe@foo.com
Login Succeeded

Now go to the the same directory where you placed docker file for the last build and let’s build again and tag it with private registry name

docker build --rm --no-cache -t registry.gitlab.example.com:4567/doe/docker-images/centos7/systemd-ansible .
docker push registry.gitlab.example.com:4567/doe/docker-images/centos7/systemd-ansible

Image is now also visible under project’s registry


To be continued …

In the next post with example of Ansible project