Introduction

Following the previous article, I will continue to details the Gitlab CI/CD I’m currently using.

When developing an application, for me you always need to define a standard.
The coding must conform to general rules.

Formatting, syntax checking and best practices are always welcomed.
Even if it means more effort from the dev, you will gain later in code visibility.

So… how to do it ? Let’s talk a bit about Python.

I mainly use black & flake8.
For the tests I’m currently trying to use pytest.
I will guide you to configure them and define automated use with pre-commit.

Please note that the usage of a virtualenv is, as always, recommended:

virtualenv -p `which python3` venv
source venv/bin/activate

Files Configuration

Here are all the files that I add in my project directory.

requirements-dev.txt

It’s how I call the requirements used for dev.
But you can do what you want.

pre-commit>=1.20.0
flake8>=3.7.9
black>=19.10b0

You can then intall the dependencies inside the virtualenv by
pip install -r requirements-dev.txt

setup.cfg

This file is used by flake8.

[flake8]
exclude = .svn,CVS,.bzr,.hg,.git,__pycache__,.tox,venv,.eggs,build
max-line-length = 88
select = C,E,F,W,B,B950
ignore = E203, E501, W503

It allows you to ignore basic issue and use a correct line length for visibility with flake

pyproject.toml

This file is used by black, repeating similar settings than previously.

[tool.black]
line-length = 88
target-version = ["py36", "py37", "py38"]
include = '\.pyi?$'
exclude = '''
/(
\.git
| \.eggs
| \.idea
| \__pycache__
| venv
| front
| _build
| build
| dist
)/
'''

Those two file will help you to define what are the standard rule you will use while coding.
Now you should be able to use:

black . # Format your file
flake8 # Check your syntax

Ok, it’s a good start… but why not automatize that at the commit time ?
Here’s how to do it with pre-commit.

.pre-commit-config.yaml

repos:
- repo: https://github.com/ambv/black
rev: stable
hooks:
- id: black
language_version: python3.6
stages: [commit]
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v1.2.3
hooks:
- id: flake8
stages: [commit]

Now you can use pre-commit install to install your commit hook.
And with a simple git commit you now get (at least):

black................................................(no files to check)Skipped
Flake8...............................................(no files to check)Skipped

OK…. but you will follow it correctly, but is it the case for everybody ?
Not necessarily, so now we will use Gitlab CI/CD and pipeline to check the code quality.

CI/CD

I prefer separate my CI/CD in function by files, so this step is usually the linting phase for me.
It translate like that.

I mainly use alpine, for it’s portability and small size.

.lint-ci.yml

checker:
stage: checker
image: alpine
except:
refs:
- master
- dev
- /v*/
before_script:
- apk --no-cache add py3-pip python3-dev gcc linux-headers musl-dev libffi-dev openssl-dev git
- pip3 install flake8 mccabe pycodestyle pyflakes
script:
- python3 -m flake8 --statistics --count .
allow_failure: true

formater:
stage: formater
image: alpine
except:
refs:
- master
- dev
- /v*/
before_script:
- apk --no-cache add py3-pip python3-dev gcc linux-headers musl-dev libffi-dev openssl-dev git
- pip3 install black
script:
- black --check .
allow_failure: false

.gitlab-ci.yml

This allow you to start the job.

include:
- local: '.lint-ci.yml'

stages:
- formater
- checker

And “voilà”.
The pipeline will trigger on new commit except on main branches and version tags.
I’m still working on the correct “only” & “except” filter so you’ll have to set your own
preference, but the basics are here, good dev ! 🙂