CATH-SM Server (BC² Conference 2021)

This is the example application for the workshop "Beyond the usual Docker tutorial - Web apps & CI".

• CATH-SM Server (BC2 Conference 2021)
  • Overview
  • Installation
    • Clone the Repository
    • Get the Docker Images
    • Configuration
    • Run the Example Application
  • Run the CATH-SM Client
  • CI/CD
    • FAQ

Overview

Here you'll find a list of commands on how to get things up & running. For the "real" README of this project, please have a look at README.original.md.

Installation

Clone the Repository

A copy of the repository is available in the home directory of your VM at $HOME/cathsm-server. But you can still clone the repository yourself with:

$ git clone https://git.scicore.unibas.ch/bienert/cathsm-server-bc2conf2021.git $HOME/cathsm-server

Get the Docker Images

There are two ways to obtain the Docker images for the workshop:

• Download them from the Docker registry using docker-compose

  $ docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml pull

• Build them via docker-compose

  $ docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml build

Configuration

As is, the example application is almost configured. There is only the IP address missing Django should respond to. This must be added to the famous ALLOWED_HOSTS variable. We serve this list via the Docker Compose .env file so changes to it do not require tinkering around with the container.

Check the IP address of your VM, open $HOME/cathsm-server/Docker/.env in a text editor, find the line with ALLOWED_HOSTS in it and add the IP address of your VM to it.

Alternatively let sed add the IP for you:

$ sed -i -e 's/ALLOWED_HOSTS=localhost,127.0.0.1/ALLOWED_HOSTS=localhost,127.0.0.1,<IP ADDRESS>/' $HOME/cathsm-server/Docker/.env

with <IP ADDRESS> replaced with the IP of your VM.

Run the Example Application

Run via Docker Compose, either in foreground

$ docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml up

or in daemon mode in the background

docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml up -d

You can turn off the application in daemon mode like this:

$ docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml down

An instance of cathsm-server should be available after start up at:

http://<IP ADDRESS>

You can also figure out the URL of your VM using host <IP ADDRESS> or dig -x <IP ADDRESS>.

Log entries are available via

$ docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml logs

And for individual containers via

# Get container names with `docker PS -a`
$ docker logs <CONTAINER NAME>

Run the CATH-SM Client

Since CATH-SM is a web API, not a web page, what you see in the browser when accessing the server is quite boring - it's advertising the API and Django's admin page in the background.

To play around with the API, there is an example client implementation, cathsm-client, which we installed on your VM. The CATH-SM server repository has some example sequence as input to the client.

Before running the client (obviously start your server first), enable the Python virtual environment that comes with it:

$ source $HOME/cathsm-client/venv/bin/activate

Then run the client like this in your VM:

$ $HOME/cathsm-client/scripts/cathsm-api --in $HOME/cathsm-server/example_data/P61851.fas --outdir results/ --api1_user bc2-2021@theb-si.de --api1_base=http://localhost:80

Make sure to use bc2-2021@theb-si.de as user as there are API-tokens set up for this address.

When running the client, you'll see the server talking to the CATH DB service first and to SWISS-MODEL afterwards.

CI/CD

Test and deploy the service automatically with GitLab CI.

Before deploying the service with the CI make sure that you are not already running an instance on the same machine: docker-compose -f $HOME/cathsm-server/Docker/docker-compose.yml down.

1. Register GitLab runners. For simplicity run both runners on the same machine.

   # Register a docker runner for running tests
   sudo gitlab-runner register --non-interactive --url "https://gitlab.com/" --registration-token "PROJECT_REGISTRATION_TOKEN" --executor "docker" --docker-image "ubuntu:latest" --description "docker-runner" --tag-list "docker"
   # Register a shell runner for deploying the service
   sudo gitlab-runner register --non-interactive --url "https://gitlab.com/" --registration-token "PROJECT_REGISTRATION_TOKEN" --executor "shell" --description "shell-runner" --tag-list "production"
   # List available runners
   sudo gitlab-runner list

   url and registration-token (and the newly registered runners) can be found on the GitLab web UI of your project: "Settings -> CI / CD -> Runners".

   ℹ️ It is also possible to register GitLab runners without sudo, however, it is necessary to start build processing with gitlab-runner run.

2. Add the content of the .env file to the CI variables on GitLab "Settings -> CI / CD -> Variables -> "Add variable". Key: ENV_FILE, Type: File, Value: the contents for the .env file.

   ⚠️ If you have a variable containing a $ character you need to escape it with another $, i.e. write $$.

3. Push a new commit to run the pipeline. Alternatively you can start the pipeline manually in the web UI "CI / CD -> Pipelines -> Run pipeline".

FAQ

Q: I'm getting

Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /home/gitlab-runner/builds/3yHi3o_-/0/bienert/cathsm-server-bc2conf2021/.git/
fatal: git fetch-pack: expected shallow list
fatal: The remote end hung up unexpectedly

What should I do?

A: The Git version on the GitLab runner machine is too old. You need to change the Git strategy for fetching changes. In the GitLab web UI, "Settings -> CI / CD -> General pipelines" change "Git strategy for pipelines" to "git clone" and "Git shallow clone" to 0.