Update gitlab CI to login before pulling the images

Currently, we are pulling images as anonymous users. When we pull the images we need to log in with the docker hub credentials in order to avoid the rate limit.

@kanitz Can you please have a look at this? I think you did something similar for the Dockerfiles travis CI.

changed milestone to %multiple

added Testing & CI/CD label

Authenticating with Docker Hub requires me to call docker login. But since we are using a Docker-based GitLab runner that would mean a Docker-in-Docker setup, which is asking for trouble.

I think you will need to figure this out with sciCORE if you want to avoid the Docker rate limits. There are some options, including:

• Trying to use the GitLab Container Registry (if available in our sciCORE GitLab instance)
• Set up a Dependency Proxy (if our GitLab instance is v13.7 or higher)
• Check/ask if there is a way the GitLab Docker runner can be configured to do the authentication

There is some more info here: https://stackoverflow.com/questions/64855241/gitlab-ci-service-and-docker-hub-auth

If you wanna try the Docker-in-Docker setup, we will need to replace the Conda image with a Docker image, see: https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#authenticate-with-registry-in-docker-in-docker (and best to read the entire doc)

But I would recommend against it, it's likely going to take quite some effort to set this up, and then there are caveats with Docker-in-Docker that are, as I mentioned, asking for trouble.

For me, the return on figuring this out is unacceptable at the moment. I'd rather wait for the limits to refresh, it's not that we have that many commits. We'd just need to re-run the CI on other days than fridays as well. At least until the migration to BioContainers is completed.

assigned to @gypas and unassigned @kanitz

I can try refreshing the CI in open !82 (merged) but could you please let me know since when? Are these daily or monthly limits? If monthly - should I wait until May?

https://www.docker.com/increase-rate-limits

The rate limits of 100 container image requests per six hours for anonymous usage, and 200 container image requests per six hours for free Docker accounts are now in effect. Image requests exceeding these limits will be denied until the six hour window elapses.

added Doing label

Will be addressed by solving #150 (closed). In the meantime, we can ask sciCORE about setting up the Dependency Proxy and/or Docker login at the level of the GitLab Docker runner, if possible. For now, we can just re-start tests failing due to the rate limit.

Since #150 (closed) is resolved, we should not have this issue anymore. Closing and we can reopen is some issue re-appears.

closed