What is package management in Python and why you should create your own package?

In Python, "pip" is a package manager that we use to install packages from many sources. It’s a similar tool to "composer" for PHP or "Maven" for Java. When installing packages, it will first resolve the dependencies, check if they are already installed on your project, and, if not, install them.

Most of the packages come from the main public source: PyPI. If you want a private package, you need to store and deploy your package differently.

For example, when working with an external provider, it will only create a part of your application. Then you can integrate their code as a dependency in your project. Or if you want to create a library and re-use it in several projects. That’s what we will use as an example in this blogpost.

The context

Suppose you want to develop a new library for your backend project, and you know this library will be used by another project. That’s is why you want to save it as a package in your Gitlab repository.

So you have two projects: MainBackendProject and NewLibrary.

The goal is to build a function hello_world() in the NewLibrary. Then you will have to deploy the project as a package in the package registry of Gitlab. Finally, you can call the function directly in your MainBackendProject.

Note: I will keep this tutorial as simple as possible, which means that you will only have the minimum setup working.

Let's get started

1. Build the python package

Create the directory of the project "NewLibrary". 

mkdir NewLibrary

Create a requirements.txt file and add these libraries to build and deploy your package: 

cd NewLibrary
# requirements.txt

wheel
twine

With:
wheel: provides command line tools to build the package and create the wheel in the "dist/" directory.
twine: publishes the package on a package registry (a private one or PyPi).

Install the requirements. To keep things simple, we will use the python virtual environment. 

python3 -m venv env && source env/bin/activate
pip install -r requirements.txt

Create a file setup.py and fill it with the information about your package.

# setup.py
from setuptools import setup, find_packages

setup(
   name="new_library_package",
   author="An amazing developer",
   description="Private Python library who provides incredible features.",
   packages=find_packages(),
   include_package_data=True,
   classifiers=[
       "Environment :: Web Environment",
       "Intended Audience :: Developers",
       "Operating System :: OS Independent"
       "Programming Language :: Python",
       "Programming Language :: Python :: 3.6",
       "Topic :: Internet :: WWW/HTTP",
       "Topic :: Internet :: WWW/HTTP :: Dynamic Content"
   ],
   python_requires='>=3.6',
   setup_requires=['setuptools-git-versioning'],
   version_config={
       "dirty_template": "{tag}",
   }
)

Add the following necessary files in a "new_library_package" folder:

mkdir new_library_package
# __init__.py
from .dummy_file import hello_world
# dummy_file.py
def hello_world():
   print("Hello New Library !")
   return

This is a recap of the minimal structure of your project: 

NewLibrary
├── env
├── new_library_package
    ├── __init__.py
    ├── dummy_file.py
├── setup.py
├── requirements.txt

Execute the setup inside the virtual environment and create the package:

python3 setup.py sdist bdist_wheel

You will see a dist folder with a file new_library_package-0.0.0-py3-none-any.whl: this is the package of your project.

You can commit your work and ignore the folders:

# .gitignore
build/*
dist/*
env/
*.egg-info/
.eggs/*

[Optional] If you want to share the .whl manually, you can install it on another project with pip install:

pip install new_library_package-0.0.0-py3-none-any.whl

Now, you can automate the process by deploying the package with a gitlab CI/CD pipeline.

2. Deploy your package on Gitlab with a tag

Create a gitlab-ci.yml file at the root of your project "NewLibrary" and commit it. The deployment will be done with twine, the library that we installed before.

# gitlab-ci.yml

stages:
  - build

build-package:
  stage: build
  image: python:3.9
  script:
    - apt-get update && apt-get -y upgrade && pip install --upgrade pip && pip install -r requirements.txt
    - python setup.py sdist bdist_wheel
    - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/*
  only:
    - tags
  interruptible: true

You will need to define a deploy token with the right privileges. For that, add a deploy token "gitlab-deploy-token" in Gitlab (Settings > Repository > Deploy tokens) with the following scopes: read_package_registry, write_package_registry.

Deploy the package with a tag:

git tag 0.0.1 -m "Release My new package"
git push origin 0.0.1

If everything goes well, you should be able to see that your job "build-package" is successful in Gitlab > CI / CD > Jobs.

Your package is now published on the Package Registry in the Gitlab of your project "NewLibrary" (Gitlab > Packages & Registries). The version of your package will be tag based.

3. Install and import the new package on another project as dependency through the Gitlab API

Create an access token in Gitlab > Settings > Access Tokens with the scope read_api to access all projects through the Gitlab API. Keep the name and the value.

cd MainBackendProject
python3 -m venv env && source env/bin/activate

[Optional] You can configure a PyPi repository in the pip.conf of the virtual environment. Check where your pip.conf is in your virtual environment with: 

pip config -v list

You will see something like: 

For variant 'venv', will try loading '/[...]/MainBackendProject/env/pip.conf'

Then edit this file: 

# pip.conf
[global]
index-url = https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/projects/<project-id>/packages/pypi/simple

With:
<personal_access_token_name>: the name of the token
<personal_access_token>: the value of the token
<project-id>: it’s the integer value, not the slug name of your project

Install the dependency in your project "MainBackendProject". It will automatically choose the latest version of your package if nothing is specified.

pip install new-library-package --no-deps

And if you don’t want to configure the index url in your pip.conf file, you can directly use the command line: 

pip install new-library-package --no-deps –index-url https://<personal_access_token_name>:<personal_access_token>@gitlab.example.com/api/v4/projects/<project-id>/packages/pypi/simple

Call the hello_world() function within your project, in a python file:

# index.py

from new_library_package.dummy_file import hello_world
hello_world()

Test it in the terminal:

python3 index.py
>>> "Hello New Library !"

Well done, you know now how to create and deploy your own private packages with Python through the Gitlab API.