There are several ways to run Python code locally. Often it is desired to run it in virtual environments and containers to be able to run multiple configurations in parallel and to easily remove configurations again.
As I developer I want to run everything in containers. Data scientists, data engineers and Python developers often seem to prefer virtual environments. I guess the main reason is that creating virtual environments from Python is easy. This post describes different options.
venv
venv is an easy option to use virtual environments.
The venv module supports creating lightweight “virtual environments”, each with their own independent set of Python packages installed in their site directories. A virtual environment is created on top of an existing Python installation
Install venv:
1
2
python -m pip install --user virtualenv
python --version
Activate the environment:
1
2
3
4
python -m venv my-env
source ./my-env/bin/activate
...
python3 -m pip install -r requirements.txt
Deactivate the environment:
1
deactivate
Miniconda
Miniconda also includes a virtual environment.
Miniconda is a free minimal installer for conda. It is a small bootstrap version of Anaconda that includes only conda, Python, the packages they both depend on, and a small number of other useful packages
Install Miniconda, for example via shell commands:
1
2
3
4
mkdir -p ~/miniconda3
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -o ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
~/miniconda3/bin/conda init zsh
Activate the environment:
1
2
conda create -n my-env
conda activate my-env
Deactivate the environment:
1
2
conda deactivate
vi ~/.zshrc
Containers
Execute the following command with Docker, Podman, etc.
1
docker run -it --name my-env -v "$PWD":/usr/src/myapp -w /usr/src/myapp python:alpine /bin/sh
And to start the container again:
1
2
docker start my-env
docker exec -it my-env /bin/sh
Jupyter
If you prefer to use notebooks, you can also run Anaconda in a container:
1
2
mkdir workspace && cd workspace
docker run -i -t -p 8888:8888 -v "$PWD":/home --name anaconda3 continuumio/anaconda3
From within the container install the dependencies and run Jupyter:
1
2
3
4
5
pip install transformers
pip install transformers[torch]
pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu
conda install pytorch torchvision torchaudio cpuonly -c pytorch
jupyter lab --ip='0.0.0.0' --port=8888 --no-browser --allow-root --notebook-dir=/home
Open Jupyter in your browser via the link which you’ll find in the terminal.
After you stop Jupyter and the container, you can restart everything:
1
2
3
docker start anaconda3
docker exec -it anaconda3 /bin/bash
jupyter lab --ip='0.0.0.0' --port=8888 --no-browser --allow-root --notebook-dir=/home
Next Steps
To learn more, check out the Watsonx.ai documentation and the Watsonx.ai landing page.