What's up with Python vs. Conda packages and environments?

Python packages are distributable archives of Python code. Even if a Python package includes code in another language, like C++ or Rust, the package would have Python "bindings" that allow that code to be called from Python. Virtual environments are isolated and independent sets of installed software, and it's a best practice to have a virtual environment per project. Python environments then refers to virtual environments for Python packages.

On the other hand, Conda packages are distributable archives of software in any language. They could include Python libraries, but system libraries, R libraries, Julia libraries, and other executable programs are also often distributed as Conda packages. Conda environments are accordingly virtual environments for Conda packages. However, Python has a special level of support in the Conda ecosystem, and you can actually use pip to install Python packages into a Conda environment as if it were a Python virtual environment. Those Python packages will be available in the Conda environment alongside any Conda packages.

Which kind of package ecosystem should I use?

• If you are working with exclusively Python packages, you will likely have a better experience using Python virtual environments and Python package tools. The experience will be simpler, and you'll be able to install packages more quickly and easily.
• However, if you are likely to need software in other languages, or if your Python dependencies have dependencies on system-level libraries, then Conda can be useful for handling everything together. This is can be common in scientific computing, data science, machine learning, and geospatial data analysis.

When do I need these different functionalities?

Functionality definitions

Managing Python installations

Installs multiple versions of Python, and provides ways to switch between them on a global or per-project basis.

Creating a virtual environment

Can create a virtual environment, i.e., a specific Python interpreter and an isolated set of installed Python packages. Typically, a Python project uses at least one virtual environment. This may be a standalone virtual environment stored in a central location, or it may be inside a project directory and tied to that project. There are two kinds of virtual environments relevant to Python development: regular Python virtual environments; and Conda environments which can include other non-Python software packages.

Managing dependencies

Provides a way to declare project dependencies and install them into a virtual environment.

Automation of multiple virtual environments

Automation for creation, dependency management, and task invocation for multiple virtual environments for a single project. This is often used to manage separate environments for tests, documentation generation, linting, and other tasks. Such tools often let you create virtual environments from a matrix of parameters.

Writing lock files

Lock files specify the exact set of packages installed in an environment with exact versions. They allow for reproducing that environment. This is most commonly useful if you are developing software that will be deployed, like an application or data pipeline. It's less relevant if you are developing a library.

Building packages

Creating file archives of Python code that can be distributed and installed by other users. There are two kinds of packages common for Python: regular Python packages distributed on PyPI, and Conda packages which can be code in any language but often is used with Python.

Publishing packages

Publishing Python packages to PyPI or Conda packages to conda-forge.

pyenv

pyenv is a tool that installs multiple versions of Python and lets you switch between them.

Website: https://github.com/pyenv/pyenv

Managing Python installations

pyenv has an pyenv install command for installing Python versions, a pyenv global command for setting a global Python version, and a pyenv local command for setting a per-project Python version.

Reference: https://github.com/pyenv/pyenv#what-pyenv-does

asdf

asdf is a tool for managing multiple versions of many language runtimes. It supports managing Python through the asdf-python plugin.

Website: https://asdf-vm.com/

Managing Python installations

asdf with the asdf-python plugin has an asdf install python command for installing Python versions, a asdf global python command for setting a global Python version, and a asdf local python command for setting a per-project Python version.

Reference: https://github.com/asdf-community/asdf-python

mise

mise (formerly rtx) is a tool for managing multiple versions of many language runtimes, including Python. It started out as a clone of asdf implemented in Rust.

Website: https://github.com/jdx/mise

Managing Python installations

mise has an mise install python@<version> command for installing Python versions and a mise use python@<version> command for setting an active Python version.

Reference: https://github.com/jdx/mise/blob/main/docs/python.md

venv

A Python standard library module for creating Python virtual environments.

Website: https://docs.python.org/3/library/venv.html

Creating a virtual environment

The venv module lets you create a virtual environment with python -m venv.

Reference: https://docs.python.org/3/library/venv.html#creating-virtual-environments

virtualenv

virtualenv is a tool for creating Python virtual environments. It has more functionality than the standard library venv module, such as creating virtual environments for any installed Python interpreter/version.

Website: https://virtualenv.pypa.io/en/latest/

Creating a virtual environment

You can create a virtual environment with virtualenv <envname>.

Reference: https://virtualenv.pypa.io/en/latest/user_guide.html

pip

Pip is the standard package installer for Python. Python installations generally come with pip pre-installed.

Website: https://pip.pypa.io/en/stable/

Managing dependencies

Pip has an pip install command for installing packages, either from a local distribution or from PyPI. It also has a requirements.txt requirements file specification that it can read and install from.

Reference: https://pip.pypa.io/en/stable/getting-started/

uv

uv is a fast Python installer and resolver designed to be a drop-in replacement for pip, pip-compile, and virtualenv.

Website: https://github.com/astral-sh/uv

Creating a virtual environment

FILL ME IN

Reference: https://github.com/astral-sh/uv?tab=readme-ov-file#getting-started

Managing dependencies

FILL ME IN

Reference: https://github.com/astral-sh/uv?tab=readme-ov-file#getting-started

Writing lock files

FILL ME IN

Reference: https://github.com/astral-sh/uv?tab=readme-ov-file#getting-started

conda

Conda is a cross-platform package and virtual environment manager. Conda packages can include libraries or executables in any language.

Website: https://docs.conda.io/en/latest/

Managing Python installations

Conda treats Python like any other conda package, and it allows you to install any version of Python as an isolated Python runtime in conda environment. To use that instance of Python, you will need to either activate the environment or use conda run -n <envname>.

Reference: https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html#managing-python

Creating a virtual environment

Conda allows you to create conda environments using the conda create -n <envname> command.

Reference: https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html#managing-envs

Managing dependencies

Conda has a environment.yml file specification for specifying dependencies to install into an environment.

Reference: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Conda has a environment.yml file specification for specifying dependencies to install into an environment. In it, you can have a subsection under pip for Python package dependencies.

Reference: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually

mamba

Mamba is a faster drop-in replacement for the conda package manager.

Website: https://mamba.readthedocs.io/en/stable/user_guide/mamba.html

Managing Python installations

As a drop-in replacement for conda, mamba can manage Python installations, which are treated like any other conda package.

Reference: https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#quickstart

Creating a virtual environment

FILL ME IN

Reference: https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#quickstart

Managing dependencies

FILL ME IN

Reference: https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#quickstart

FILL ME IN

Reference: https://mamba.readthedocs.io/en/latest/user_guide/mamba.html#quickstart

tox

tox is a tool for automated testing in multiple virtual environments. You can configure environments using an INI-format configuration file, and it provides CLI commands to easily run tasks in those environments.

Website: https://tox.wiki/en/stable/

Automation of multiple virtual environments

tox allows you to configure environments using a tox.ini configuration file.

Reference: https://tox.wiki/en/stable/user_guide.html

tox supports integration with conda environments and packages with the tox-conda plugin.

Reference: https://github.com/tox-dev/tox-conda

Nox

Nox is a tool for automated testing in multiple virtual environments similar to tox. Unlike tox, Nox uses a Python script for configuration rather than an INI file.

Website: https://nox.thea.codes/en/stable/

Automation of multiple virtual environments

Nox can use venv or virtualenv as a virtual environment backend. This will create Python environments, and it allows you to script Python package installations with pip.

Reference: https://nox.thea.codes/en/stable/config.html#defining-sessions

Nox allows you to use conda or mamba as a virtual environment backend. This will create conda environments and allows you to install conda packages.

Reference: https://nox.thea.codes/en/stable/config.html#configuring-a-session-s-virtualenv

Pipenv

Pipenv is a tool for managing Python environments and dependencies for a project. It uses its own Pipfile and Pipefile.lock file formats for specifying and locking dependencies.

Website: https://pipenv.pypa.io/en/stable/

Creating a virtual environment

When you use pipenv install for the first time in a project, it will automatically create a virtual environment for that project.

Reference: https://pipenv.pypa.io/en/latest/installation.html#virtualenv-mapping-caveat

Managing dependencies

When you use pipenv install to install a package, Pipenv tracks your dependencies in a file called Pipfile.

Reference: https://pipenv.pypa.io/en/stable/installation/#installing-packages-for-your-project

Writing lock files

Pipenv also manages a file called Pipfile.lock which tracks the exact versions of packages installed.

Reference: https://pipenv.pypa.io/en/latest/pipfile.html

Hatch

Hatch is a Python project manager with a variety of features in environment management as well as building and publishing packages.

Website: https://hatch.pypa.io/latest/

Creating a virtual environment

If you use Hatch's virtual environment management functionality, it will create a virtual environment named "default" that is the primary environment for a project.

Reference: https://hatch.pypa.io/latest/environment/#creation

Managing dependencies

Hatch lets you define dependencies in your project's pyproject.toml file. It will automatically install them in any virtual environments that it manages.

Reference: https://hatch.pypa.io/latest/environment/#dependencies

Automation of multiple virtual environments

Hatch lets you configure multiple virtual environments in your project's pyproject.toml file. It has advanced configurability like inheritance between environments and matrices of parameters. Hatch will keep dependencies in environments synchronized, and provides a hatch run command for executing scripts in one or multiple environments.

Reference: https://hatch.pypa.io/latest/environment/

Building packages

Hatch is a build frontend, and it also comes with a build backend named Hatchling. Both are compliant with modern Python packaging specifications (PEP 517 and PEP 621). You can use Hatch to build a package with the hatch build command.

Reference: https://hatch.pypa.io/latest/build/

Publishing packages

Hatch has a hatch publish CLI command for publishing a package to PyPI.

Reference: https://hatch.pypa.io/latest/publish/

Poetry

FILL ME IN

Website: https://python-poetry.org/

Creating a virtual environment

FILL ME IN

Reference: https://python-poetry.org/docs/basic-usage/#using-your-virtual-environment

Automation of multiple virtual environments

FILL ME IN

Reference: https://python-poetry.org/docs/managing-environments/

Managing dependencies

FILL ME IN

Reference: https://python-poetry.org/docs/basic-usage/#specifying-dependencies

Writing lock files

FILL ME IN

Reference: https://python-poetry.org/docs/basic-usage/#installing-without-poetrylock

Building packages

FILL ME IN

Reference: https://python-poetry.org/docs/libraries/#packaging

Publishing packages

FILL ME IN

Reference: https://python-poetry.org/docs/libraries/#publishing-to-pypi

PDM

FILL ME IN

Website: https://pdm.fming.dev/latest/

Creating a virtual environment

FILL ME IN

Reference: https://pdm.fming.dev/latest/usage/venv/

Automation of multiple virtual environments

FILL ME IN

Reference: https://pdm.fming.dev/latest/usage/venv/

FILL ME IN

Reference: https://pdm.fming.dev/latest/usage/venv/#virtualenv-auto-creation

Managing dependencies

FILL ME IN

Reference: https://pdm.fming.dev/latest/usage/dependency/

Building packages

FILL ME IN

Reference: https://pdm.fming.dev/latest/usage/publish/

Rye

FILL ME IN

Website: https://rye-up.com/

Managing Python installations

FILL ME IN

Reference: https://github.com/mitsuhiko/rye#some-of-the-things-it-does

Creating a virtual environment

FILL ME IN

Reference: https://rye-up.com/guide/basics/#first-sync

Managing dependencies

FILL ME IN

Reference: https://rye-up.com/guide/basics/#adding-dependencies

Writing lock files

FILL ME IN

Reference: https://rye-up.com/guide/sync/

Building packages

Rye provides a rye build command to build packages, which uses build under the hood.

Reference: https://rye-up.com/guide/publish/

Publishing packages

Rye provides a rye publish command to publish to PyPI, which uses twine under the hood.

Reference: https://rye-up.com/guide/publish/

pip-tools

pip-tools is a set of command-line programs for creating a lock file for Python packages and for syncing a virtual environment with a lock file. It uses the requirements.txt format used by pip.

Website: https://github.com/jazzband/pip-tools/

Writing lock files

The pip-compile command generates a fully pinned lock file (fully pinned requirements.txt) from a set of abstract requirements. The pip-sync command synchronizes an environment with a lock file.

Reference: https://github.com/jazzband/pip-tools/#example-usage-for-pip-compile

conda-lock

Conda-lock generates multiplatform fully reproducible lock files for conda environments.

Website: https://github.com/conda/conda-lock

Writing lock files

Use the conda-lock CLI with a conda environment.yml file containing abstract requirements to generate a lock file. You can use the conda-lock install command to install from a lock file.

Reference: https://github.com/conda/conda-lock#basic-usage

Conda-lock supports locking Python package dependencies installed by pip in a conda environment specification.

Reference: https://conda.github.io/conda-lock/pip/

Build

Build is a simple build frontend implementing modern Python packaging standards (PEP 517).

Website: https://pypa-build.readthedocs.io/en/stable/

Building packages

Use Build to build a package with python -m build. You will need to have a PEP 517 build backend specified in your pyproject.toml file.

Reference: https://pypa-build.readthedocs.io/en/stable/index.html

Twine

Twine is a tool for publishing Python packages to PyPA.

Website: https://twine.readthedocs.io/en/stable/

Publishing packages

Twine has a twine upload command for publishing a built package archive.

Reference: https://twine.readthedocs.io/en/stable/

Flit

Flit is a simple build tool for pure Python packages.

Website: https://flit.pypa.io/en/stable/

Building packages

Flit is a build frontend, and it includes a build backend flit_core. Both are compliant with modern Python packaging specifications (PEP 517 and PEP 621). You can use Flit to build a package with the flit build command.

Reference: https://flit.pypa.io/en/stable/index.html#usage

Publishing packages

Flit has a flit publish command for publishing a package to PyPI.

Reference: https://flit.pypa.io/en/stable/upload.html