From a664d96e69081a73578d93a62b797cb177b61bf0 Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 23 Jul 2026 17:24:06 +0100 Subject: [PATCH 1/3] Initial changes for using pdm --- episodes/12-virtual-environments.md | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 5e9e7354f..21630ef76 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -72,15 +72,9 @@ and manage our external dependencies. So what exactly are virtual environments, and why use them? -A Python virtual environment helps us create an **isolated working copy** of a software project -that uses a specific version of Python interpreter -together with specific versions of a number of external libraries -installed into that virtual environment. -Python virtual environments are implemented as -directories with a particular structure within software projects, -containing links to specified dependencies -allowing isolation from other software projects on your machine that may require -different versions of Python or external libraries. +A Python virtual environment helps us create an **isolated workspace** for our software project. +This workspace comes with a Python interpreter together with the versions of any external libraries that your project needs (e.g. NumPy or SciPy). +Python virtual environments are implemented as directories with a particular structure, containing links to specified dependencies allowing isolation from other software projects on your machine that may require different versions of Python or external libraries. As more external libraries are added to your Python project over time, you can add them to its specific virtual environment @@ -127,7 +121,7 @@ from different virtual environments. ### Managing Python Virtual Environments -There are several commonly used command line tools for managing Python virtual environments: +There are many commonly used command line tools for managing Python virtual environments: - `venv`, available by default from the standard `Python` distribution from `Python 3.3+` - `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions @@ -135,15 +129,13 @@ There are several commonly used command line tools for managing Python virtual e - `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community) - `poetry`, a modern Python packaging tool which handles virtual environments automatically +- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by OpenAI +- `pdm`, a modern Python package and dependency manager supporting the latest PEP standards. While there are pros and cons for using each of the above, all will do the job of managing Python virtual environments for you and it may be a matter of personal preference which one you go for. -In this course, we will use `venv` to create and manage our virtual environment -(which is the preferred way for Python 3.3+). -The upside is that `venv` virtual environments created from the command line are -also recognised and picked up automatically by the IDEs we will use in this course, -as we will see in the next episode. +In this course, we will use `pdm` to create and manage our virtual environment. ### Managing External Packages From b7ca966a8611aa23cf98b3379d67b364a7ba310d Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Wed, 19 Aug 2026 18:20:45 +0100 Subject: [PATCH 2/3] Update a few more sections to use pdm in environment episode --- episodes/12-virtual-environments.md | 204 ++++++++++++++-------------- learners/setup.md | 1 + 2 files changed, 105 insertions(+), 100 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index 21630ef76..e8f0140f3 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -7,7 +7,8 @@ exercises: 0 ::::::::::::::::::::::::::::::::::::::: objectives -- Set up a Python virtual environment for our software project using `venv` and `pip`. +- Set up a Python virtual environment for our software project using `pdm`. +- Declare, install and update our project's external dependencies using `pdm`. - Run our software from the command line. :::::::::::::::::::::::::::::::::::::::::::::::::: @@ -87,14 +88,14 @@ that they make sharing your code with others much easier Here are some typical scenarios where the use of virtual environments is highly recommended (almost unavoidable): -- You have an older project that only works under Python 2. - You do not have the time to migrate the project to Python 3 +- You have an older project that is pinned to an older version of Python. + You do not have the time to migrate the project to a newer version, or it may not even be possible as some of the third party dependencies - are not available under Python 3. - You have to start another project under Python 3. + have not been updated. + You have to start another project that requires a recent version of Python. The best way to do this on a single machine is to set up two separate Python virtual environments. -- One of your Python 3 projects is locked to use +- One of your projects is locked to use a particular older version of a third party dependency. You cannot use the latest version of the dependency as it breaks things in your project. In a separate branch of your project, @@ -124,84 +125,77 @@ from different virtual environments. There are many commonly used command line tools for managing Python virtual environments: - `venv`, available by default from the standard `Python` distribution from `Python 3.3+` -- `virtualenv`, needs to be installed separately but supports both `Python 2.7+` and `Python 3.3+`versions -- `pipenv`, created to fix certain shortcomings of `virtualenv` +- `virtualenv`, needs to be installed separately but offers more features and supports more Python versions - `conda`, package and environment management system (also included as part of the Anaconda Python distribution often used by the scientific community) +- `pipenv`, created to fix certain shortcomings of `virtualenv` - `poetry`, a modern Python packaging tool which handles virtual environments automatically -- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by OpenAI +- `uv`, an extremely fast Python package and project manager, written in Rust, and owned by Astral - `pdm`, a modern Python package and dependency manager supporting the latest PEP standards. +The first few of these tools manage *environments* only. +The later ones (`pipenv`, `poetry`, `uv`, `pdm`) are **project managers**: they look after the virtual environment, the packages installed into it, and the metadata describing your project, all through a single command line tool. While there are pros and cons for using each of the above, all will do the job of managing Python virtual environments for you and it may be a matter of personal preference which one you go for. -In this course, we will use `pdm` to create and manage our virtual environment. +In this course, we will use `pdm` to create and manage our virtual environment because is that it removes a whole class of common mistakes: +you don't have to remember which environment is active, +which `pip` belongs to which Python installation, +or which packages you installed by hand three months ago. -### Managing External Packages +As this comic points out, managing Python and its environments used to be quite complex without these tools. -Part of managing your (virtual) working environment involves -installing, updating and removing external packages on your system. -The Python package manager tool `pip` is most commonly used for this - -it interacts and obtains the packages from the central repository called -[Python Package Index (PyPI)](https://pypi.org/). -`pip` can now be used with all Python distributions (including Anaconda). +![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} ::::::::::::::::::::::::::::::::::::::::: callout ## A Note on Anaconda and `conda` -Anaconda is an open source Python distribution commonly used for scientific programming - -it conveniently installs Python, package and environment management `conda`, -and a number of commonly used scientific computing packages -so you do not have to obtain them separately. +Anaconda is an open source Python distribution commonly used for scientific programming - it conveniently installs Python, package and environment management through `conda`, and a number of commonly used scientific computing packages so you do not have to obtain them separately. +However, recent [licence changes](https://www.datacamp.com/blog/navigating-anaconda-licensing) have made Anaconda less appealing. +There are truly open alternatives like [conda-forge](https://conda-forge.org/). + `conda` is an independent command line tool -(available separately from the Anaconda distribution too) with dual functionality: -(1) it is a package manager that helps you find Python packages -from remote package repositories and install them on your system, and -(2) it is also a virtual environment manager. -So, you can use `conda` for both tasks instead of using `venv` and `pip`. +(available separately from the Anaconda distribution) with dual functionality: +1. It is a package manager that helps you find Python and non-Python packages from remote package repositories and install them on your system, and +2. It is also a virtual environment manager. + So, you can use `conda` for both tasks instead of using `venv` and `pip`. :::::::::::::::::::::::::::::::::::::::::::::::::: -### Many Tools for the Job - -Installing and managing Python distributions, -external libraries and virtual environments is, well, complex. -There is an abundance of tools for each task, -each with its advantages and disadvantages, -and there are different ways to achieve the same effect -(and even different ways to install the same tool!). -Note that each Python distribution comes with its own version of `pip` - -and if you have several Python versions installed you have to be extra careful to -use the correct `pip` to manage external packages for that Python version. - -`venv` and `pip` are considered the *de facto* standards for virtual environment -and package management for Python 3. -However, the advantages of using Anaconda and `conda` are that -you get (most of the) packages needed for scientific code development included with the distribution. -If you are only collaborating with others who are also using Anaconda, -you may find that `conda` satisfies all your needs. -It is good, however, to be aware of all these tools, and use them accordingly. -As you become more familiar with them you will realise that -equivalent tools work in a similar way even though the command syntax may be different -(and that there are equivalent tools for other programming languages too -to which your knowledge can be ported). +Let us have a look at how we can create and manage a virtual environment +and its packages from the command line using `pdm`. -![Python Environment Hell from [XKCD](https://xkcd.com/1987/) (Creative Commons Attribution-NonCommercial 2.5 License)](fig/python-environment-hell.png){alt='Python environment hell XKCD comic'} +:::::::::::::::::::::::::::::::::::::::::: prereq -Let us have a look at how we can create and manage virtual environments from the command line -using `venv` and manage packages using `pip`. +### Making Sure You Can Invoke PDM and Python -:::::::::::::::::::::::::::::::::::::::::: prereq +Install PDM according to its [website instructions](https://pdm-project.org/en/latest/#recommended-installation-method), which at the time of writing is a `curl` command: + +```bash +curl -sSL https://pdm-project.org/install.sh | bash +``` + +You can inspect the Bash script at the URL `curl` is grabbing from if you want to make sure it isn't doing anything sketchy. + +Then, test that PDM is available on your `PATH` by executing: + +```bash +pdm --version +``` + +```output +PDM, version 2.28.2 +``` -### Making Sure You Can Invoke Python +If this fails, revisit the [setup instructions](../learners/setup.md) for this course. -You can test your Python installation from the command line with: +PDM can manage Python interpreters for you, +but it is good to check that you also have a system Python available: ```bash -$ python3 --version # on Mac/Linux -$ python --version # on Windows — Windows installation comes with a python.exe file rather than a python3.exe file +python --version ``` If you are using Windows and invoking `python` command causes your Git Bash terminal to hang with no error message or output, you may @@ -210,73 +204,83 @@ need to create an alias for the python executable `python.exe`, as explained in :::::::::::::::::::::::::::::::::::::::::::::::::: -### Creating Virtual Environments Using `venv` +### Creating a Virtual Environment Using `pdm` -Creating a virtual environment with `venv` is done by executing the following command: +Our project already contains a `pyproject.toml` file, the standard file that describes a Python project, its metadata and its dependencies. +Because of this, we do not need to create the project from scratch; we can ask PDM to set everything up for us with a single command. + +First, ensure you are within the project root directory, then: ```bash -$ python3 -m venv /path/to/new/virtual/environment +pdm install ``` -where `/path/to/new/virtual/environment` is a path to a directory where you want to place it - -conventionally within your software project so they are co-located. -This will create the target directory for the virtual environment -(and any parent directories that don't exist already). +```output +WARNING: Lockfile does not exist +Updating the lock file... +WARNING: Project requires a python version of >=3.10, The virtualenv is being created for you as it cannot be matched to the right version. +INFO: python.use_venv is on, creating a virtualenv for this project... +Virtualenv is created successfully at +/home/user/python-intermediate-inflammation/.venv +Changes are written to pdm.lock. + 0:00:00 🔒 Lock successful. +All packages are synced to date, nothing to do. + ✔ Install python-intermediate-inflammation 0.0.0 successful + + 0:00:00 🎉 All complete! 0/0 +``` -::::::::::::::::::::::::::::::::::::::::: callout +Your output will look a little different depending on the Python version and paths on your machine. +That one command did several things for us: + +1. it created a virtual environment for the project in the `.venv` directory, +2. it created a **lock file** called `pdm.lock`, which records the exact version of every package the environment should contain, and +3. it installed our own project into that environment (more on this below). -## What is `-m` Flag in `python3` Command? +Our project does not declare any external dependencies yet, so there was not much for PDM to install... yet. -The Python `-m` flag means "module" and tells the Python interpreter to treat what follows `-m` -as the name of a module and not as a single, executable program with the same name. -Some modules (such as `venv` or `pip`) have main entry points -and the `-m` flag can be used to invoke them on the command line via the `python` command. -The main difference between running such modules as standalone programs -(e.g. executing "venv" by running the `venv` command directly) -versus using `python3 -m` command seems to be that -with latter you are in full control of which Python module will be invoked -(the one that came with your environment's Python interpreter vs. -some other version you may have on your system). -This makes it a more reliable way to set things up correctly -and avoid issues that could prove difficult to trace and debug. +::::::::::::::::::::::::::::::::::::::::: callout +## Choosing a Python Interpreter -:::::::::::::::::::::::::::::::::::::::::::::::::: +By default PDM will pick a suitable Python interpreter from the ones installed on your machine and remember the choice in a file called `.pdm-python`. +You can inspect what it selected with `pdm info`, change it with `pdm use`, and even have PDM download and install a Python version for you with, for example, `pdm python install 3.14`. +This is handy when a project needs a Python version that is not available through your operating system. -For our project let us create a virtual environment called "venv". -First, ensure you are within the project root directory, then: +:::::::::::::::::::::::::::::::::::::::::::::::::: -```bash -$ python3 -m venv venv -``` +### What PDM Actually Created -If you list the contents of the newly created directory "venv", on a Mac or Linux system +The virtual environment PDM built for us is an ordinary Python virtual environment of the kind that `venv` produces; there is nothing magic about it. +If you list the contents of the `.venv` directory, on a Mac or Linux system (slightly different on Windows as explained below) you should see something like: ```bash -$ ls -l venv +$ ls -l .venv ``` ```output -total 8 -drwxr-xr-x 12 alex staff 384 5 Oct 11:47 bin -drwxr-xr-x 2 alex staff 64 5 Oct 11:47 include -drwxr-xr-x 3 alex staff 96 5 Oct 11:47 lib --rw-r--r-- 1 alex staff 90 5 Oct 11:47 pyvenv.cfg +total 20 +drwxrwxr-x 2 user user 4096 Aug 19 18:08 bin/ +-rw-rw-r-- 1 user user 194 Aug 19 18:08 CACHEDIR.TAG +drwxrwxr-x 2 user user 4096 Aug 19 18:08 include/ +drwxrwxr-x 3 user user 4096 Aug 19 18:08 lib/ +-rw-rw-r-- 1 user user 750 Aug 19 18:08 pyvenv.cfg ``` -So, running the `python3 -m venv venv` command created the target directory called "venv" -containing: +So, a virtual environment is a directory containing: -- `pyvenv.cfg` configuration file - with a home key pointing to the Python installation from which the command was run, -- `bin` subdirectory (called `Scripts` on Windows) - containing a symlink of the Python interpreter binary used to create the environment - and the standard Python library, -- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) - to contain its own independent set of installed Python packages isolated from other projects, and +- `pyvenv.cfg` configuration file with a home key pointing to the Python installation it was created from, +- `bin` subdirectory (called `Scripts` on Windows) containing a symlink of the Python interpreter binary used to create the environment and the standard Python library, +- `lib/pythonX.Y/site-packages` subdirectory (called `Lib\site-packages` on Windows) to contain its own independent set of installed Python packages isolated from other projects, and - various other configuration and supporting files and subdirectories. +Had we not been using PDM, we could have created exactly this ourselves with `python -m venv .venv` and then installed packages into it by using `pip`. +It is worth knowing this is all a virtual environment is, so that the environment does not feel like a black box. +But from here on, we will let PDM do the work of managing this environment. + +TODO current position + ::::::::::::::::::::::::::::::::::::::::: callout ## Naming Virtual Environments diff --git a/learners/setup.md b/learners/setup.md index b76d1a5bd..1346bf501 100644 --- a/learners/setup.md +++ b/learners/setup.md @@ -8,6 +8,7 @@ You will need the following software and accounts setup to be able to follow the - Command line tool (such as Bash, Zsh or Git Bash) - Git version control program +- The `curl` command line utility - GitHub account - Python 3 distribution - Integrated development environment (IDE) - PyCharm or Visual Studio Code (VS Code) From cc3ff6eb6c8904be7e84f3e9b14d5c2d9ede4b1b Mon Sep 17 00:00:00 2001 From: bielsnohr <6177028+bielsnohr@users.noreply.github.com> Date: Thu, 20 Aug 2026 16:17:35 +0100 Subject: [PATCH 3/3] Amend up through new section "Running Command in Your Environment" --- episodes/12-virtual-environments.md | 104 ++++++++++------------------ 1 file changed, 38 insertions(+), 66 deletions(-) diff --git a/episodes/12-virtual-environments.md b/episodes/12-virtual-environments.md index e8f0140f3..8b0ae71f8 100644 --- a/episodes/12-virtual-environments.md +++ b/episodes/12-virtual-environments.md @@ -279,110 +279,82 @@ Had we not been using PDM, we could have created exactly this ourselves with `py It is worth knowing this is all a virtual environment is, so that the environment does not feel like a black box. But from here on, we will let PDM do the work of managing this environment. -TODO current position +Note that since our software project is being tracked by Git, the newly created `.venv` directory will show up in version control. +We will see how to tell Git to ignore it in one of the subsequent episodes. ::::::::::::::::::::::::::::::::::::::::: callout -## Naming Virtual Environments - -What is a good name to use for a virtual environment? -Using "venv" or ".venv" as the name for an environment -and storing it within the project's directory seems to be the recommended way - -this way when you come across such a subdirectory within a software project, -by convention you know it contains its virtual environment details. -A slight downside is that all different virtual environments on your machine -then use the same name -and the current one is determined by the context of the path you are currently located in. -A (non-conventional) alternative is -to use your project name for the name of the virtual environment, -with the downside that there is nothing to indicate that such a directory contains a virtual environment. -In our case, we have settled to use the name "venv" instead of ".venv" -since it is not a hidden directory and we want it to be displayed by the command line -when listing directory contents -(the "." in its name that would, by convention, make it hidden). -In the future, you will decide what naming convention works best for you. -Here are some references for each of the naming conventions: +## Naming and Locating Virtual Environments + +Storing the environment inside the project directory and calling it "venv" or ".venv" is the usual convention. +This way when you come across such a subdirectory within a software project, you know it contains its virtual environment details. +PDM uses `.venv` in the project root by default. +You can ask PDM to create additional, named environments elsewhere (e.g. `pdm venv create --name py312 3.12`) and list them with `pdm venv list`, which is useful when you want to test your code against several Python versions. +Here are some references for the naming conventions: - [The Hitchhiker's Guide to Python](https://docs.python-guide.org/dev/virtualenvs/) notes that "venv" is the general convention used globally - [The Python Documentation](https://docs.python.org/3/library/venv.html) indicates that ".venv" is common - ["venv" vs ".venv" discussion](https://discuss.python.org/t/trying-to-come-up-with-a-default-directory-name-for-virtual-environments/3750) - :::::::::::::::::::::::::::::::::::::::::::::::::: -Once you've created a virtual environment, you will need to activate it. +### Running Commands In Your Environment -On Mac or Linux, it is done as: +With PDM you do not normally activate the virtual environment at all. +Instead, you prefix commands with `pdm run` and PDM makes sure they are executed using the project's environment: ```bash -$ source venv/bin/activate -(venv) $ +pdm run python --version ``` -On Windows, recall that we have `Scripts` directory instead of `bin` -and activating a virtual environment is done as: - -```bash -$ source venv/Scripts/activate -(venv) $ +```output +Python 3.14.2 # your version will differ depending on your system ``` -Activating the virtual environment will change your command line's prompt -to show what virtual environment you are currently using -(indicated by its name in round brackets at the start of the prompt), -and modify the environment so that running Python will get you -the particular version of Python configured in your virtual environment. +This works from anywhere inside the project, and it works the same way for you and for your collaborators, regardless of what is or is not activated in their shell. -You can verify you are using your virtual environment's version of Python -by checking the path using the command `which`: +If you would still like an activated shell---for example because you are running many commands in a row---PDM will print the activation command for you to evaluate: ```bash -(venv) $ which python3 +echo $(pdm venv activate) ``` ```output -/home/alex/python-intermediate-inflammation/venv/bin/python3 +# The output will vary depending on your OS and current shell, which is precisely why this command is useful +source /home/user/python-intermediate-inflammation/.venv/bin/activate ``` -When you're done working on your project, you can exit the environment with: +Activating the virtual environment will change your command line's prompt to show what virtual environment you are currently using (indicated by its name in round brackets at the start of the prompt): ```bash -(venv) $ deactivate +eval $(pdm venv activate) ``` -If you have just done the `deactivate`, -ensure you reactivate the environment ready for the next part: - -```bash -$ source venv/bin/activate -(venv) $ +```output +# Your prompt should look something like this: +(python-intermediate-inflammation-3.14) $ ``` -::::::::::::::::::::::::::::::::::::::::: callout - -## Python Within A Virtual Environment +Materially, this modifies the environment so that running `python` will get you the particular version of Python configured in your virtual environment. +You can verify this by checking the path with the command `which`: -Within an active virtual environment, -commands `python3` and `python` should both refer to the version of Python 3 -you created the environment with (note you may have multiple Python 3 versions installed). - -However, on some machines with Python 2 installed, -`python` command may still be hardwired to the copy of Python 2 -installed outside of the virtual environment - this can cause errors and confusion. +```bash +which python +``` -You can always check which version of Python you are using in your virtual environment -with the command `which python` to be absolutely sure. -We continue using `python3` in this material to avoid mistakes, -but the command `python` may work for you as expected. +```output +/home/user/python-intermediate-inflammation/.venv/bin/python +``` +When you're done working on your project, you can exit the environment with: -:::::::::::::::::::::::::::::::::::::::::::::::::: +```bash +deactivate +``` -Note that, since our software project is being tracked by Git, -the newly created virtual environment will show up in version control - -we will see how to handle it using Git in one of the subsequent episodes. +For the rest of this course we will write commands using `pdm run`, so you do not need to keep an environment activated. ### Installing External Packages Using `pip`