Creating a Custom Python Environment (ipykernel)

Open a terminal tab in your JupyterLab, and create your custom environment in Conda or Mamba. Be mindful of where you put the environment, as your default home directory may not have enough space to accommodate all the packages you want to install. The following is a suggested example, which is similar to that of creating a custom Conda environment on the Python examples page.

Note

The examples below will all use Mamba instead of Conda. These tools are largely compatible, and even Conda has switched to using libmamba-solver to do package dependency resolution when installing. Miniforge has both of them installed, and they both default to using the conda-forge channel.

Note

The example below uses the variable $PI, which defines the Primary Investigator (your research advisor). You will either want to set this variable to your PI’s surname, or type it in manually if you don’t want to set it. Below is an example of setting it in the default Bash shell:

Setting the $PI variable used in the examples below.

<default.sif>[USER@c-X-X:~/demo]$ export PI=researcherlastname
<default.sif>[USER@c-X-X:~/demo]$ echo $PI
researcherlastname

The $USER variable will automatically expand to your username.

The first step is to create your new Mamba environment. It’s suggested to create your environment in your research space, but your PI may tell you exactly where to put it instead.

mamba create -p /cluster/research-groups/$PI/workspace/$USER/miniforge3/envs/mycoolenvname

Using the -p option to create allows you to specify the path where you want it to create the environment, instead of the default location. You will not have access to create a new environment in the default location, so we need to specify where to put the new environment. The final directory name will be the environment name, in this example it’s “mycoolenvname”. This name will be displayed by default on the launcher icon in JupyterLab, but you can also override it at a later step.

With the environment created, we need to activate it. Unfortunately because the environment was put in a non-standard location, we need to tell the activate command where to look for it.

mamba activate /cluster/research-groups/$PI/workspace/$USER/miniforge3/envs/mycoolenvname

The prompt will change to reflect that the environment has been activated. The name should appear inside of () to the left of your normal prompt. Once it’s activated, the final step is running a small script that will take the necessary information from your environment and convert it to a kernel that Jupyter understands. This process has been greatly simplified for our users by the addition of the conda2kern and mamba2kern scripts included in the default container. This script will ensure that the appropriate packages are installed, call the appropriate module or library to create the kernel file, then modify the kernel file to set additional environment variables so Mamba can be correctly interacted with. Without the last step, users would not be able to !mamba install pkg_name from their notebooks.

There are a few options that can be passed to the mamba2kern script to help guide it if it can not auto-determine which language your environment is using. To get the full list of options, pass it the -h option.

<default.sif>[USER@c-X-X ~/demo]$ mamba2kern -h
Usage: /usr/bin/mamba2kern [-d <display name>] [-n <kernel name>] [-h]
-h:                Display this help message
-d <display name>: The display name
-n <kernel name>:  The kernel name
-p:                Force Python environment
-r:                Force R environment

As shown in the session above, you can specify various override options to help the script if it can not determine what you’re trying to do. By default the option:-d will be set to the name of the Mamba environment, as will -n. If you would prefer a more descriptive name, you can adjust the display name to something that suits your needs. One use case for this is if you want to remind yourself that you’re using Python version 3.8 explicitly for an environment, instead of the usual version of 3.11. You could use -d 'My Cool Env (Py 3.8)' to set “My Cool Env (Py 3.8)” as your display name.

However, with a newly created and empty Mamba environment, the script is unable to determine if this should be a Python environment or an R environment. So you will want to pass it the appropriate -p or -r option to specify which it should be. If you have already installed the ipykernel package or the r-irkernel package, the script should be able to guess which one you want. For this example, we can force it to a Python environment and let it do the work for us:

(mycoolenvname) <default.sif>[USER@c-1-0 ~]$ mamba2kern -p

Looking for: ['ipykernel']

conda-forge/noarch                                  18.9MB @  11.4MB/s  2.1s
conda-forge/linux-64                                41.6MB @  11.0MB/s  4.3s
Transaction

  Prefix: /cluster/research-groups/mcgrew/workspace/USER/miniforge3/envs/mycoolenvname

  Updating specs:

   - ipykernel


  Package                    Version  Build               Channel           Size
──────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────

  + python_abi                  3.13  5_cp313             conda-forge        6kB
  ... CUT FOR SPACE ...
  + ipykernel                 6.29.5  pyh3099207_0        conda-forge      119kB

  Summary:

  Install: 63 packages

  Total download: 45MB

──────────────────────────────────────────────────────────────────────────────────


python_abi                                           6.2kB @ 114.0kB/s  0.1s
... CUT FOR SPACE ...
python                                              33.2MB @  40.8MB/s  0.7s

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Installed kernelspec mycoolenvname in /cluster/home/USER/.local/share/jupyter/kernels/mycoolenvname
Inserting custom env section...
(mycoolenvname) <default.sif>[USER@c-X-X ~]$

And with that, if you open a new launcher tab (click the +) you should now see a new launcher icon under both Notebook and Console, which will either let you run a notebook using your newly created environment, or launch a Python console session with it activated.

With it activated, you can run interactive commands to install new packages, such as !mamba install -y pyfiglet and then run it in the next cell with !pyfiglet 'Jupyter!'.

     _                   _            _
    | |_   _ _ __  _   _| |_ ___ _ __| |
 _  | | | | | '_ \| | | | __/ _ \ '__| |
| |_| | |_| | |_) | |_| | ||  __/ |  |_|
 \___/ \__,_| .__/ \__, |\__\___|_|  (_)
            |_|    |___/

Enjoy your new custom environment in Jupyter!