Creating a Custom Python Environment (ipykernel)

Open a terminal tab in your JupyterLab, and create your custom environment in Conda. Be mindful of where you put the environment, as your default home directory may not have enough space to accommodate all the packages you want to install. The following is a suggested example, which is similar to that of creating a custom Conda environment on the Python examples page.

Note

The example below uses the variable $PI, which defines the Primary Investigator (your research advisor). You will either want to set this variable to your PI’s surname, or type it in manually if you don’t want to set it. Below is an example of setting it in the default Bash shell:

Setting the $PI variable used in the examples below.

<default.sif>[USER@c-X-X:~/demo]$ export PI=researcherlastname
<default.sif>[USER@c-X-X:~/demo]$ echo $PI
researcherlastname

The $USER variable will automatically expand to your username.

The first step is to create your new Conda environment. It’s suggested to create your environment in your research space, but your PI may tell you exactly where to put it instead.

conda create -p /cluster/research-groups/$PI/workspace/$USER/miniconda3/envs/mycoolenvname

Using the -p option to create allows you to specify the path where you want it to create the environment, instead of the default location. You will not have access to create a new environment in the default location, so we need to specify where to put the new environment. The final directory name will be the environment name, in this example it’s “mycoolenvname”. This name will be displayed by default on the launcher icon in JupyterLab, but you can also override at a later step.

With the environment created, we need to activate it. Unfortunately because the environment was put in a non-standard location, we need to tell the activate command where to look for it.

conda activate /cluster/research-groups/$PI/workspace/$USER/miniconda3/envs/mycoolenvname

The prompt will change to reflect that the environment has been activated. The name should appear inside of () to the left of your normal prompt. Once it’s activated, the final step is running a small script that will take the necessary information from your environment and convert it to a kernel that Jupyter understands. This process has been greatly simplified for our users by the addition of the conda2kern script included in the default container. This script will ensure that the appropriate packages are installed, call the appropriate module or library to create the kernel file, then modify the kernel file to set additional environment variables so Conda can be correctly interacted with. Without the last step, users would not be able to !conda install pkg_name from their notebooks.

There are a few options that can be passed to the conda2kern script to help guide it if it can not auto-determine which language your environment is using. To get the full list of options, pass it the -h option.

<default.sif>[USER@c-X-X ~/demo]$ conda2kern -h
Usage: /usr/bin/conda2kern [-d <display name>] [-n <kernel name>] [-h]
-h:                Display this help message
-d <display name>: The display name
-n <kernel name>:  The kernel name
-p:                Force Python environment
-r:                Force R environment

As shown in the session above, you can specify various override options to help the script if it can not determine what you’re trying to do. By default the option:-d will be set to the name of the Conda environment, as will -n. If you would prefer a more descriptive name, you can adjust the display name to something that suits your needs. One use case for this is if you want to remind yourself that you’re using Python version 3.8 explicitly for an environment, instead of the usual version of 3.11. You could use -d 'My Cool Env (Py 3.8)' to set “My Cool Env (Py 3.8)” as your display name.

However, with a newly created and empty Conda environment, the script is unable to determine if this should be a Python environment or an R environment. So you will want to pass it the appropriate -p or -r option to specify which it should be. If you have already installed the ipykernel package or the r-irkernel package, the script should be able to guess which one you want. For this example, we can force it to a Python environment and let it do the work for us:

(mycoolenvname) <default.sif>[USER@c-X-X ~]$ conda2kern -p
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /cluster/research-groups/researcherlastname/workspace/USER/miniconda3/envs/mycoolenvname

  added / updated specs:
    - ipykernel


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    libsqlite-3.45.1           |       h2797004_0         839 KB  conda-forge
    python-3.12.2              |hab00c5b_0_cpython        30.8 MB  conda-forge
    ------------------------------------------------------------
                                           Total:        31.6 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  ... CUT FOR SPACE ...
  zipp               conda-forge/noarch::zipp-3.17.0-pyhd8ed1ab_0



Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Installed kernelspec mycoolenvname in /cluster/home/USER/.local/share/jupyter/kernels/mycoolenvname
Inserting custom env section...
(mycoolenvname) <default.sif>[USER@c-X-X ~]$

And with that, if you open a new launcher tab (click the +) you should now see a new launcher icon under both Notebook and Console, which will either let you run a notebook using your newly created environment, or launch a Python console session with it activated.

With it activated, you can run interactive commands to install new packages, such as !conda install -y -c conda-forge pyfiglet and then run it in the next cell with !pyfiglet 'Jupyter!'.

     _                   _            _
    | |_   _ _ __  _   _| |_ ___ _ __| |
 _  | | | | | '_ \| | | | __/ _ \ '__| |
| |_| | |_| | |_) | |_| | ||  __/ |  |_|
 \___/ \__,_| .__/ \__, |\__\___|_|  (_)
            |_|    |___/

Enjoy your new custom environment in Jupyter!