GROMACS

Overview

The Gromacs site describes itself as

GROMACS is a versatile package to perform molecular dynamics, i.e. simulate the Newtonian equations of motion for systems with hundreds to millions of particles.

—About GROMACS

This package is made available to all cluster users of both environments by leveraging a Apptainer container that contains both the GROMACS package as well as all needed libraries to run it. This allows us to easily update the package when requested and for older versions to exist if a researcher has a need for an older version.

Because of the way that the GROMACS container is deployed in our environment, we will leverage a small wrapper script that will help start the container and call the real GROMACS (gmx) binary as well as stage up your data in the appropriate directories.

Example Files

Download the Needed Files

Download all as .tar.xz file

• gromacs_example.tar.xz

  Once you have downloaded the file, you can copy files to the cluster using SCP and then extract it with the following command. It will create a new gromacs_example directory in the directory where you extract it.

  tar -xJf gromacs_example.tar.xz
  

Download all of the following files and place them in the same directory in the cluster environment. If you downloaded them to your local system instead of directly to the cluster, you can use SCP to copy them to the cluster. If you didn’t download and extract the gromacs_example.tar.xz file in the sidebar, you will need to ensure that gmx.sh and OMP+MPIscript.parallel files are executable, which you can do by running chmod +x gmx.sh OMP+MPIscript.parallel on the cluster head node where you’ve put the files. The mdrun.htsub submission file will also create it’s log files in a logs/ directory. This again will be created for you if you download and extract the gromacs_example.tar.xz file. You can create your own logs directory by running mkdir logs on the cluster head node where you’ve put your files, or modify the mdrun.htsub and remove the logs/ prefix in the three lines where logging is configured.

• gmx.sh

• OMP+MPIscript.parallel

• mdrun.htsub

Interacting with GROMACS via HTCondor

Many of the GROMACS commands expect the user to type input to answer a selection prompt. It’s possible to script this out and provide the input needed via HTCondor, but unless you know the exact number of your inputs ahead of time, this becomes difficult to get right. The easy workaround is to have HTCondor allocate resources on a compute node and provide you an interactive shell there so you can run commands as if you were sitting down at the compute node instead of the head node.

Getting an Interactive Prompt on a Compute Node

HTCondor’s condor_submit makes getting an interactive shell prompt easy. Simply add the -interactive flag when you run the command. Be sure to request some resources for yourself, since it’s default may be too low for your needs.

condor_submit -interactive request_cpus=4 request_memory=4GB

The command will inform you that it’s submitted the interactive job to the queue and then waits to be run on a compute node. You’ll see the warning about limitations of interactive jobs, and finally be presented by a new prompt informing you of which compute node your utilizing.

USER@cse-head:~/gromacs_example$ condor_submit -interactive request_cpus=4 request_memory=4GB
Submitting job(s).
1 job(s) submitted to cluster 1182.
Welcome to slot1_1@c-X-X.cluster.cs.wwu.edu!
You will be logged out after 3600 seconds of inactivity.


**********
 WARNING!
**********
This HTCondor session will end after 24 hours, or 1 hour of idle time.
To run for more than 24 hours, submit the job without "-interactive".


USER@c-X-X:~/gromacs_example$

Once you have a prompt at the compute node, you can run the gmx.sh script to access all of the GROMACS commands, including mdrun.

Tip

Be careful! The mdrun command can run for a long time and need a lot of resources to complete. Because of this, it has it’s own job submission file that you should use to ensure that it has the proper runtime and resources allocated to it. Unless you know your run will complete quickly, it is suggested you use the mdrun.htsub provided here and submit the mdrun job separately from your interactive session.

Example use case (after getting an interactive shell on a compute node):

USER@c-X-X:~/gromacs_example$ ./gmx.sh pdb2gmx -f 1ubq.pdb -o 1ubq.gro -p 1ubq.top -ignh
                   :-) GROMACS - gmx pdb2gmx, 2022 (-:

Executable:   /gromacs/bin.AVX_256/gmx_mpi
Data prefix:  /gromacs
Working dir:  /cluster/home/USER/gromacs_example
Command line:
  gmx_mpi pdb2gmx -f 1ubq.pdb -o 1ubq.gro -p 1ubq.top -ignh

Select the Force Field:

From '/gromacs/share/gromacs/top':

 1: AMBER03 protein, nucleic AMBER94 (Duan et al., J. Comp. Chem. 24, 1999-2012, 2003)

...

You’ll notice that the printed “Command line” is calling a different program. This is because the gmx.sh is setting up the Apptainer environment for GROMACS and running the command inside of it. Don’t worry about this. The script inside the Apptainer environment will also determine the fastest available executable to run for your compute node, so you may see a different response there as well.

Note

There is a leading ./ on the gmx.sh to indicate that the command you want to run should be the file gmx.sh in your current working directory. gmx.sh is not in your $PATH so typing gmx.sh without the ./ will print gmx.sh: command not found.

Once you’ve prepared all of your files for the mdrun command, you can submit the mdrun job to be run for it’s larger, longer calculations.

Submitting the mdrun job

While possible to run the mdrun command in an interactive session, it may run for longer than the 24 hours allowed, or need multiple computer’s resources working together in parallel to get it’s results. Because of this, we submit a separate job to the queue to take advantage of HTCondor’s parallel universe, which will allocate multiple computers to one job.

Note

You will need to modify the mdrun.htsub file to adjust the filename(s) and any options needed to be passed to the mdrun command before submitting the job. You may also want to adjust the amount of resources being requested to match your needs. See the explanations of each line in the file in mdrun.htsub below for information about which lines do what.

Tip

Editing files in a terminal can be daunting, but nano(1) makes it easy! The bottom two lines print what key commands are available. The ^ symbol represents the Ctrl key. So Ctrl + O will save a file, and Ctrl + X will exit.

After modifying the mdrun.htsub for your needs, you can submit it using the condor_submit command.

condor_submit mdrun.htsub

Tip

You can add the option -batch-name with the argument “$(basename $(pwd))” to set your current directory’s name as your job’s batch name. This is really useful when you want to see at a glance the status of multiple jobs. It’s not required, but is a nice to have!

condor_submit -batch-name "$(basename $(pwd))" mdrun.htsub

Once the mdrun job has been submitted, you can check it’s status in the queue with condor_q or condor_watch_q. Check the FAQ for explanations. After the job is in a running state, the output of the main node that is running the job will be in two files, logs/mdrun.0.err and logs/mdrun.0.out. The 0 indicates it’s the first node (The node index begins counting at 0), and the .err and .out indicate if the output came from STDERR or STDOUT. GROMACS tends to put most of it’s messages on STDERR, so you can follow along as your job runs by using the tail command’s -f option.

tail -f logs/mdrun.0.err

This will continuously watch the file for new data to be written to it and display it on the screen. When you want to stop watching the file, press Ctrl+C to exit.

Example Files Explained

What follows is an explanation of each of the supplied files, and how they play a role in running GROMACS in our environment.

gmx.sh

Think of the gmx.sh file as a replacement for anywhere you would have run the gmx command. Any flags or commands you would have given to it, you should pass to this script and it will pass them on to the real command. You will most likely not need to modify this file for your work.

GROMACS wrapper script to startup Apptainer container and run the gmx command – gmx.sh

#!/bin/sh

exec apptainer exec -B "$(pwd)":/data /cluster/share/singularity/gromacs-2022.2.sif gmx ${@}

That second line has a lot to unpack here, but the first line is simple. It tells the system that we’re a shell script and should be run with the sh shell. If you’re curious about the internals here, keep reading. Otherwise feel free to skip onto the next file.

(Optional) Technical breakdown of the second line

• exec

  Tell’s the shell to replace itself with the following command

• apptainer

  The real command we want to run, apptainer. The full path to the binary is prefixed here in case the user doesn’t have this in their path because they’ve overridden it. Everything that follows this is “sub-commands” to the Apptainer command.

  • exec

    Tells Apptainer to execute the command it’s given on the command line, not the command it may have been pre-configured to run.

    • -B “$(pwd)”:/data

      Tells Apptainer to “bind” the current directory to /data inside the container, which is where GROMACS intends to operate on its data inside the contained environment.

    • /cluster/share/singularity/gromacs-2022.0.sif

      Tells Apptainer the path the GROMACS Apptainer Image File (sif) that holds the container we want to run. It is made available on all nodes so you don’t have to worry about copying it over in your submission file.

    • gmx

      Tells Apptainer which command want to run inside the container.

    • ${@}

      Finally any arguments that were given to the wrapper script are passed to the above gmx command. You may be tempted to modify this to include your arguments, but you will break the wrapper for everything else.

OMP+MPIscript.parallel

The OMP+MPIscript.parallel script is a customized form of the openmpiscript that HTCondor provides as an example.

Due to the length of the file, it is not inlined here, but the high level overview is a follows:

This shell script does the heavy lifting of setting and tearing down the support nodes when you run an MPI (Message Passing Interface) based job. It can be broken down into what it does into three cases.

1. On all nodes it sets up the environment variables that configure the library paths and executable search paths needed to find a specified version of OpenMPI. It also sets any relevant options, such as enabling OpenMP (Open Multi-Processing) for leveraging multiple processes on each node that we run on.

2. On all nodes except the first node, it starts the orted (Open Run-Time Environment Daemon) process to wait and accept work from the first node.

3. On the first node it starts the main mpirun process with the specified arguments. This part is crucial for us, as its arguments will be the gmx.sh script and all of its commands, flags, etc. This shows the layered approach we take to have a component setup the next component for success.

mdrun.htsub

From the mdrun manual page:

gmx mdrun is the main computational chemistry engine within GROMACS. Obviously, it performs Molecular Dynamics simulations, but it can also perform Stochastic Dynamics, Energy Minimization, test particle insertion or (re)calculation of energies. Normal mode analysis is another option. In this case mdrun builds a Hessian matrix from single conformation. For usual Normal Modes-like calculations, make sure that the structure provided is properly energy-minimized. The generated matrix can be diagonalized by gmx nmeig.

—gmx mdrun manual page

Because the mdrun command tends to need more resources and cores than the other shorter running commands, it needs it’s own submission file to leverage HTCondor’s parallel universe in order to split it’s work on multiple compute nodes at the same time.

HTCondor submission script to run the mdrun command of gmx– mdrun.htsub

# Sample
filename=nvt

# Basic job attributes
Executable      = OMP+MPIscript.parallel
Arguments       = gmx.sh mdrun -v -s $(filename).tpr -o $(filename).trr -x $(filename).xtc -cpo $(filename).cpt -c $(filename).gro -e $(filename).edr -g $(filename).log

# Need to call out not vanilla universe becase we want to mix hosts
Universe        = parallel

Output          = logs/mdrun.$(filename).$(Node).o
Error           = logs/mdrun.$(filename).$(Node).e
Log             = logs/mdrun.$(filename).log

# How much memory each process will need --
# please adjust this to reflect your needs.
request_memory  = 1GB

# 4 compute nodes, 3 cpus on each node, 1GB on each node
# Adjust as you need more or less resources
request_cpus    = 3
machine_count   = 4

queue

Lines 1-2:

This defines a variable (only used by the HTCondor submission script) of what the name of the file to use is, instead of writing it multiple times on the arguments line. It also gets reused to name each log file with the associated filename.

Lines 4-6:

Our executable we want to run the wrapper script for setting up the OpenMPI environment. The gmx.sh that would be gmx command becomes the first argument, followed by its usual mdrun argument and associated flags. The previously defined filename variable is used here multiple times to allow for this submission file to be quickly reused in the future. Line 6 is the line where you will adjust your arguments to the mdrun command to specify any needed parameters.

Lines 8-9:

We specify that we want to use HTCondor’s “parallel” universe, because we want the ability to run on multiple nodes instead of one single node.

Lines 11-13:

Logs are written to the logs/ directory to reduce clutter in our directory, and the files are aptly named after the job, with the output, error, and HTCondor log being saved as separate files. Most of the output of the mdrun process will be in the logs/mdrun.$(filename).0.e or logs/mdrun.$(filename).0.o file.

Lines 15-17:

Takes a guess at how much memory you will need. This will be the amount used per machine, and is not not scaled by the cpu count. Please be considerate of others and only request what you need.

Lines 19-22:

Asks for the number of cpus per requested machine. If you request three (3) cpus, and four (4) machines, you will consume a total of 12 cpus for this job. GROMACS can run in a hybrid OMP+OpenMPI mode, where OpenMPI distributes the jobs to the requested machines, and OpenMP scales across the requested cpus for each machine.

Line 24:

Finally we enqueue one instance of the job we just defined above.

Additional notes:

The total of 12 cpus was an arbitrary example. Using fewer machines, and a higher number of cpus can decrease runtime, but may take longer to find available nodes depending on how busy the cluster is. If you request 24 cpus and 3 machines, you are essentially asking for three entire machines to yourself, which may be very hard to come by when many researchers are running smaller jobs. Mixing and matching smaller counts may yield reasonable wait times in the queue along with decent performance.

Currently Available Versions

There is currently only one version available, 2022.0. If updated or additional versions are needed please reach out to .

Additional References

1. GROMACS documentation

2. HTCondor User’s Manual

3. WWU’s McCarty Group’s Wiki Example