Multi Node

In this section we are going to use MPICH MPI to create images for x86 (C5) and G4dn instances to run distributed simulations.

Connect to Compute

Rather than compiling GROMACS on the head-node let us allocate a compute node to compile. In case the terminal is still busy compiling something else, please open another terminal.

salloc -N1 -p c5n
ssh $SLURM_NODELIST

Spack Container Dev

We are using a branch of Spack which introduces a more flexible way of using spack containerize.

source /fsx/spack-container/share/spack/setup-env.sh

Some environment vars.

  • TAG: Amazonlinux tag of the docker base image
  • ARCH: Archtiecture according to spack. E.g. linux-amzn2-skylake_avx512
TAG=2
ARCH=$(spack arch)

ECR Login

First we make sure we are logged into ECR.

export AWS_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)"
eval $(aws ecr get-login --no-include-email --region "${AWS_REGION}")

GROMACS 2021.1

This image is using MPICH MPI to be used with the installed Intel MPI.

TECH=mpich
mkdir -p ~/spack/gromacs/2021.1/${ARCH}/${TECH}/
cd ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cat >> spack.yaml << EOF
spack:
  view: true
  specs:
  - gromacs@2021.1 ^mpich
  container:
    os_packages:
      final:
      - libgomp
    extra_instructions:
      final: "RUN (echo /opt/view/lib && echo /opt/view/lib64) > /etc/ld.so.conf.d/spack-view.conf && ldconfig"
    images:
      os: amazonlinux:2
      spack:
        ref: develop
    labels:
      app: "gromacs"
      mpi: "mpich"
EOF

We are going to create a Dockerfile out of the spack spec above.

spack containerize > Dockerfile 

Create Repo

Let’s create an ECR repo specific to this GROMACS image

aws ecr create-repository --repository-name=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"
export GROMACS_BASE=$(aws ecr describe-repositories --repository-names=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"| jq -r '.repositories[0].repositoryUri')
echo "# GROMACS_BASE=${GROMACS_BASE}"

Now, we can build and push the image.

docker build -t ${GROMACS_BASE}:${TAG} .
docker push ${GROMACS_BASE}:${TAG}
Import into Sarus

To make this image available, we are going to export the image locally and import it into sarus.

mkdir -p /fsx/images/gromacs/2021.1/${ARCH}
docker save ${GROMACS_BASE}:${TAG} -o /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar
sarus load /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar gromacs/2021.1/${ARCH}:${TECH}

CUDA

For instances with NVIDIA GPUs we are going to build GROMACS with CUDA (and MPICH MPI).

TECH=cuda-mpich
mkdir -p ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cd ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cat >> spack.yaml << EOF
spack:
  view: true
  specs:
  - gromacs@2021.1 +cuda ^cuda@11.4.0 ^mpich
  container:
    os_packages:
      final:
      - libgomp
    extra_instructions:
      final: "RUN (echo /opt/view/lib && echo /opt/view/lib64) > /etc/ld.so.conf.d/spack-view.conf && ldconfig"
    images:
      os: amazonlinux:2
      spack:
        ref: develop
    labels:
      app: "gromacs"
      mpi: "mpich"
EOF

Let’s create an ECR repo specific to this GROMACS image

aws ecr create-repository --repository-name=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"
export GROMACS_BASE=$(aws ecr describe-repositories --repository-names=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"| jq -r '.repositories[0].repositoryUri')
echo "# GROMACS_BASE=${GROMACS_BASE}"

We remove the epel-release install as it is not available this way with Amazon Linux 2.

spack containerize \
  | sed -e '/RUN find -L /,+4d' > Dockerfile 

Now, we can build and push the image.

docker build -t ${GROMACS_BASE}:${TAG} .
docker push ${GROMACS_BASE}:${TAG}
Import into Sarus

To make this image available, we are going to export the image locally and import it into sarus.

mkdir -p /fsx/images/gromacs/2021.1/${ARCH}
docker save ${GROMACS_BASE}:${TAG} -o /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar
sarus load /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar gromacs/2021.1/${ARCH}:${TECH}