Single Node

Our first image is going to be using Thread MPI, which uses processes instead of ranks as it is not interacting with other hosts.

Connect to Compute

Rather than compiling GROMACS on the head-node let us allocate a compute node to compile.

salloc -N1 -p c5n
ssh $SLURM_NODELIST

Spack Container Dev

source /fsx/spack-container/share/spack/setup-env.sh
TAG=2
ARCH=$(spack arch)

ECR Login

First we log into ECR.

export AWS_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)"
eval $(aws ecr get-login --no-include-email --region "${AWS_REGION}")

GROMACS 2021.1

C5

This Spack Specification installs GROMACS with Thread MPI (~mpi in Spack excludes the use of MPI) to run on C5 instances.

TECH=tmpi
mkdir -p ~/spack/gromacs/2021.1/${ARCH}/${TECH}/
cd ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cat >> spack.yaml << EOF
spack:
  view: true
  specs:
  - gromacs@2021.1 ~mpi
  container:
    os_packages:
      final:
      - libgomp
    extra_instructions:
      final: "RUN (echo /opt/view/lib && echo /opt/view/lib64) > /etc/ld.so.conf.d/spack-view.conf && ldconfig"
    images:
      os: amazonlinux:2
      spack:
        ref: develop
    labels:
      app: "gromacs"
      mpi: "tmpi"
EOF

We are going to create a Dockerfile out of the spack spec above.

spack containerize > Dockerfile 
Build and Push

Let’s create an ECR repo specific to this GROMACS image

aws ecr create-repository --repository-name=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"
export GROMACS_BASE=$(aws ecr describe-repositories --repository-names=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"| jq -r '.repositories[0].repositoryUri')
echo "# GROMACS_BASE=${GROMACS_BASE}"

Now, we can build and push the image.

docker build -t ${GROMACS_BASE}:${TAG} .
docker push ${GROMACS_BASE}:${TAG}
Import into Sarus

To make this image available, we are going to export the image locally and import it into sarus.

mkdir -p /fsx/images/gromacs/2021.1/${ARCH}
docker save ${GROMACS_BASE}:${TAG} -o /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar
sarus load /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar gromacs/2021.1/${ARCH}:${TECH}

CUDA

For instances with NVIDIA GPUs we are going to build GROMACS with CUDA and Thread MPI

TECH=cuda-tmpi
mkdir -p ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cd ~/spack/gromacs/2021.1/${ARCH}/${TECH}
cat >> spack.yaml << EOF
spack:
  view: true
  specs:
  - gromacs@2021.1 +cuda ~mpi
  container:
    os_packages:
      final:
      - libgomp
    extra_instructions:
      final: "RUN (echo /opt/view/lib && echo /opt/view/lib64) > /etc/ld.so.conf.d/spack-view.conf && ldconfig"
    images:
      os: amazonlinux:2
      spack:
        ref: develop
    labels:
      app: "gromacs"
      mpi: "tmpi"
EOF

We are going to create a Dockerfile out of the spack spec above.

spack containerize \
  | sed -e '/RUN find -L /,+4d' > Dockerfile
Build and Push

Let’s create an ECR repo specific to this GROMACS image

aws ecr create-repository --repository-name=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"
export GROMACS_BASE=$(aws ecr describe-repositories --repository-names=gromacs/2021.1/${ARCH}/${TECH} --region "${AWS_REGION}"| jq -r '.repositories[0].repositoryUri')
echo "# GROMACS_BASE=${GROMACS_BASE}"

Now, we can build and push the image.

spack containerize > Dockerfile
docker build -t ${GROMACS_BASE}:${TAG} .
docker push ${GROMACS_BASE}:${TAG}
Import into Sarus

To make this image available, we are going to export the image locally and import it into sarus.

mkdir -p /fsx/images/gromacs/2021.1/${ARCH}
docker save ${GROMACS_BASE}:${TAG} -o /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar
sarus load /fsx/images/gromacs/2021.1/${ARCH}/${TECH}.tar gromacs/2021.1/${ARCH}:${TECH}