ParallelCluster

In this chapter we are going to install and configure ParallelCluster within a CloudShell environment. ParallelCluster is installed using python-pip.

The following scripts are executed every time a new terminal (CloudShell) is opened (as they are added to ~/.bashrc). With CloudShell it might happen that a new instance is used and we need to reinstall.

We are using ParallelCluster 2.11.x in this workshop, the latest is ParallelCluster 3. Please consider using the latest.

cat > ${HOME}/pcluster-install.sh << EOF
#!/bin/bash

if ! command -v pcluster &> /dev/null
then
  echo ">> pcluster is missing, reinstalling it"
  sudo pip3 install 'aws-parallelcluster<3'
else
  echo ">> Pcluster $(pcluster version) found, nothing to install"
fi
EOF
chmod +x ${HOME}/pcluster-install.sh
echo "bash ${HOME}/pcluster-install.sh" >> ~/.bashrc
bash ${HOME}/pcluster-install.sh

This will result in the following output:

Tools

We’ll use a couple of tools to make life easier.

  • moreutils: The command sponge allows us to read and write to the same file (cat a.txt|sponge a.txt)
  • jq: Manipulating json
  • wildq: Tool on-top of jq to manipulate INI files
cat > ${HOME}/prereq-install.sh << EOF
if ! command -v sponge &> /dev/null
then
  echo ">> sponge is missing, reinstalling prerequisites"
  sudo amazon-linux-extras install epel -y
  sudo yum --enablerepo epel install -y moreutils
  sudo yum install -y jq
  sudo pip3 install wildq
else
  echo ">> prerequisites are installed, nothing to do"
fi
EOF
chmod +x ${HOME}/prereq-install.sh
echo "bash ${HOME}/prereq-install.sh" >> ~/.bashrc
bash ${HOME}/prereq-install.sh

Wildq & Sponge use

In the workshop we will present changes to the configuration files in two ways. First, as the snippet that needs to change:

Key: value

And afterwards how the snippet can be achieved with a magic command using wildq and sponge.

cat file.ini \
  |wildq -i ini -M '.Key = "value"' \
  |sponge file.ini

This does the following.

  1. cat file.ini outputs the file content to stdout
  2. wildq -i ini -M '.Key = "value"' takes the content (which it expects to be in the ini format), and updates the value for Key to be value. In case the key does not exists it will create one.
  3. sponge file.ini reads the output of the previous command (the altered content) and overwrites the content of the file file.ini just before it exits. So that the original file is replaced. That’s because cat file.ini > file.ini will result in an empty file no matter the initial content (without going into details why; just try each step in echo test > test.txt; cat test.txt ;cat test.txt > test.txt;cat test.txt).