OzStar setup

Setting up an account
Installing modules
Running Jupyter notebook


Setting up an account on OzStar

In order to use OzStar one needs:

  1. Create an account
  2. Either join a project, or submit a request for a new project

This can be done here.
Some errors during further steps can be related to user not being in a project.

The work on OzStar machine is carried out remotely from a computer terminal program (MacOS) or a command line (Unix/Windows) using ssh-connection. There are two tips for how to simplify your everyday connection to OzStar:


Installing modules on OzStar

By default there are almost none modules installed on OzStar for new accounts.

See loaded modules with module list
See available for installation modules with module spider

To install modules:

  1. Copy names and versions of required modules to the file .modulelist in your home directory, when ssh-connected to OzStar
  2. Run while read in; do module load "$in"; done < .modulelist
  3. If after step 2 all modules work, and no errors displayed, try adding step 2 command in .bashrc file in OzStar home directory. This way modules are loaded every time you log in. Then try logging in from another window. This way if something is wrong during the login process, you could revert the changes in .bashrc (sometimes it can happen).

If some modules are not working after the installation (i.e. python3 in my case), I recommend adding them in the file .modulelist one-by-one. It turned out that python3 requires gcc installed, while I have not seen this message when I tried to install all my packages at once.

Instructions were adapted from Colm Talbot.


Using Jupyter notebook for python coding on OzStar

There is no trivial way to open a web browser and start using Jupyter notebook on OzStar.
We need to (1) set it all up, (2) run some commands.
We also need to accomplish the previous step "Installing modules on OzStar"

Step I. Setting up Jupyter notebook

1. In your home directory on OzStar create the file jupyter.sh with the following content (no need to change anything):


#!/bin/bash
#SBATCH --nodes 1
#SBATCH --time 4:00:00
#SBATCH --job-name jupyter-test
#SBATCH --output jupyter-log-%J.txt
## get tunneling info
XDG_RUNTIME_DIR=""
ipnport=$(shuf -i8000-9999 -n1)
ipnip=$(hostname -i)
## print tunneling instructions to jupyter-log-{jobid}.txt
echo -e "
Copy/Paste this in your local terminal to ssh tunnel with remote
-----------------------------------------------------------------
ssh -N -L $ipnport:$ipnip:$ipnport [email protected]
-----------------------------------------------------------------
Then open a browser on your local machine to the following address
------------------------------------------------------------------
localhost:$ipnport (prefix w/ https:// if using password)
------------------------------------------------------------------
"
module load gcc/6.4.0
module load python/3.6.4
. ~/jupyter/bin/activate
## start an ipcluster instance and launch jupyter server
jupyter-notebook --no-browser --port=$ipnport --ip=$ipnip

2. In your home directory on OzStar run following commands:

virtualenv ~/jupyter
. jupyter/bin/activate
pip install --user jupyter

Step II. Running Jupyter notebook

1. In your home directory on OzStar run sbatch jupyter.sh

The output will give you the job number and produce a file jupyter-log-{whatever your job number is here}.txt.

2. In your home directory on OzStar run cat jupyter-log-{whatever your job number is here}.txt

The output will give you an IP address, and a port (4 digits).

3. On your local machine run ssh -N -L port:IP:port [email protected]

4. In the web browser go to address: http://localhost:port

5. When token is requested in the web browser, copy a token from a terminal output.

Instructions are adapted from Paul Easter and Ipyrad API - HPC tunnel.

[!!!] On OzStar whatever is installed with "pip install --user" is not available in Jupyter notebook, that is run with sbatch on localhost. This way Enterprise will not be available in Jupyter.

Recent Work