Getting Started with OzReactor

Introduction to OzReactor

OzReactor is a collection of tools designed to make it easier to work with earth observation data on OzSTAR. With it you can search supported data sources for imagery and related data, and download the data you select directly to the supercomputer. If you wish, OzReactor will use OzSTAR’s processing power to convert the data into a common format, so you can view it as a layered map, or perform image analysis without an intermediate processing step.

If you really want to, you can use a subset of OzReactor’s functionality on your computer at home, but that is outside the scope of this tutorial. See developer-documentation for more information.

Installation

There’s some setup to do when using OzReactor for the first time, but if it’s been installed for you, skip to the usage instructions.

Prerequisites

This guide assumes you have access to OzSTAR, the national high performance computing facility at Swinburne University of Technology. You’ll also need:

  • A computer with an internet connection and a web browser

  • An SSH client (preinstalled on most modern desktop systems)

  • The OzReactor software itself – available at https://github.com/OzReactor/OzReactor-HPC/ if you are a member of the GitHub organisation.

Ozrun Installation

Download the OzReactor software on OzSTAR. First, you’ll need to access OzSTAR via ssh. On recent versions of Windows 10, you can simply connect from Windows PowerShell:

Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.

Try the new cross-platform PowerShell https://aka.ms/pscore6

PS C:\Users\bob> ssh bob@ozstar.swin.edu.au
bob@ozstar.swin.edu.au's password:

On MacOS or most Linux distributions, open Terminal and run ssh:

$ ssh bob@ozstar.swin.edu.au
bob@ozstar.swin.edu.au's password:

Now you’re logged into OzSTAR, and you’ll be presented with a bash prompt. Navigate to the folder you want to install OzReactor in; this can be in your home directory, or in a project directory if you have a shared folder under /fred/.

Use mkdir OzReactor to make a directory named “OzReactor”, and use cd OzReactor to enter it.

[alice@farnarkle1 ~]$ mkdir OzReactor
[alice@farnarkle1 ~]$ cd OzReactor/
[alice@farnarkle1 OzReactor]$ pwd
/home/alice/OzReactor
[alice@farnarkle1 OzReactor]$

Now, if you have access to the GitHub repository, you can download the latest version from GitHub.

[alice@farnarkle1 OzReactor]$ module load git/2.18.0
[alice@farnarkle1 OzReactor]$ git clone https://github.com/OzReactor/OzReactor-HPC.git
Cloning into 'OzReactor-HPC'...
remote: Enumerating objects: 1222, done.
remote: Counting objects: 100% (1222/1222), done.
remote: Compressing objects: 100% (805/805), done.
remote: Total 1222 (delta 635), reused 903 (delta 376), pack-reused 0
Receiving objects: 100% (1222/1222), 336.86 KiB | 3.83 MiB/s, done.
Resolving deltas: 100% (635/635), done.
[alice@farnarkle1 OzReactor]$ cd OzReactor-HPC/

Now you can run oz_install, which will take care of the rest of the installation process for you.

[alice@farnarkle1 OzReactor-HPC]$ ./oz_install
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.10.1
  latest version: 4.10.3

Please update conda by running

    $ conda update -n base -c defaults conda

...

Proceed ([y]/n)? y

Press Enter when it asks you if you want to proceed. Ignore the message about updating conda. The installer will run for some time. When it returns you to a prompt, installation has finished.

Installing collected packages: pyparsing, urllib3, toml, py, pluggy, packaging, iniconfig, idna, charset-normalizer, certifi, attrs, requests, pytest, gdal2tiles, auscophub
Successfully installed attrs-21.2.0 auscophub-1.2.2 certifi-2021.10.8 charset-normalizer-2.0.7 gdal2tiles-0.1.9 idna-3.3 iniconfig-1.1.1 packaging-21.0 pluggy-1.0.0 py-1.10.0 pyparsing-2.4.7 pytest-6.2.5 requests-2.26.0 toml-0.10.2 urllib3-1.26.7

OzReactor installation completed.
[alice@farnarkle1 OzReactor-HPC]$

Usage

Using the Web-based Job Configurator

The online job configurator provides a graphical interface for creating OzReactor jobs. On its own, it doesn’t provide any results; instead it produces a JSON configuration file, which is used by ozrun to process the data on OzSTAR. This is unfortunately cumbersome in the current implementation, but has the advantage of running large jobs very quickly on the supercomputer.

Start by opening https://ozreactor-webconfig.pages.dev/ in a web browser.

  1. Enter a name for your job.

  2. Enter a range of dates for the data you want to retrieve.

  3. Select regions of interest on the map. Using the tools on the left of the map, draw a rectangle or arbitrary polygon surrounding your selected area. Click Export underneath the map to confirm the selection.

  4. Under Data Sources, check the boxes for the bands you want to be processed. For true colour imagery, choose “Bands 2-4: True Colour Composite”.

  5. Scroll down and expand Advanced Configuration Settings > Base Directory. This is the path on the supercomputer where your data will be stored. Enter a directory for which you have write access; you will most likely want to use a location inside your project directory e.g. /fred/ozXXX/ozreactor/data_store, where ozXXX is your project identifier (see https://supercomputing.swin.edu.au/account-management/project_memberships). Contact your project leader if you need help choosing a location.

When you’re ready, click “Export JSON” and save the file. You’ll need to copy this to OzSTAR somehow. One way is using an FTP client like FileZilla; alternatively you can copy and paste it into a file through the terminal with nano.

[bob@farnarkle1 ~]$ cd OzReactor/OzReactor-HPC/
[bob@farnarkle1 OzReactor-HPC]$ nano jobs/demo-job.config

You’ll be presented with a terminal-based text editor. Paste in your JSON file and save with Ctrl-X.

  GNU nano 2.3.1        File: jobs/demo-job.config
{
        "job_name": "fkjhgkjhgkjhg",
        "acquisition": {
                "startDate": "2021-10-04",
                "endDate": "2021-10-28"
        },
        "ingest": {
                "collections": [
                        "sen2-b05",
                        "sen2-b06",
                        "sen2-tci"
                ],
                "roi": [
                        {
                                "name": "region_of_interest_0",
                                "geometry": [
                                        {

^G Get Help ^O WriteOut ^R Read File^Y Prev Page^K Cut Text ^C Cur Pos
^X Exit     ^J Justify  ^W Where Is ^V Next Page^U UnCut Tex^T To Spell

Using Ozrun

Ozrun is a command-line utility. For now, you’ll need to navigate to your OzReactor installation directory to use it. Then enter ./ozrun config.json, where config.json is the path of your job configuration file.

[bob@farnarkle1 ~]$ cd OzReactor/OzReactor-HPC/
[bob@farnarkle1 OzReactor-HPC]$ ./ozrun jobs/demo-job.config

You’ll see a lot of text output, and some messages about your jobs being queued. Depending on the server load, it might take a while for your job to run. When it finishes, your data will be in the data store you specified.