In part one of this series, we showed how to set up an NVIDIA BlueField-2 DPU with the aim of simply using it as a NIC (https://rcblog.erc.monash.edu.au/blog/2022/02/how-do-i-use-a-dpu-as-nic/). Today we’re going to delve a bit deeper and offload network layer encryption to the DPUs so that VMs running on the Host can use more of their allocated resources while still communicating securely.

Background​

Servers and processors are becoming faster and more powerful, but at the same time the requirement for greater network security is increasing and end-to-end network encryption is mandatory in some of the more sensitive data research workloads. As the data throughput on the CPU is pushed to the limits, processors start to steal compute cycles for the network to be encrypted away from computing tasks for research work.

Our aim is to try to mitigate part of this issue by offloading the work of encryption and decryption of traffic from the processors on the hypervisor(host) to the DPU and enable our researchers to work more efficiently and securely.

Set up​

Below is a diagram of the setup we used while testing:

Figure: Diagram of the setup we are describing. Two hosts running one VM each, each host has a DPU installed in a PCI slot, the DPUs are connected directly to each other (back to back).

As we explained in our previous blog post, the DPU’s have programmable hardware for data processing, in this example the DPU has been programmed to process data coming in on port p0 which is then passed to the host which in turn is presented to the VM as the Virtual Function (vf0). The virtual function interface appears at the VM as eth1.

Setting up the DPU

Once the environment is set up as above we can move on to configure Open vSwitch and strongSwan encryption software on the DPU(yes that is the correct capitalisation stylisation)

To set up the OVS bridges and strongSwan (5.9.0bf) IPSec tunnel we used NVIDIA’s example from this website
https://docs.nvidia.com/doca/sdk/east-west-overlay-encryption/index.html

The change we needed to make was to add the interface pf0vf0 to the OVS Bridge vxlan-br0 on both DPUs to allow the VM’s to communicate with each other.

Example of OVS settings from one of the DPUs

ubuntu@localhost:~$ sudo ovs-vsctl show
352ec404-2519-4751-9a1b-3fd33780543c
    Bridge vxlan-br0
        Port pf0vf0
            Interface pf0vf0
        Port vxlan-br0
            Interface vxlan-br0
                type: internal
        Port vxlan11
            Interface vxlan11
                type: vxlan
                options: {dst_port="4789", key="100", local_ip="192.168.50.1", remote_ip="192.168.50.2"}
        Port pf0hpf
            Interface pf0hpf
    ovs_version: "2.15.1-d246dab"

With the above OVS configuration, the logical flow of packets will be as below.

Here is a diagram of the “Path” defined by the above configuration:

Diagram of the logical setup:

Figure: Logical flow of packets through the DPU to the VM.

Our current setup uses the Slow Path

The diagram was referenced on NVIDIA’s original from __(https://docs.nvidia.com/doca/sdk/l4-ovs-firewall/index.html)

Setting IPSec Full Offload Using strongSwan

strongSwan configures IPSec HW full offload using a new value added to its configuration file.

By default two files are created in /etc/swanctl/conf.d when flashing the DPUs with DOCA SDK.

BFL.swanctl.conf and BFR.swanctl.conf

We only want one of these on each host. BFL on Host 16 and BFR on Host 17

We also want to make some changes to the .conf files.

On DPU 16​

cd /etc/swanctl/conf.d/
mv BFR.swanctl.conf BFR.swanctl.conf.old
vi /etc/swanctl/conf.d/BFL.swanctl.conf
#Note edit this file manually, copying the below output will probably result in issues
cat /etc/swanctl/conf.d/BFL.swanctl.conf
# LEFT: strongswan BF-2 config file
connections {
    BFL-BFR {
        local_addrs = 3.3.3.2
        remote_addrs = 3.3.3.3
        local {
            auth = psk
            id = host2
        }
        remote {
            auth = psk
            id = host1
        }
        children {
            bf {
                local_ts = 3.3.3.2/24 [udp/4789]
                remote_ts = 3.3.3.3/24 [udp/4789]
                esp_proposals = aes128gcm128-x25519-esn
                mode = transport
                policies_fwd_out = yes
                hw_offload = full
            }
        }
        version = 2
        mobike = no
        reauth_time = 0
        proposals = aes128-sha256-x25519
    }
}
secrets {
    ike-BF {
        id-host1 = host1
        id-host2 = host2
        secret = <large string>
    }
}

On DPU 17

cd /etc/swanctl/conf.d/
mv BFL.swanctl.conf BFL.swanctl.conf.old
vi /etc/swanctl/conf.d/BFR.swanctl.conf
#Note edit this file manually, copying the below output will probably result in issues
cat /etc/swanctl/conf.d/BFR.swanctl.conf
# RIGHT: strongswan BF-2 config file
connections {
    BFL-BFR {
        local_addrs = 3.3.3.3
        remote_addrs = 3.3.3.2
        local {
            auth = psk
            id = host1
        }
        remote {
            auth = psk
            id = host2
        }
        children {
            bf {
                local_ts = 3.3.3.3/24 [udp/4789]
                remote_ts = 3.3.3.2/24 [udp/4789]
                esp_proposals = aes128gcm128-x25519-esn
                mode = transport
                policies_fwd_out = yes
                hw_offload = full
            }
        }
        version = 2
        mobike = no
        reauth_time = 0
        proposals = aes128-sha256-x25519
    }
}
secrets {
    ike-BF {
        id-host1 = host1
        id-host2 = host2
        secret = <large string>
    }
}

Note: Make sure there is a new line at the end of these files or the config may not be applied correctly.

Commands to load strongSwan configuration​

Commands to switch offloading on and off​

Experiments and results​

We can now transmit data between the VMs which will automatically be encrypted by the DPUs as the information goes over the wire. We use iperf3 to generate network traffic between the VM’s while switching on and off hardware offloading capability on the DPU.

We start with hw-offloading disabled to observe the following results

Offload Disabled

With offloading turned off via OVS on the DPU, we can see a ksoftirqd process with high cpu utilisation in the DPU.

Next we turn on hw-offloading and observed the change in behaviour

Offload Enabled

With hw-offload enabled we no longer see the ksoftirqd having high CPU utilisation on the DPU

And the rate of data transfer increased more than 10 times than what we were seeing with hw-offload disabled.

We can also view the offloaded flows on the DPU when hw-offload is enabled:

Which proves that the offloaded flows were indeed executed on the DPU.
The above output shows the offloaded flows and their source IP address, destination IP address, source port, destination port, protocol and the number of packets and bytes in the flow.

Conclusion​

At this point we’ve set up two VMs and built an ipsec tunnel between them on the DPUs which automatically encrypts data sent over the wire.

We’ve shown a noticeable speed difference between enabling and disabling hw-offload and shown how to check the flows when they are being offloaded and the impact on performance when they are not being offloaded.

Future​

As we provide IaaS (Infrastructure-as-a-Service) using OpenStack, the next step is to automate the setup of this configuration in the infrastructure using our CICD configuration management at DPUs so that the network traffic from the research workload VMs are encrypted underneath. We also consider having dedicated IPSec tunnels for the VMs from different projects running on the host so that underlay encrypted network fabric is separated for each project.

Written by Ben Boreham, Shahaan Ayyub, Swe Aung and Steve Quenette as part of a partnership between Nvidia, the Australian Research Data Commons (ARDC), and Monash University

Associate Professor Paul Lacaze is the head of the Public Health Genomics Program within the School of Public Health and Preventive Medicine, Monash University. Since 2015 this program has formed an integral part of the ASPREE studyhttps://aspree.org/aus/ and ASPREE Healthy Ageing Biobankhttps://aspree.org/aus/about-the-aspree-project/sub-studies/. Their strategy was to partner with genomic sequencing facilities from across the globe, who each bought distinct expertise to the challenge of sequencing thousands of ASPREE participants for precision medicine applications. Each partnership has since been on a mission to integrate the study’s phenotypic and clinical outcome data with its novel techniques to understand the role of genetics in healthy ageing and diseases.

Hence our multiple global research collaborations needed an environment where they could discover how to join sensitive and big data such that insights could emerge. Solving such multi-disciplinary techno-social problems is the bread & butter of the digital cooperatives group within the Monash eResearch Centre (MeRC). We activated a hybrid of HPC-like and cloud resources appropriate for the active merging of sensitive clinical trial data with the targeted DNA sequencing data to solve this problem. The researchers explored a range of computational tools & techniques that were in themselves still an experiment. Learnings from engagements like this inform the processes and procedures we have today. Most pertinently, however, we took those communities and their respective organisations through the journey, and they now enjoy low barriers to generating impacts from these collaborations.

One of the research-led sequencing technologies or techniques is the “Targeted Sequencing” (Super Panel). The program collaborated with the Icahn School of Medicine at Mount Sinai to design a panel with around ~700 distinct genes that capture the following gene groups: Cancer Genes, Cardiovascular Genes, PGX Genes, ACMG 56 Genes, Resilience Genes and Maturity Onset Diabetes of the Young (MODY) Genes.

From a logistical point of view, one of the advantages of target sequencing is the smaller storage footprint compared to whole-genome sequencing (which can be ten times bigger in terms of file size). The Mount Sinai group sequenced 13,000 ASPREE samples over several months. That generated ~30TB of sequence alignment files (BAMs) and variation files (VCFs). The first task (back then) was to establish a secure transfer channel for the data to Monash for both storage and downstream analysis. Immediately Paul identified that he did not have the tool or service readily available for this task. That’s when he engaged with MeRC. The Research Cloud at Monash (R@CMon) and digital cooperatives teams within MeRC provided the solution to address the project’s data transfer, storage, and computational requirements (see Figure 1 below). We have since co-operated this infrastructure with them.

Figure 1. ASPREE Targeted Panel Ecosystem

In addition to collaborating with the Public Health Genomics Program and the program collaborating with clinical genomics leaders from across the globe, we collaborated with a software vendor. Together with BC Platforms, we designed a custom-build information system that meets the requirements of both clinical and genomics data processing. The digital cooperatives team provided hosting through the Research Cloud and Research Data Storage. We configured the analysis servers deployed on the Research Cloud with bioinformatics tools for processing the genomics data. Additionally, we deployed three core commercial products from BC Platforms: BC Genome – a secure online database (data warehousing system) for storing and dynamic analysis of genotype and phenotype data, BC Safebox – a secure remote desktop environment for controlled access and collaborative research management, and BC Predict – a web service for variant interpretation, curation and reporting, designed for clinical and medical researcher uses in pathogenicity. Figure 2 below shows an example of the variant curation interface in BC Predict.

Figure 2. Variant Curation in BC Predict

The digital cooperatives team deployed BC Platforms and the surrounding environment in a manner appropriate for sensitive genomics information. In collaboration with Monash University’s central IT (eSolutions), we contracted an external security penetration testing service to assess the deployment for handling sensitive information without losing the inherent scalability and configurability of the Monash Research Cloud. Figure 3 shows the high-level components of the ASPREE genomics information system.inherent scalability and configurability of the Monash Research Cloud. A high-level components diagram of the ASPREE genomics information system is shown in Figure 3.

Figure 3. ASPREE Genomics Information Systems Components

After four years of operations, the genomics system continues to establish close collaborations with national and international research communities. It has produced high impact research outcomes along the way*Genet Med. 2020 Nov;22(11):1883-1886. doi: 10.1038/s41436-020-0881-7. Epub 2020 Jul 1. Medically actionable pathogenic variants in a population of 13,131 healthy elderly individuals. Paul Lacaze 1, Robert Sebra 2 3, Moeen Riaz 4, Jane Tiller 4, Jerico Revote 4, James Phung 4, Emily J Parker 4, Suzanne G Orchard 4, Jessica E Lockery 4, Rory Wolfe 4, Maya Strahl 2, Ying C Wang 2, Rong Chen 2 3, Daniel Sisco 3, Todd Arnold 3, Bryony A Thompson 5, Daniel D Buchanan 5 6, Finlay A Macrae 5, Paul A James 5, Walter P Abhayaratna 7, Trevor J Lockett 8, Peter Gibbs 9, Andrew M Tonkin 4, Mark R Nelson 4 10, Christopher M Reid 4 11, Robyn L Woods 4, Anne M Murray 12, Ingrid Winship 5, John J McNeil 4, Eric Schadt 2 3 *Genomic Risk Prediction for Breast Cancer in Older Women. Lacaze P, Bakshi A, Riaz M, Orchard SG, Tiller J, Neumann JT, Carr PR, Joshi AD, Cao Y, Warner ET, Manning A, Nguyen-Dumont T, Southey MC, Milne RL, Ford L, Sebra R, Schadt E, Gately L, Gibbs P, Thompson BA, Macrae FA, James P, Winship I, McLean C, Zalcberg JR, Woods RL, Chan AT, Murray AM, McNeil JJ. Cancers (Basel). 2021 Jul 14;13(14):3533. doi: 10.3390/cancers13143533. *Rare Germline Pathogenic Variants Identified by Multigene Panel Testing and the Risk of Aggressive Prostate Cancer. Nguyen-Dumont T, Dowty JG, MacInnis RJ, Steen JA, Riaz M, Dugué PA, Renault AL, Hammet F, Mahmoodi M, Theys D, Tsimiklis H, Severi G, Bolton D, Lacaze P, Sebra R, Schadt E, McNeil J, Giles GG, Milne RL, Southey MC. Cancers (Basel). 2021 Mar 24;13(7):1495. doi: 10.3390/cancers13071495.. The R@CMon team is excited about supporting the ASPREE Genomics team as it scales up its research endeavours.

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve; Lacaze, Paul (2022): ASPREE information systems for genomic medicine using targeted panel sequencing. Monash University. Online resource. https://doi.org/10.26180/19669584.v1.

The facility provides researchers access to the 3D structure of biological macromolecules, necessary for the rational design of therapeutics. It does this by hosting an extensive array of fully-automated and high-throughput crystallisation equipment. Every year the facility processes about 1300 crystallisation plates for Monash users, each plate allowing 96 experiments for one or two samples at a time. In the second half of 2021 the facility also welcomed users from the Walter and Eliza Hall Institute of Medical Research who ran experiments at the facility on an interim basis until the first quarter of 2022. Figure 1 below shows the MMCF facility with its instrumentations.

Figure 1. The MMCF Facility

Back in 2016*https://rcblog.erc.monash.edu.au/blog/2016/09/mmcf-upgrade-on-rcmon/, The MMCF partnered with Formulatrix, Monash eSolutions and the Monash eResearch Centre to upgrade the facility’s IT infrastructure for a new generation of crystallisation technology. A high-level diagram of the MMCF connection flow and architecture is shown in Figure 2. The R@CMon team provisioned a custom Microsoft Windows-based infrastructure on the Monash node of the NeCTAR Research Cloud for hosting the platform’s crystallisation and imaging system. An enterprise database has been configured and maintained by R@CMonMonash eSolutions to support this new system. The facility’s networking infrastructure has been completely revamped by Monash eSolutions too to accommodate the new interconnected MMCF services. The R@CMon team worked with the vendor, Formulatrix, to deploy the instrument’s software stack. Through Research Data Storage (RDS), a dedicated research data storage has been provisioned for the facility’s experiments and imaging data, and has since been expanded to 50 TB to cope with the uptake in experiments. The new MMCF platform utilised the infrastructure as a service capability of R@CMon, providing on-demand compute and storage resources.

Figure 2. MMCF Connection Flow/Architecture.

The feedback on the facility has been consistently positive through the annual MTRP user feedback surveys. The facility has supported research on SARS-CoV-2https://www.monash.edu/discovery-institute/news-and-events/news/2020-articles/melbourne-researchers-map-the-structure-of-a-covid-19-protein, the virus responsible for the Covid-19 pandemic, and facilitated the gaining of major funding and rewards by various researchers such as an NHMRC Synergy Grant in 2019https://www.grants.gov.au/Ga/Show/7592a1cb-d64b-d970-8077-e81d75a8d51c. The Director of the facility, Prof Jamie Rossjohn*https://research.monash.edu/en/persons/jamie-rossjohn-faa-fahms-flsw-fmedsci, whose laboratory is a major user of the facility, is ranked in the top one percent by citations in the field of immunology every year from 2018 to 2021. The R@CMon team continues to support the platform to provide high-throughput crystallisation services to researchers. The team at Monash is also working with Formulatrix to test the next generation capabilities and improvements that further improves user experience and processing.

In this series we are exploring the Nvidia BlueField 2 DPUs (Data Processing Units). We predict that before too long, DPUs will be pervasive in the datacenter, where many users will be using DPUs without realising. This series is for data centre, cloud and HPC builders, who seek to understand and master DPUs for themselves. Most will use DPUs via software some other company has written (e.g. your favourite cybersecurity software). However, some will encode some business critical function (like encryption) onto DPUs, ensuring the users have no idea (no performance loss, no need to learn anything). Check out Steve's GTC 2021 talk - "Securing Health Records for Innovative Use with Morpheus and DPUs" for a good introduction to DPUs for this series.

For the purposes of this series, our goal is to offload encryption from virtual machines running on each host onto the DPUs. This has two important benefits:

1. Eliminates the need for VM users (researchers in our context) to add transport layer security themselves, creating a lower level of entry knowledge required for them to do their work breaking down the technical barrier.
2. Achieves higher work / processing throughput as the security work is offloaded from the CPU itself.

A DPU is specialised programmable hardware for data processing outside of the CPU, but still on the server. The DPUs contain their own CPU, some accelerators (e.g. for encrypting), a NIC and can be programmed to behave differently depending on your needs.

A photo of one of our DPUs

In this blog we are looking at the most basic functionality: configuring DPUs as NICs for communication between two hosts. We’ve compiled some steps and a list of some of the things that caught us out. Each of these steps were run on both hosts unless otherwise noted.

By default the DPU should act like a NIC out of the box. However it may have already been used for something else. Sometimes the DPU will be loaded with the image you want... sometimes it won’t. Hence we will assume we will need a fresh start to work from. If you are anything like us, you’re using a pair of Ubuntu 20.04.3 LTS installations running on Dell servers with a mix of brand new DPUs and older DPUs.

Glossary of terms:
DOCA = Data Center-on-a-Chip Architecture
DPU = Data Processing Unit
NIC = Network Interface Card
OVS = Open Virtual Switch (also known as Open vSwitch)

What are we trying to achieve?

In the logical (OVS) diagram provided by Nvidia we see that inside the DPU, the physical port connects to the p0 interface which is forwarded to the pf0hp0, which then appears inside the host as the PF0 interface. In the diagram below we see two of the modes the DPU can run.

The “Fast Path” mode bypasses the DPU processors. Conversely the “Slow Path” will use the DPU’s processors. Our understanding is that all new connections first occur through the Slow Path. Then if the DPU is configured to behave as a NIC, the E-Switch knows it can bypass the DPU processors themselves. The Slow Path is the stepping stone to doing much more interesting things.

Source: Nvidia (https://docs.nvidia.com/doca/sdk/l4-ovs-firewall/index.html)

See our practical implementation for the simple DPU as a NIC case below. We keep the eth0 interfaces of the host connected to a switch for management purposes. The p1p1 (PF0) interfaces of the Bluefield 2 DPU cards are connected directly to each other.

Diagram of how our machines are connected.

Installing drivers, flashing the device (and installing DOCA via the NVIDIA SDK manager)​

Once the DPU is installed in a PCI slot in your host machine you’ll probably want to install drivers and connect to the DPU.

DPU usually comes with Ubuntu OS installed as default. In that case we just need to install a MOFED driver on the host to be able to use the DPU.

If you want to reimage the operating system of the DPU, you will need the NVIDIA DOCA SDK installed via the NVIDIA SDK manager.

Additional information about setting up the NVIDIA SDK manager on the host can be found at: https://developer.nvidia.com/networking/doca/getting-started

In our case this meant installing the latest version of DOCA for Ubuntu 20.04.

First we downloaded the sdkmanager_1.7.2-9007_amd64.deb package and transferred it to the host. To download this file you need to be logged in to Nvidia’s dev portal so it’s best to do this from a browser)

sudo dpkg -i sdkmanager_1.7.2-9007_amd64.deb
#If you get dependency errors run the following
sudo apt-get update && sudo apt-get upgrade
sudo apt-get -f install
sudo apt-get install libxss1
sudo apt-get install docker
sudo dpkg -i sdkmanager_1.7.2-9007_amd64.deb
#Then confirm that you have the latest version with
sudo apt install sdkmanager -y
#Then run the sdkmanager
sdkmanager

On the first run you will need to log in to NVIDIA’s devzone services (the sdkmanager tool prompts you to log in to a website and enter a code / scan a QR code).
We opted to use X11 forwarding to log in via the GUI.

Further information about this process can be found at:

https://docs.nvidia.com/sdk-manager/download-run-sdkm/index.html#login

Once the NVIDIA SDK manager has been installed you can install the drivers and flash the DPU using the following command:

#Note if you have a previous version of DOCA installed you can uninstall it using this command
sdkmanager --cli uninstall --logintype devzone --product DOCA --version 1.1 --targetos Linux --host


#(Re)installing DOCA
sdkmanager --cli install --logintype devzone --product DOCA --version 1.1.1 --targetos Linux --host --target BLUEFIELD2_DPU_TARGETS --flash all

#Note: Default username on the DPU is: ubuntu

On the first run you will need to log in to NVIDIA’s devzone services (the sdkmanager tool prompts you to log in to a website and enter a code / scan a QR code). Further information about this process can be found at: https://docs.nvidia.com/doca/sdk/installation-guide/index.html

A successful NVIDIA DOCA SDK installation (note newer versions look different)

Once the DPU has been successfully flashed you will need to reboot the host to ensure the new interfaces (p1p1 and p1p2) are present.

Note: We renamed the interfaces to p1p1 and p1p2 so that it is easier to remember and use in configuration management

#On both Hosts
sudo reboot
#Once they reboot check that p1p1 and p1p2 are present in
ip a

You should see something like this in: ip link show

6: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:e7:1e:b2 brd ff:ff:ff:ff:ff:ff
7: p1p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 0c:42:a1:e7:1e:b3 brd ff:ff:ff:ff:ff:ff

There should also be management and rshim interfaces of the DPU present.

10: enp66s0f0v0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether ca:29:81:20:cb:51 brd ff:ff:ff:ff:ff:ff
13: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff

You can verify the drivers and firmware that are installed by using the command:

ethtool -i p1p1

ubuntu@HOST-17:~$ ethtool -i p1p1
driver: mlx5_core
version: 5.5-1.0.3 ← Mellanox ofed driver version
firmware-version: 24.32.1010 (MT_0000000561) ← Firmware version of DPU
expansion-rom-version:
bus-info: 0000:42:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Connecting to DPU

Now that DOCA has been installed and the DPU has been flashed with the firmware, we can connect to the DPU. In our case here we configure an ip address at the rshim interface to access DPU.

 ip addr add 192.168.100.1/24 dev tmfifo_net0

9: tmfifo_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:1a:ca:ff:ff:02 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 scope global tmfifo_net0
       valid_lft forever preferred_lft forever
    inet6 fe80::21a:caff:feff:ff02/64 scope link
       valid_lft forever preferred_lft forever

We access DPU via ssh from the Host, but other methods of connecting are listed here: https://docs.mellanox.com/display/MFTV4120/Remote+Access+to+Mellanox+Devices

#Connect to the DPU from the Host
ssh ubuntu@192.168.100.2

#To check the driver and firmware (on the DPU)
ethtool -i p0

#Query from flint (On the DPU)
sudo flint -d /dev/mst/mt41686_pciconf0 q

#Check default OVS configuration (On the DPU)
sudo ovs-vsctl show

The query from flint should look something like this on the DPU:

ubuntu@localhost:~$ sudoflint -d /dev/mst/mt41686_pciconf0 q
Image type: FS4
FW Version: 24.32.1010
FW Release Date: 1.12.2021
Product Version: 24.32.1010
Rom Info: type=UEFI Virtio net version=21.2.10 cpu=AMD64
type=UEFI Virtio blk version=22.2.10 cpu=AMD64
type=UEFI version=14.25.17 cpu=AMD64,AARCH64
type=PXE version=3.6.502 cpu=AMD64
Description: UID GuidsNumber
Base GUID: 0c42a10300e71eb2 12
Base MAC: 0c42a1e71eb2 12
Image VSD: N/A
Device VSD: N/A
PSID: MT_0000000561
Security Attributes: N/A

The default config of OVS should look something like this on the DPU:

ubuntu@localhost:~$ sudo  ovs-vsctl show
10c2d713-1ca3-4106-8eea-1178f3c1348d
    Bridge ovsbr1
        Port p0
            Interface p0
        Port pf0hpf
            Interface pf0hpf
        Port ovsbr1
            Interface ovsbr1
                type: internal
    Bridge ovsbr2
        Port p1
            Interface p1
        Port pf1hpf
            Interface pf1hpf
        Port ovsbr2
            Interface ovsbr2
                type: internal
    ovs_version: "2.14.1"

Thanks to the default OVS (Open Virtual Switch) configuration you can also add IP addresses to the p1p1 interfaces on the hosts to enable connection between them.

#On Host-16
sudo ip addr add 10.10.10.16/24 p1p1
sudo ip link set p1p1 up

#On Host-17
sudo ip addr add 10.10.10.17/24 p1p1
sudo ip link set p1p1 up
ping -I p1p1 10.10.10.16

Troubleshooting:

If your OVS configuration does not match the example or the ping test fails, you might want to try removing all existing OVS configurations using the “ovs-vsctl del-br“

For example if you had a bridge called “arm-ovs” you could delete it with the following command

sudo ovs-vsctl del-br arm-ovs

Then recreate the default OVS bridges ovsbr1 and ovsbr2 with the following commands:

ovs-vsctl add-br ovsbr1
ovs-vsctl add-port ovsbr1 pf0hpf
ovs-vsctl add-port ovsbr1 p0

# configure p1p2 (optional in our case, p1p2 is not used)
ovs-vsctl add-br ovsbr2
ovs-vsctl add-port ovsbr2 pf1hpf
ovs-vsctl add-port ovsbr2 p1

ip link set dev ovsbr1 up
ip link set dev ovsbr2 up

Important note: Adding p0 and p1 of DPU to the same ovs bridge could cause a loop and potentially create a multicast issue in the network.

Some observations, noting we’re dealing with very new technology:​

1. Installing DOCA sdk via the command line is not yet simple.
2. Sometimes DOCA install may fail and trying it again usually works. (Note: The newest version “1.2.0” does not seem to have this issue)
3. To be able to use both ports of the DPU, we observe they need to be configured with IP addresses of the different vlan on the host.

Conclusion

Once the DPUs have been installed and flashed correctly you can easily add IP addresses to the p1p1 interfaces on the hosts to enable network configuration. In the next post we’ll look at the NVIDIA DOCA East-West Overlay Encryption Reference Application.

Written by Ben Boreham, Swe Aung and Steve Quenette as part of a partnership between Nvidia, the Australian Research Data Commons (ARDC), and Monash University

The IndySCC is a 48 hour contest where students run a range of benchmarking software (this year - HPL and HPCG), well established scientific applications (Gromacs, John The Ripper) and a mystery program (Devito), whilst also keeping power consumption to under 1.1KW. That’s right - even the most advanced digital research infrastructure has meaningful Net Zero aspirations!

The six students - the Student Cluster Team - are part of an undergraduate team called Deep Neuron. Deep Neuron itself is part of a larger group of Engineering Teams that offer a range of extra-curricular activities. DeepNeuron is focused on improving the world through the combination of Artificial Intelligence (AI) and High-Performance Computing (HPC).

“The experience of participating in such a well known competition and the opportunity to collaborate with different students and experts allowed us to learn valuable skills outside of our classroom. We feel privileged and would like to thank all the support from DeepNeuron, supervisors and the faculty”

Yusuke Miyashita, HPC Lead, Deep Neuron

This achievement is even more impressive given that the students have never physically met each other due to covid restrictions. Earlier this year, the students also entered the Asia Supercomputing Community 2021 virtual Student Cluster Competition (ASC21 SCC), where they won the Application Innovation Award (shared with Tsinghua University) for the fastest time for the Mystery Application. That team was led by Emily Trau, who also works as a casual at MeRC.

“Despite the COVID lockdown, the students from Monash University’s Deep Neuron have hit well above their weight, winning significant prizes in two prestigious International Student Cluster Competitions. Well done to all involved”

Simon Michnowicz, Monash HPC team

All teams in the competition were tasked with configuring a resource made available to them on the Chameleon Cloud for each benchmark. Chameleon is similar to the Nectar Research Cloud, in that it provides Infrastructure as a Service to researchers. However Chameleon focus is experiments in edge, cloud and HPC (experiments on the infrastructure itself). The Research Cloud focus is being a resource for, and the instigator of collaboration for all research disciplines. Where Chameleon and the Research Cloud and Monash are particularly similar is being the test bed for new hardware and software technologies pertinent to digital research infrastructure. For example, MASSIVE and Monash’s own MonARCH HPC are built on the Research Cloud.

"It is formally the end of the competition. What a journey! You all did an excellent job and we are impressed by how smart, hard-working and dedicated all the teams were. You all deserve a round of applause”

IndySCC21 Chairs Aroua Gharbi and Darshan Sarojini

JohntheRipper cracking passwords

GROMACS simulation of a model membrane

To this end, and after nearly a decade in the making, the eResearch Centre has joined the Open Infrastructure Foundation as an associate member. See the announcement here.

Universities are living laboratories for research-driven infrastructure. They require perpetual & bespoke computing at scale, which when combined, are the killer app for #opensource infrastructure, the associated communities and their practices.

“Monash University has long believed in the power of using open source solutions to provide infrastructure for research, so it is with great pleasure that we formalize our long relationship and welcome them as a new associate member.”

Thierry Carrez, vice president of engineering at the OpenInfra Foundation, partnership announcement

Over the last decade open data and open source software have established legal entities (foundations) to ensure priorities, quality and sustainability of the data/tool are managed at commercial / real-world levels. Our partnership Open Infrastructure Foundation helps our researchers access tools for their own digital instruments that are in-turn produced, curated and maintained at the rate of global cloud development (across all industries). In this regard we’re amongst a pioneering set of institutions including CERN, Boston University and others. We give back by ensuring our research workloads are driving the community and infrastructure, pushing new technologies and expectations through the ecosystem.

"Open source and in particular the OpenInfra ecosystem is the language by which we craft HPC, highly sensitive, cloud and research data instruments at scale in a way that is closer to research needs, and with access mechanisms that is closer to research practice. We look forward to continued sharing of learnings with the community and pioneering of digital research infrastructure."

Steve Quenette, Deputy Director of the Monash eResearch Centre, partnership announcement

To provide some indication of impact - 0.5 billion users (including our ~1000 research CIs) using 1.8m servers / 8.4m virtual machines and 4.5m public IP addresses benefit every contribution made by the global community. (From 2020 survey, which is certainly under-reported)

This article can also be found, published created commons

• Quenette, Steve (2021): Monash University Joins OpenInfra Foundation as Associate Member. Monash University. Online resource. https://doi.org/10.26180/16936072.v1

Even disruptive research tools created as recently as 10years ago, and yet fundamental to improving human interactions with information and computers, are susceptible to the onslaught of cyber security threats that exist today! Sometimes, all that the research fraternity needs is access to small amounts of skilled engineers (both crowd sourced and research software engineers) to make the small changes needed to keep such research infrastructure robustly safe. For community focused research the longevity of the solution is very important. Yet, the research prototypes quite often use open source software which if not updated can attract some security risks.

The technical team at R@CMon is staying vigilant to ensure the research prototypes produced as a result of the research projects can stay usable and useful to the communities even after the research part of the projects were completed. A good example of such an impactful, long research prototype Breast Cancer Knowledge Online which survived many years of use thanks to the hard work of the researchers supported by the R@CMon team.

Professor Frada Burstein, Department of Human Centred Computing, Monash Data Futures Institute, Victorian Heart Institute (VHI)

The Monash Faculty of IT initiative led by Professors Frada Burstein and Sue McKemmish in collaboration with BreastCare Victoria and the Breast Cancer Action Group developed a comprehensive online portal of information pertinent to those facing serious health issues related to breast cancer. This work was supported by Australian Research Council and philanthropic funding (Linkage Grant (2001-2003), Discovery (2006-2009), Telematics Trust (2010, 2012), and the Helen Macpherson Smith Trust (2011), resulting in three consecutive implementation efforts of the unique smart health information portal. The full project team is listed on the portal's "Who We Are" page. The research focussed on the role of personalised searching and retrieval of information, where for example, the needs and preferences of women with breast cancer and their families change over the trajectory of their condition. In contrast a web search bar 10 years ago was generic with very little situational awareness about the person who is searching. The resultant tool, Breast Cancer Knowledge Online (BCKOnline), empowers the individual user to determine the type of information which will best suit her needs at any point in time. The BCKOnline portal uses metadata-based methods to present users with a quality score for data from other public resources carefully curated by breast cancer survivors and other well informed domain experts. The portal’s metadata descriptions of information resources also describe resources in terms of attributes like Author, Title, and Subject. A summary of the information resource and a quality report is also provided. The quality report provides information on where the information came from and who wrote it so the woman can decide if she ‘trusts’ the source.

The underlying technical infrastructure of the portal is utilising open source solutions and has been released to the public in two distinct versions (see Figure 1a and 1b for the interfaces for the personalised search for BCKOnline).

Figure 1a - BCKOnline personalised search (version 2)

The 2009 paper *McKemmish, S., Manaszewicz, R., Burstein, F. and Fisher, J. (2009), Consumer empowerment through metadata-based information quality reporting: The Breast Cancer Knowledge Online Portal. J. Am. Soc. Inf. Sci., 60: 1792-1807. describes the solution as a paradigm shift in quality health information provision sharing, specifically for women and their families affected by breast cancer. BCKOnline has been used for over 100K+ personalised searches across its over 1K curated quality resources. It has been a valuable resource to teach information management students about the process and value of metadata cataloging. More about this research can be found in these papers McKemmish, S., Burstein, F., Manaszewicz, R., Fisher, J., & Evans, J. (2012). Inclusive research design: Unravelling the double hermeneutic spiral. _Information, Communication & Society, 15(7), 1106-1135. Burstein, F., Mckemmish, S., Fisher, J., Manaszewicz, R., & Malhotra, P. (2006). A role for information portals as intelligent decision support systems: breast cancer knowledge online experience. In _Intelligent Decision-making Support Systems (pp. 359-383). Springer, London. Xie, J., Burstein, F., & Evans, J. (2008). Intelligent Support for Resource Quality Evaluation and Description in Health Information Portals. In _HIC 2008 Conference: Australias Health Informatics Conference; The Person in the Centre, August 31-September 2, 2008 Melbourne Convention Centre (p. 225). Health Informatics Society of Australia. Xie, J. (2011). _A user-sensitive resource quality assessment approach for health information portals (Doctoral dissertation, Monash University). Xie, J., & Burstein, F. (2011, April). Using machine learning to support resource quality assessment: an adaptive attribute-based approach for health information portals. In _International Conference on Database Systems for Advanced Applications (pp. 526-537). Springer, Berlin, Heidelberg. De Silva, D., & Burstein, F. (2014). An intelligent content discovery technique for health portal content management. _JMIR medical informatics, 2(1), e2671. Nguyen, B. V., Burstein, F., & Fisher, J. (2015). Improving service of online health information provision: a case of usage-driven design for health information portals. _Information Systems Frontiers, 17(3), 493-511..

Figure 1b. BCKOnline’s personalised search based on user profiles (version 3).

The search results page example is shown in Figure 2 below.

A few years later​

Nine years on (in 2019) the maintainers of BCKOnline led by Dr Jue (Grace) Xie, who’s PhD was also connected to the portal development, reached out to the Research Cloud at Monash team (R@CMon), seeking assistance to migrate BCKOnline from its legacy infrastructure to a modern cloud environment and contemporary security controls. Through the ARDC Nectar Research Cloud [2], a new hosting server was deployed for the revamped BCKOnline. Our team walked Frada and Grace through the standard operating procedure to migrate the application to its new home on the research cloud, where Frada and her team have full transparency and control over the application’s lifecycle. The revamped BCKOnline includes a host of security best practices for digital research infrastructure, such as a long term support operating system and proper SSL termination in the web server.

Figure 2. BCKOnline search results, showing a curated list of resources with additional filtering options.

Another step in security best practices for research applications​

Recently, the Monash University Cyber Risk & Resilience (CISO’s office) and our teams embarked on a journey to uplift the security profile of all applications on our Research Cloud infrastructure. It is a strategic step change in the University’s expectations regarding security best practices. In partnership with Bugcrowd the Research Cloud at Monash participates in the Vulnerability Disclosure Program (VDP), where all applications are regularly scanned for active threats and vulnerabilities. Bugcrowd are novel in that they vet what is essentially a crowd-sourced team of cyber security engineers. When vulnerabilities are indeed identified, we kick in with a standard operating procedure that is cognisant of research practice and culture to address the issues. This procedure includes end-to-end communication and coordination between the security team, the Research Cloud team and the affected service owners (the chief investigators).

In a recent security scan, we discovered that the BCKOnline portal was vulnerable to “Cross Site Scripting (XSS)”, a method often used by bad actors to conduct attacks like phishing, temporary defacements, user session hijacking, possible introduction of worms etc. Typically these vulnerabilities are quick to fix for a research group (a handful of hours or at most days), and our evidence suggests researchers are motivated to fix them quickly to ensure their systems stay both alive and reputedly safe.

Fixing this vulnerability was complicated by commonplace research realities. The original developers were no longer available (the PhD students had long moved on). The source code to the impacted part of the application was not within a version control system. After some time and a bit of detective work, the R@CMon team managed to recover the original source and upload it into a private GitLab. With that complexity solved, the next step was to apply a fix for the XSS vulnerability. Realising the R@CMon DevOps team didn’t have the expertise nor capacity to fix the problem, we attempted to outsource the problem to professional contractors. However, after two false attempts a new approach was taken. The R@CMon team reached out to another team within the Monash eResearch Centre. The Software Development (SD) team brings with them an extensive array of software development expertise and best practices, including DevOps and security practices, which have been vital assets for this software engineering activity. We effectively crowd-source this remediation work to the team (where individuals pick which cases work for them, and they are appropriately rewarded for work they do in their own time).

Simon Yu, a veteran developer within the software development team pinpointed the actual source of the vulnerability in the code. He then quickly implemented a fix by creating a custom “filter” and “interceptor”. The resultant fix is efficient in both its load on the computing resource and its ability to protect other parts of the BCKOnline application with little/no research effort. Now any incoming requests (e.g user input, searches) will pass through the filter and interceptor first, validating its payload before being processed by the BCKOnline search engine. This ensures that only legitimate payloads are processed. We additionally placed the BCKOnline portal URL (https://bckonline.erc.monash.edu/) behind a web application firewall (WAF) managed by the Monash Cyber Risk and Resilience team. This provides an additional layer of security as all incoming traffic (payloads) are first sanitised by the WAF before forwarding it to the actual server. The original security advisory has since been resolved and the BCKOnline portal is back serving the online community with their personalised health searches.

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve; Xie, Jue; Burstein, Frada (2021): Breast Cancer Knowledge Online. Monash University. Online resource. https://doi.org/10.26180/16735006.v1.

Associate Professor Paul Lajbcygier and his colleague Rohan Fletcher from the Monash Business School are custodians of a longitudinal study seeking to understand when stability has been superseded by innovations. To their surprise, the recent Nectar Research Cloud refresh has caused a digital disruption to their own research, reducing analysis time from many months to one week, and in turn changing the focus of research.

How deep does the digital disruption rabbit hole go? We asked Paul and Rohan to tell us about it...

“With the advent of the IT revolution, financial exchanges have changed beyond recognition. With the Centre's help, we have focused our research on how the digital disruption has affected financial markets, considered welfare implications, and potential regulatory changes, with ramifications for regulators, traders, superannuants and all equity market stakeholders."

Associate Professor Paul Lajbcygier

Recently, the Monash eResearch centre has supported Paul’s team by upgrading new essential hardware and infrastructure necessary to interrogate the vast data generated from the Australian equity markets.

“Without the Centre's help, our research would be impossible”.

Associate Professor Paul Lajbcygier

To get an understanding how the refreshed Research Cloud would affect the team, they benchmarked the new hardware against their data. They rerun database MySQL code which searches vast amounts of ASX stock data in order to understand the costs of stock trading using new, innovative price impact models. That prior work led to an A* publication in 2020 in the Journal of Economics, Dynamics and Control *Manh Cuong Pham, Heather Margot Anderson, Huu Nhan Duong, Paul Lajbcygier, The effects of trade size and market depth on immediate price impact in a limit order book market, Journal of Economic Dynamics and Control, Volume 120, 2020, 103992, ISSN 0165-1889, https://doi.org/10.1016/j.jedc.2020.103992. .

This analysis interrogates over 1300 stock’s and their related trades and orders from 2007 to 2013, representing three terabytes of data.

“In order to implement this huge processing task, we have automated the breakdown of these MySQL queries by stock, year and month. This generates over 80,000 SQL scripts, which is a total of 3 gigabytes of SQL analysis code alone."

With the latest hardware provided by Monash eResearch centre, this query took around one week, in contrast to the many months of required running time prior to the provision of the latest hardware.”

Associate Professor Paul Lajbcygier

The architecture of the embarrassingly parallel Stock Price infrastructure hosted on the Nectar Research Cloud consists of a large memory machine running Ubuntu LTS, and a number of smaller staging, analysis machines based also on Ubuntu LTS and Windows.

The project uses MySQL and bash scripts to facilitate the templating of SQL jobs. Scripts are generated on the small VM and then moved to the big memory machine for execution. The number of scripts on the big memory machine is monitored and is held at a pre-set maximum. On completion of a script on the MySQL database the next available script is launched automatically.

Testing of a single script may be performed on the staging database after some automatic modifications have been made to make it compatible for individual execution.

Below is an example chart comparing the execution time for an SQL script when utilising each of the four underlying database storage technologies now available on a big memory machine on the Monash node of the Research Cloud. These being: MySQL’s Memory Storage Engine, utilising RAM Drive, utilising Flash Drive and utilising a mounted volume (a separate Ceph cluster via RoCE). Clearly, the use of the memory engine (blue line) provides the best performance.

“Repeating our published benchmark, the MySQL memory engine is approximately forty times better than the flash drive and mounted volume. This outstanding Memory engine performance occurs because the memory engine is internal to MySQL, thereby avoiding input/output lags required of the file system.”

Associate Professor Paul Lajbcygier

Stock Price search performance based on storage backends.

The team then sought observables to explain why the memory approach performed so well. Below is an example of system load as recorded using Ganglia running the same SQL query. To the left is the Memory Engine CPU load usage, followed by examples of the Flash Drive, Mounted Volume and RAM Drive CPU load usage.

“It is possible to see that the Memory engine utilises all 120 CPU processes consistently, in contrast to the right hand graph which shows other memory methods which do not efficiently utilise the new hardware and incur overheads due to the requirement that they must use the file system."

Associate Professor Paul Lajbcygier

In addition to fine tuning MySQL to the specialist hardware (what they named the “Big Memory Machine”), the benchmarking necessitated the integration of a bespoke Microsoft Windows ecosystem of tools. They used the open source tool HeidiSQL, to both visualise and automate the decomposition of the analysis problem to 120 parallel executing SQL scripts.

Parallel Stock Price SQL executions.

To summarise, We've asked Paul what the overall impact of the revamped Stock Price infrastructure in answering their research questions.

"We're able to utilise data analysis on a more comprehensive data set including the ASX and the US NASDAQ, perform rapid prototyping with quick feedback; and complete analyses that would be intractable using the previous infrastructure."

Associate Professor Paul Lajbcygier

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve; Fletcher, Rohan; Duong, Huu Nhan; Lajbcygier, Paul (2021): Revisiting the next generation of StockPrice infrastructure. Monash University. Online resource. https://doi.org/10.26180/16386552.v1.

Back in 2018 the lab asked for advice on publishing apps online. It turned out the applications are a battery of interactive web applications that are designed to measure cognitive impulsivity of its users. This project was supported by an ARC Linkage Project (LP150100770), which aimed to study and measure the cognitive skills that can produce (or avoid) impulsive human behaviour.

• 

• 

• 

  "Prospector's Gamble" mini-game, a mining game where the right prospector need to be chosen for the job.

The project engaged a local 3rd party developer (Torus Games) to develop the games (ahem, cognitive applications) to a pre-production level using Google’s Firebase platform (enables developers to develop iOS, Android and Web apps easier). Our job was to help the lab to migrate the application into the Nectar Research Cloud at Monash (R@CMon), which also meant mapping a pathway away from Firebase. The project and its data custodians have full governance on the data captured by the applications as part of their data collection activities.

The resulting application suite, the "Cognitive Impulsivity Suite (CIS)", is a series of connected services. The web-app itself is a Unity-based WebGL build with a RESTful API using an ASP.NET backend. The app stores the user observations (generated "measures") from the "trials" into a relational database backend. A Windows-based server is required to host the .NET-based application using the Internet Information Services (IIS) web server. R@CMon provided the required cloud resources and web configuration (.Net, IIS) to migrate the "CIS" application suite from Google Firebase. A high level pipeline diagram of CIS deployment is shown below.

The Cognitive Impulsivity Suite (CIS) pipeline on the Monash Research Cloud

3 years on, the lab is currently conducting 5 major projects using the "CIS" infrastructure on the Research Cloud. The largest of these studies assess impulsivity in more than 1000 US and Australian help-seeking and anonymous participants with drug, alcohol and gambling problems. There are 2 articles *Verdejo-Garcia A, Tiego J, Kakoschke N, Moskovsky N, Voigt K, Anderson A, Koutoulogenis J, Lubman DI, Belgrove MA. A unified online test battery for cognitive impulsivity reveals relationships with real-world impulsive behaviours. Nat Hum Behav. 2021. https://doi.org/10.1038/s41562-021-01127-3 Voigt K, Giddens E, Stark R, Frisch E, Moskovsky N, Kakoschke N, Stout JC, Bellgrove MA, Andrews ZB, Verdejo-Garcia A. The Hunger Games: Homeostatic State-Dependent Fluctuations in Disinhibition Measured with a Novel Gamified Test Battery. _Nutrients. 2021; 13(6):2001. https://doi.org/10.3390/nu13062001 that have been published recently from various impulsivity studies. These research outcomes have utilised the full capabilities of the "Cognitive Impulsivity Suite (CIS)" on the Research Cloud.

The research cloud has enabled us to reliably collect and store cognitive impulsivity data for thousands of participants around the world. We have been able to run several instances of the CIS task, related to different projects at one time. This has been fundamental to the success of each research project and the ability to efficiently separate participant data. We are grateful for the ongoing support we have received from the research cloud team. They have quickly responded to our needs and have offered valued solutions and technical support to enable each of our projects to run smoothly.

Alexandra Anderson, Addiction and Impulsivity Research (AIR) Lab, Turner Institute for Brain and Mental Health, Monash University.

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve; Anderson, Alexandra; Verdejo-Garcia, Antonio (2021): Triple Mechanism Cognitive Impulsivity Battery. Monash University. Online resource. https://doi.org/10.26180/15121956.v2.

Why is this? Researchers are increasingly connected, increasingly participating in grand challenge problems, and increasingly reliant on technology. Problem solving for big global challenges, as distinct from fundamental research, can involve large-scale human-related data, which is sensitive and sometimes commercial-in-confidence. Researchers are rewarded to be first to discovery. One way to accelerate discovery is to be the “first to market” with disruptive technology. That is, develop the foundational research discovery tool (think software or instrument that provides the unique lens to see the solution, a “21st century microscope” so to speak). If we think of research communities as instrument designers and builders, they must then build the scientific applications that span the Internet (across local infrastructure, public cloud and edge devices).

What is an example 21st century microscope for a mission-based problem? To prove the effectiveness of an experimental machine learning based algorithm running on an NVIDIA Jetson-connected edge device controlling a building’s battery. It’s informed by bleeding-edge economics theory, participates in a microgrid of power generators (e.g. solar), storage and consumers (buildings) at the scale of a small city, and is itself connected to the local power grid. Through the Smart Energy City project within the Net Zero Initiative we are doing just that.

A tension is observed between mission-based endeavours involving researchers from any number of organisations, and the responsibility for data governance, which ultimately resides with each researcher’s organisation. Contemporary best practices in technological and process controls adds more work to researchers and technology alike, potentially slowing research down. And yet cyber threats are an exponential reality. It cannot be ignored. How do we make it safe and easy for researchers to explore and develop instruments in this ecosystem? How do we create an environment that scales to any number of research missions?

What is the technological and process approach that enables a globe’s worth of individual research contributions to mission-based problems that will also scale with the evolving cyber landscape?

In February, NVIDIA, Monash University’s eResearch Centre, Monash University’s Cyber Risk & Resilience team and the Australian Research Data Commons (ARDC), commenced a partnership to explore the role DPUs play in this microcosm. Monash now hosts ten NVIDIA BlueField-2 DPUs residing in its Research Cloud, essentially a private cloud, which itself forms part of the ARDC Nectar Research Cloud, Australia’s federated research cloud, which is funded through the National Collaborative Research Infrastructure Strategy (NCRIS). The partnership is to explore the paradigm of off-loading (what is ultimately) micro-segmentation onto DPUs, thus removing the burden of increased security from CPUs, GPUs and top-of-rack / top-of-organisation security appliances. Concurrently Monash is exploring a range of contemporary appliances, microsegmentation software and automations of research data governance.

Steve Quenette, Deputy Director of the Monash eResearch Centre and lead of this project states:

“Micro-segmenting per-research application would ultimately enable specific datasets to be controlled tightly (more appropriately firewalled) and actively & deeply monitored, as the data traverses a researcher's computer, edge devices, safe havens, storage, clouds and HPC. We’re exploring the idea that the boundaries of data governance are micro-segmented, not the organisation or infrastructures. By offloading technology and processes to achieve security, the shadow-cost of security (as felt by the researcher, e.g. application hardening and lost processing time) is minimised, whilst increasing the transparency and controls of each organisation’s SOC. It is a win-win to all parties involved.”

Dan Maslin, Monash University Chief Information Security Officer:

“As we continue to push the boundaries of research technology, it’s important that we explore new and innovative ways that utilise bleeding edge technology to protect both our research data and underpinning infrastructure. This partnership and the exploratory use of DPUs is exciting for both Monash University and the industry more broadly.”

Carmel Walsh, Director eResearch Infrastructure & Service, ARDC:

“To support research at a national and international level requires investment in leading edge technology. The ARDC is excited to partner with the Monash eResearch Centre and NVIDIA to explore how to apply DPUs to research computing and how to scale this technology nationally to provide our Australian researchers with the competitive advantage.”

This is an example of the emerging evolution in security technology to security everywhere or distributed security. By shifting the security function as orthogonal to the application (including the operating system), the data centre (Monash in this case) can affect it’s own chosen depth introspection and enforcement, at the same rate that clouds and applications are growing.

“The transformation of the data center into the new unit of computing demands zero-trust security models that monitor all data center transactions in real time,” said Ami Badani, Vice President of Marketing at NVIDIA. “NVIDIA is collaborating with Monash University on pioneering cybersecurity breakthroughs powered by the NVIDIA Morpheus AI cybersecurity framework, which uses machine learning to anticipate threats with real-time, all-packet inspection.”

We are presently forming the team involving cloud and security office staff, and performing preliminary investigations in our test cloud. We’re expecting to communicate findings incrementally over the year.

The ASPREE team had a single commodity physical PC located at the ASPREE office in the School of Public Health and Preventive Medicine. Despite the ASPREE team streamlining processes to appropriately allow project collaborations, an innovation in its own right, collaborators could only perform analysis by being physically in the office. The protocol required data custodians to copy ASPREE phenotypic datasets (via USB sticks) onto the PC, whilst also physically disconnecting the ethernet cable to ensure no unintended access. Collaborators would fly into Melbourne just to run their analysis. This logistically-taxing workflow made collaboration hard and significantly delayed research outcomes. As new project requests emerged from ASPREE sub studies, it became apparent that data management, data governance and the analysis ecosystem would need to be revamped to support the growing demand. A scalable and secure “safe haven” was required.

The Research Cloud team approached the situation from a pragmatic point of view. The team first critiqued the scalability of the analysis environment. We discovered the environment would require security-hardening to protect against intentional and unintentional data leakage. Furthermore the interfacet needed improvement to become intuitive for non-academic, external and international collaborators.

Figure 1. A typical ASPREE Analysis Environment

Fortunately Leostream, a remote desktop (VDI) scheduling platform, was already being used by virtual laboratories on the Monash zone of the Research Cloud. Leostream provides a high-level interface for allocating remote desktops to users. It also allows access to these remote desktops through a web-based (HTML5) viewer. To scale out the analysis environment, the team deployed a number of Windows-based instances on the Monash zone of the Research Cloud. These instances have been pre-configured with analytical tools chosen by the ASPREE community (e.g R/RStudio, SAS, SPSS) and connected to the Monash license servers. A typical analysis environment is shown in Figure 1. above. Access is managed through the Monash Active Directory (Domain) and each analysis instances have been configured with Group Policy Objects (GPOs). These GPOs enforced a number of rules or security controls inside the instances, e.g preventing users from changing desktop settings, access to registry tools and much more. A reserved set of hypervisors have been used to host these secure instances, which also reside on a segregated private network. Hyper-threading has been turned-off on the hypervisors to minimise the risk of Spectre/Meltdown-type vulnerabilities.

Figure 2. ASPREE safehaven architecture

A high-level architecture diagram for the ASPREE safehaven is shown in Figure 2. Monash eResearch Centre’s Research Data Storage (RDS) provides a scalable storage backend to the safehaven. The team augments the storage pool with further controls to appropriately segregate the data. A dedicated user share is created for each approved ASPREE user. This user share is autonomously mounted into the analysis environment upon user login. ASPREE data custodians (managers) have elevated rights to the safe haven storage. They can review (approve or deny) what data goes in (ingress) and data going out (egress). Thus the technology / workflow automates the overall data governance of the ASPREE clinical trial by incorporating it to their own access management system (AMS).

Now operational for more than 3 years, the Research Cloud at Monash and Helix teams cooperate to provide user support for ASPREE safe havens. Several other registries and clinical trials have leveraged this ASPREE solution as their own safe haven. To date, over 100+ internal and international collaborators have used the ASPREE safe haven. This work has be foundational to Monash eResearch, the Research Cloud and Helix’s initiatives towards the next-generation safe havens (e.g. SeRP, which further automates and audits generalised governance workflows).

“The ASPREE data is an NIH-supported clinical trial, and the NIH rightly demands full accountability for data handling. The team has been understanding, professional, flexible and fast. They gave extra consideration for ASPREE's urgent need (in 2016-17) to share our large and unique dataset to collaborators, whilst also supporting confidentiality in an active clinical trial. The co-design approach took into consideration our Data Manager's detailed requirements and produced an excellent environment for effective use and international collaboration centred on ASPREE data. The successfully funded extension study ASPREE-XT depended on getting this right.”

Dr Carlene Britt, ASPREE Senior Research Manager and ASPREE Data Custodian

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve; Padmanabhan, Komathy (2021): Secure safehaven for the ASPREE clinical trial - The need. Monash University. Online resource. https://doi.org/10.26180/14176766.v4.

Wick RR, Heinz E, Holt KE and Wyres KL 2018. Kaptive Web: user-friendly capsule and lipopolysaccharide serotype prediction for Klebsiella genomes. Journal of Clinical Microbiology: 56(6). e00197-18

The R@CMon team provided its standard LAMP platform to host Kaptive on the Research Cloud. This included helping Kelly transition Kaptive from its original web2py mechanism (to quickly create web applications), to a production grade LAMP stack including a dedicated web server and storage backend. Now transitioned, the team can efficiently and effectively cooperate with Kaptive alongside a critical mass of other domain-specific LAMP based applications across all disciplines of research. The team also assisted in applying additional security controls (e.g HTTPS/SSL, reCAPTCHA) on the server to improve its security posture. As a measure of impact, more than 3000 searches (and associated computing jobs) have been submitted into Kaptive to date. As new reference databases become ready and curated, it’ll then be incorporated into Kaptive and made available to the research community.

This article can also be found, published created commons

• Revote, Jerico; Quenette, Steve (2021): Kaptive – How novel searches within bacterial genomic data are presented and hosted on R@CMon. Monash University. Online resource. https://doi.org/10.26180/13601336.v2.

Pham, Manh Cuong and Duong, Huu Nhan and Lajbcygier, Paul, A Comparison of the Forecasting Ability of Immediate Price Impact Models (September 18, 2015). Available at SSRN: https://ssrn.com/abstract=2515667 or http://dx.doi.org/10.2139/ssrn.2515667

Fast forward to 2020 and despite the current world and local circumstances, Paul and his team continue to excel in producing more high impact research outcomes. Their recent successes include a “Journal of Economic Dynamics and Control” publication entitled “The effects of trade size and market depth on immediate price impact in a limit order book market” and an Interfaculty Seeding Grant with the Monash Business School and Faculty of Information Technology to study high frequency trading using machine learning methodologies. There are also numerous research outputs to be submitted towards the end of 2020 and many more towards Q1 of 2021. This surge in high impact outputs correlates to a recent optimisation in the way big queries are executed on the memory engine of the underlying R@CMon-hosted database.

The speed up compared to previous data runs is around four times. This means we can now use more of the memory in the big memory machine effectively.

Paul Lajbcygier, Associate Professor, Banking & Finance, Monash Data Futures Institute

The R@CMon team are currently preparing for the next round of cloud resources uplift in 2021 where “persistent memory” (e.g Intel Optane DC) components are being considered to be included in the resource pool (flavours) available to research cloud users. This could provide even more substantial speedups to big queries on stock price big data. Once ready, the R@CMon team will engage Paul’s team again to utilise these resources.

This article can also be found, published created commons

• Revote, Jerico; Quenette, Steve; Aung, Swe Win (2021): Co-designing clouds for the data future of fintech. Monash University. Online resource. https://doi.org/10.26180/13571279.v2.

An integrated platform and meta-learner for feature engineering, machine-learning analysis and modeling of DNA, RNA and protein sequence data : the impact of making machine learning good practice readily available to the community.

Associate Professor Jiangning Song is a long-standing user of the Monash Research Cloud (R@CMon). He is the lead of the Song Lab within the Monash Biomedicine Discovery Institute. Jiangning's journey began with the deployment of the Protease Specificity Prediction Server - PROSPER app in 2014. Since then the lab has launched more than 30 bioinformatics web services, all of which are made available to research communities worldwide.

Their latest contribution, iLearn, addresses key obstacles to the adoption of machine learning applied to sequencing data. Well-annotated DNA, RNA and protein sequence data is increasingly accessible to all biological researchers. However, at the scale of this data it is challenging if not impossible for an individual to manually investigate. Similarly, another obstacle to broad scale access is that investigation and validation through wet laboratory experiments is time consuming and expensive. Hence when presented appropriately, machine learning can play an import role making higher-level biological data accessible to many researchers in the biosciences.

Many of the previous works and tools only focus on a specific step within a data-processing pipeline. The user is then responsible for chaining these tools together, which in most cases is challenging due to incompatibilities between tools and data formats. iLearn has been designed to address these limitations, using common patterns informed by the lab and its collaborators.

An emerging breakdown of the pipeline steps is:

• Feature extraction
• Clustering
• Normalization
• Selection
• Dimensionality reduction
• Predictor extraction
• Performance evaluation
• Ensemble training
• Results visualisation

iLearn packages these steps for use in two ways. Users can use iLearn through an online environment (web server) or as a stand-alone python toolkit. Whether your interest is in DNA, RNA or protein analysis, iLearn provides a common workflow pattern for all three cases. Users input their sequence data (normally in FASTA format), and then enters various descriptors and parameters for the analysis.

The results page shows the various output, once again informed by the Lab's good-practices. They can be downloaded from the web server in various formats (e.g CSV, TSV). High quality diagrams and visualisations are also generated by iLearn within the web server:

Since iLearn’s release, more than 5K unique users have used the web server worldwide. The user community and resultant impact continues to grow, with 60 citations since the tool's seminal publication.

iLearn has been used as an efficient and powerful complementary tool for orchestrating machine-learning-based modelling which in turn improves the speed in biomedical discoveries through genomics and data analysis. As new descriptors get developed and optimised, iLearn aims to incorporate these into future releases to further improve its performance with the R@CMon team providing support to tackle the potential increase in computational and storage complexities.

This article can also be found, published created commons

• Revote, Jerico; Aung, Swe Win; Quenette, Steve (2020): iLearn on R@CMon. Monash University. Online resource. https://doi.org/10.26180/13166729.v2.

The R@CMon team has worked with Paul's group for several years now, supporting their "big data analysis" workflows on the research cloud. Enabling them to crunch more data, which contributed in several high-impact publications, ARC grant submissions and attainment of a major SEED funding. The international financial workshop event marks the culmination of Paul's groups accomplishments in high frequency trading research over the years and serves as foundation for future critical mass of research in financial markets. The R@CMon team will continue to support Paul's group and the Department of Banking and Finance as they work on more high-impact research and in tackling various computational challenges that they may encounter along the journey.

One of the many structures Ceph makes use of to allow intelligent data access as well as reliability and scalability is the Placement Group or PG. What is that exactly? You can find out here, but in a nutshell PGs are used to map pieces of data to physical devices. One of the functions associated with PGs is 'scrubbing' to validate data integrity. Let's look at how to check the status of PG scrubs.

Let's find a couple of PGs that map to osd.0 (as their primary):

[admin@mon1 ~]$ ceph pg dump pgs_brief | egrep '\[0,|UP_' | head -5
dumped pgs_brief
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
57.5dcc active+clean [0,614,1407] 0 [0,614,1407] 0
57.56f2 active+clean [0,983,515] 0 [0,983,515] 0
57.55d8 active+clean [0,254,134] 0 [0,254,134] 0
57.4fa9 active+clean [0,177,732] 0 [0,177,732] 0
[admin@mon1 ~]$

For example, the PG** 57.5dcc** has an ACTING osd set [0, 614, 1407]. We can check when the PG is scheduled for scrubbing on it's primary, osd.0:

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.5dcc"))'
{
 "pgid": "57.5dcc",
 "sched_time": "2018-04-11 06:17:39.770544",
 "deadline": "2018-04-24 03:45:39.837065",
 "forced": false
}
[root@osd1 admin]#

Under normal circumstances, the sched_time and deadline are determined automatically by OSD configuration and effectively define a window during which the PG will be next scrubbed. These are the relevant OSD configurables:

[root@osd1 admin]# ceph daemon osd.0 config show | grep scrub | grep interval
 "mon_scrub_interval": "86400",
 "osd_deep_scrub_interval": "2419200.000000",
 "osd_scrub_interval_randomize_ratio": "0.500000",
 "osd_scrub_max_interval": "1209600.000000",
 "osd_scrub_min_interval": "86400.000000",
 [root@osd1 admin]#

[root@osd1 admin]# ceph daemon osd.0 config show | grep osd_max_scrub
 "osd_max_scrubs": "1",
 [root@osd1 admin]#

What happens when we tell the PG to scrub manually?

[admin@mon1 ~]$ ceph pg scrub 57.5dcc
 instructing pg 57.5dcc on osd.0 to scrub
[admin@mon1 ~]$

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.5dcc"))'
 {
 "pgid": "57.5dcc",
 "sched_time": "2018-04-12 17:09:27.481268",
 "deadline": "2018-04-12 17:09:27.481268",
 "forced": true
 }
 [root@osd1 admin]#

The sched_time and deadline have updated to now, and forced has changed to 'true'. We can also see the state has changed to active+clean+scrubbing:

[admin@mon1 ~]$ ceph pg dump pgs_brief | grep '^57.5dcc'
 dumped pgs_brief
 57.5dcc active+clean+scrubbing [0,614,1407] 0 [0,614,1407] 0
 [admin@mon1 ~]$

Since the osd has osd_max_scrubs configured to 1, what happens if we try to scrub another PG, say 57.56f2:

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.56f2"))'
 {
 "pgid": "57.56f2",
 "sched_time": "2018-04-12 01:45:52.538259",
 "deadline": "2018-04-25 00:57:08.393306",
 "forced": false
 }
 [root@osd1 admin]#

[admin@mon1 ~]$ ceph pg deep-scrub 57.56f2
 instructing pg 57.56f2 on osd.0 to deep-scrub
 [admin@mon1 ~]$

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.56f2"))'
 {
 "pgid": "57.56f2",
 "sched_time": "2018-04-12 17:11:37.908137",
 "deadline": "2018-04-12 17:11:37.908137",
 "forced": true
 }
 [root@osd1 admin]#

[admin@mon1 ~]$ ceph pg dump pgs_brief | grep '^57.56f2'
 dumped pgs_brief
 57.56f2 active+clean [0,983,515] 0 [0,983,515] 0
 [admin@mon1 ~]$

The OSD has updated sched_time, deadline and set 'forced' to true as before. But the state is still only active+clean (not scrubbing), because the OSD is configured to process a max of 1 scrub at a time. Soon after the first scrub completes, the second one we initiated begins:

[admin@mon1 ~]$ ceph pg dump pgs_brief | grep '^57.56f2'
 dumped pgs_brief
 57.56f2 active+clean+scrubbing+deep [0,983,515] 0 [0,983,515] 0
 [admin@mon1 ~]$

You will notice after the scrub completes, the sched_time is again updated. The new timestamp is determined by the osd_scrub_min_interval (1 day) and osd_scrub_interval_randomize_ratio (0.5). Effectively, it randomizes the next scheduled scrub between 1 and 1.5 days since the last scrub:

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.56f2"))'
 {
 "pgid": "57.56f2",
 "sched_time": "2018-04-14 02:37:05.873297",
 "deadline": "2018-04-26 17:36:03.171872",
 "forced": false
 }
 [root@osd1 admin]#

What is not entirely obvious is that a ceph pg repair operation is also a scrub op and lands in the same queue of the primary OSD. In fact, a pg repair is a special kind of deep-scrub that attempts to fix irregularities it finds. For example, lets run a repair on PG **57.5dcc **and check the dump_scrubs output:

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.5dcc"))'
{
 "pgid": "57.5dcc",
 "sched_time": "2018-04-14 03:43:29.382655",
 "deadline": "2018-04-26 17:18:37.480484",
 "forced": false
}
[root@osd1 admin]#

[admin@mon1 ~]$ ceph pg dump pgs_brief | grep '^57.5dcc'
dumped pgs_brief
57.5dcc active+clean [0,614,1407] 0 [0,614,1407] 0
[admin@mon1 ~]$ ceph pg repair 57.5dcc
instructing pg 57.5dcc on osd.0 to repair
[admin@mon1 ~]$ ceph pg dump pgs_brief | grep '^57.5dcc'
dumped pgs_brief
57.5dcc active+clean+scrubbing+deep+repair [0,614,1407] 0 [0,614,1407] 0
[admin@mon1 ~]$

[root@osd1 admin]# ceph daemon osd.0 dump_scrubs | jq '.[] | select(.pgid |contains ("57.5dcc"))'
{
 "pgid": "57.5dcc",
 "sched_time": "2018-04-13 16:02:58.834489",
 "deadline": "2018-04-13 16:02:58.834489",
 "forced": true
}
[root@osd1 admin]#

This means if you run a pg repair and your PG is not immediately in the repair state, it could be because the OSD is already scrubbing the maximum allowed PGs so it needs to finish those before it can process your PG. A workaround to get the repair processed immediately is to set **noscrub **and nodeep-scrub, restart the OSD (to stop current scrubs), then run the repair again. This will ensure immediate processing.

In conclusion, the sched_time and deadline from the dump_scrubs output indicate what could be a scrub, deep-scrub, or **repair **while the forced value indicates if it came from a scrub/repair command.

The only way to tell if next (automatically) scheduled scrub will be a deep-scrub is to get the last deep-scrub timestamp, and work out if osd_deep_scrub_interval will have passed at the time of the next scheduled scrub:

[admin@mon1 ~]$ ceph pg dump | egrep 'PG_STAT|^57.5dcc' | sed -e 's/\([0-9]\{4\}\-[0-9]\{2\}\-[0-9]\{2\}\) /\1@/g' | sed -e 's/ \+/ /g' | cut -d' ' -f1,21
 dumped all
 PG_STAT DEEP_SCRUB_STAMP
 57.5dcc 2018-03-18@03:29:25.128541
 [admin@mon1 ~]$

In this case, the last scrub was almost exactly 4 weeks ago, and the osd_deep_scrub_interval is 2419200 seconds (4 weeks). By the time the next scheduled scrub comes along, the PG will be due for a deep scrub. The dirty sed command above is due to the pg dump output having irregular spacing and spaces in the time stamp :)

Simultaneous Cumulus network change. Commendation. video...

https://cumulusnetworks.wistia.com/medias/869nn39q1r

Dr. Jiangning Song from the Department of Biochemistry and Molecular Biology at Monash University and his collaborators have designed and developed a bioinformatics tool - GlycoMine-Struct for predicting glycosylation sites. GlycoMine-Struct is a comprehensive tool for the systematic in-silico identification of N-linked and O-linked glycosylation sites in the human proteome. Through R@CMon, a dedicated cloud project with computational and storage resources has been provisioned to develop and host the GlycoMine-Struct tool. The flexible and scalable R@CMon-powered development environment enabled rapid prototyping, testing and re-deployment of the tool.

GlycoMine-Struct is now a publicly accessible web service, available to the wider research community. Users can now easily submit protein structure input files in PDB (Protein Data Bank) format to perform sites prediction on GlycoMine-Struct. Since it went public, GlycoMine-Struct has been accessed and used by thousands of local and international users, and still growing. A scientific reports paper has been published, highlighting the collaborative work done to develop GlycoMine-Struct, as an essential bioinformatics tool for improving the prediction of human glycosylation sites. The R@CMon team is actively supporting the GlycoMine-Struct project as it continues to serve the research community and develop performance improvements.

Sri and his group at the lab studies metabolomics compounds in various samples by conducting a "search" and "identification" process using a pipeline of analysis and visualisation tools. The lab has acquired the license to use the commercial XCMSPlus metabolomics platform from SCIEX on their workflow. XCMSPlus provides a powerful solution for analysis of untargeted metabolomics data in a stand-alone configuration, which will greatly increase the lab's capacity to analyse more samples, with faster and easeful results generation and interpretation.

During the first engagement meeting with Sri and the lab, it's been highlighted that a specialised hosting platform (with appropriate storage and computational capacity) would be required for XCMSPlus. XCMSPlus is distributed as stand-alone appliance (personal cloud) from the vendor. As an appliance, XCMSPlus has been optimised and packaged to be deployed on a single, multi-core and high-memory machine. An added minor complication is that this appliance was distributed in VMWare's appliance format, which need to be translated into an OpenStack-friendly format. The R@CMon team provided the hosting platform required for XCMSPlus through the Monash node of the Nectar Research Cloud.

A dedicated Nectar project has been provisioned for the lab, which is now being used for hosting XCMSPlus. This project also has enough capacity for future expansion and new analysis platform deployments. The now R@CMon-hosted (and supported) XCMSPlus platform for the Immunproteomics Laboratory is the first custom XCMSPlus deployment in Australia. Due to being the first in Australia, there were some early minor issues encountered during its first test runs. These technical issues were eventually sorted out due to collaborative troubleshooting efforts from the R@CM team, the lab and the vendor. And after several months of usage, hundred of jobs submitted and processed by XCMSPlus, and counting, the lab is continuing to fully integrate it as part of their analysis workflow. The R@CMon team is actively engaging with the lab for supporting its adaption of XCMSPlus and planning for future analysis workflow expansions.

The articles abstract provides a great overview:

Criminal investigations invariably involve the triage or cursory examination of relevant electronic media for evidentiary value. Legislative restrictions and operational considerations can result in investigators having minimal time and resources to establish such relevance, particularly in situations where a person is in custody and awaiting interview. Traditional uninformed search methods can be slow, and informed search techniques are very sensitive to the search heuristic's quality. This research introduces Monte-Carlo Filesystem Search, an efficient crawl strategy designed to assist investigators by identifying known materials of interest in minimum time, particularly in bandwidth constrained environments. This is achieved by leveraging random selection with non-binary scoring to ensure robustness. The algorithm is then expanded with the integration of domain knowledge. A rigorous and extensive training and testing regime conducted using electronic media seized during investigations into online child exploitation proves the efficacy of this approach.

In this research Janis et al pioneered Monte-Carlo Filesystem Search (MCFS), a specialised use of Monte-Carlo Tree Search (MCTS), a heuristic search technique commonly employed in game theory. This approach resulted in a high-efficacy (in terms of positive determination, speed, and low resource requirement) approach to classifying potential digital assets of interest in active criminal investigations.

Janis provided the following excellent feedback when we asked him how things were going:

Our project, “Monte Carlo Filesystem Search”, was proposed as a means for accelerating the process of digital forensics by identifying interesting files on suspect hard disk drives. We initially conducted tests on a simulated dataset (a laptop HDD containing simulated illegal photos and movies) hosted on an old server hosted here at FIT. Performance was slow (we were searching entire HDDs, after all!), and the mundane aspects of hosting (ensuring power, network connectivity etc) proved to be far more time consuming than I’d anticipated. We were extremely keen to use the Research Cloud when it became available, as we could spin up instances and conduct parallel tests without having to find rack space, power points, network connections, whitelist MAC addresses, and drive into campus at 1am because a machine had frozen.

Whilst cloud computing is an overused catchphrase (in my opinion), it makes sense for research such as mine. Whilst we couldn’t use shared infrastructure for our final round of tests for the paper (the dataset couldn’t leave the owner’s premises), we heavily exploited R@CMon as a means for testing and bedding down our approach before going onsite.

We’re back on R@CMon for new research (we’re using openly available data), and loving it. No more issues with physical access, hardware/network configuration. Hardware failure? Someone else’s problem, and with redundancy, probably not even noticed. Outages? Advertised in advance. I may need to run experiments a few more times in order to identify outliers caused by network congestion etc, but so what? I don’t need to be there, and I’ve got more than enough other work to get on with in the meantime.