Architecture and Infrastructure

The DoT Application architecture is divided into 3 main key areas:

1. Backend Architecture and Data Platform
2. Visualisation WebApp
3. External APIs to receive data from data sources such as Jira, Zephyr, and Automation tools

This section describes the various areas along with security practices in place.

Backend Architecture

This diagram outlines the ingestion flow of DoT, focusing on three core components: DoT Data Sources, DoT Data Lake, and QEngine. The objective is to establish a secure framework for handling and transferring data within these components.

Data Sources

The Data Sources component serves as the primary interface for inbound data within the DoT system. Inbound data is processed through two distinct methods: an HTTPS POST operation (Push) or an internal DoT connector (Pull) designed to retrieve data from external systems.

Examples of data sources include a push of JUnit xml from an automation execution run or a pull from a Jira data connector which is scheduled on a nightly basis.

Data Lake

After undergoing a validation process, where the ingested data is carefully checked against expected formats and compared with existing duplicates in the Data Transformation system, the processed documents are stored in a data lake. At the same time, these documents are indexed in the associated database.

QEngine

DoT’s QEngine is responsible for converting ingested data into searchable time series entries. These entries are utilized as the basis for generating graphs and projects within the DoT Web App. The QEngine’s role is focused on providing the necessary functionality for powering data visualisations in the web application.

Data in Transit

All public-facing endpoints for DoT are configured to utilize HTTPS exclusively, providing secure communication. SSL/TLS encryption is employed to protect data during transit over the public internet. Similarly, internal communications within the AWS environment between DoT services and infrastructure are secured using established AWS encryption protocols, ensuring the confidentiality and integrity of data in transit.

See AWS Data In Transit

Data at Rest

DoT stores data in three different repositories: S3, DynamoDB, and OpenSearch.

Each of these AWS managed services is encrypted at rest.

• AWS applies data encryption at rest for S3 buckets using Amazon S3 managed keys (SSE-S3). All new documents uploaded into DoT are automatically encrypted through this mechanism. See AWS S3 Encryption

• AWS DynamoDB encrypts all data at rest. See AWS DynamoDB Encryption

• DoT uses an AWS OpenSearch domain with data encryption at rest. All indexes are encrypted at rest. All data uploaded by a DoT tenant is stored in an OpenSearch index. See AWS OpenSearch Encryption

Direct access to data at rest stored in S3, DynamoDB, or OpenSearch is highly restricted by the established access controls.

Access Controls

To provide controls on who to can access the AWS infrastructure hosting DoT, the DoT Development Team follows a principle of least privilege. This means we only provide AWS administrators access to a small subset of users. In addition, all user/dev accounts are configured to use the Planit SSO for Dev, Test, and UAT environments. DoT production accounts are limited to support personnel and DOT administrators.

A limited number of people within Planit have access to the AWS console and access to the data stored in DoT’s infrastructure.

This role is controlled by Planit’s internal change management process and requires a series of approvals from DoT’s product owner, Planit Security, and Planit’s Executive General Manager for Technology & Innovation.

Planit Data Policies

Planit has achieved ISMS (Information Security Management System) based on ISO 27001 standards across Australia (Sydney, Adelaide, Melbourne, Brisbane, Perth), Hyderabad, and the Philippines. In addition, Planit has published a Data Usage agreement which specifies how data is collected and retained. A copy of this agreement will be provided to the client.

Planit Secure Development Policy

Planit has developed a secure development policy that guides the creation of code within Planit. This policy covers secure practices in areas of app configuration, secret storage and use, source control, dependency audit, docker container security, and IDE extension usage. This policy is reviewed on a yearly basis and updated if necessary.

DoT’s development follows Planit’s Secure Development Policy. This document can be provided upon request from the client.