Virtualized Petascale High-Performance Computing

System Case Study for New Material Modelling

Computational Efficiency

*

Rishabh Sinha, x21171203, Cloud Architecture, MSc. Cloud Computing,

National College of ireland, Dublin, Ireland, x21171203@student.ncirl.ie

Abstract—The objective of this project report is to assess the

prospective technical and financial advantages associated with

the transitioning of a leading European research center to a

that can be effectively used in technological frameworks for

material modeling, material screening, selection, and recommen-

dation. The report will analyze the feasibility of implementing

a VMWare virtualized computing model for the existing high-

performance computing (HPC) systems of the research facility.

The research is structured into five primary segments, namely:

an introduction, contextual information, an evaluation of the on-

site high-performance computing (HPC) proposal of the research

facility, functional and non-functional details of HPC petascale,

and a cost analysis that considers multiple variables. The project

Index Terms—New Material Modelling, HPC, Virtualization,

Petascale Computing, Machine Learning, Material Science, Cost

Analysis.

I. INTRODUCTION

HPC has revolutionized material science by allowing re-

searchers to model complex crystal structures and study

environmental factors like temperature, pressure, and more

for New Material Modelling and discovery. This technology

helped find new materials and their uses. HPC has helped

researchers model and evaluate complex material architectures,

resulting in faster and more accurate simulations and lower

costs for creating new materials. New material modeling

has many applications, and materials science and engineer-

ing scholars can use open ICSD, quantum materials, and

SuperCon databases. Pre-processing and feature engineering

prepare raw data for model creation, followed by AI-based

learning tools like ANN and optimization methods like GAs,

[3], [4], [5] [6] However, computational models slow down

tera-scale computing systems and delay result production.

Petascale computing is needed to handle multi-scale, multi-

physics models. This paper suggests petascale computing at a

A. Understanding the application workflow for New Material

Modelling

novel materials with unique properties—is crucial to scientific

study. Trial-and-error experimentation has traditionally been

used to find new materials. This method is expensive and time-

consuming, with limited results. The study facility’s material

discovery research has benefited from computational and ma-

chine learning methods. These methods allow rapid discovery

and modeling of novel materials with unique properties. New

the current HPC environment and enable them to petascale

and exascale if more computational tasks are introduced into

the facility for new research or modern ML tasks.

processing, model building and validation, and inverse

design comprise material discovery [Fig 1.]. Infrastructure

servers manage compute units, GPUs, storage, and networking

in on-premises environments. This system’s infrastructure

server, a Dell-EMC-PowerEdge-R740 with an Intel Xeon-

Silver-4210 processor, will schedule ML jobs for the research

facility’s material modeling and finding process. Compute-

nodes handle ML workloads. This system’s compute unit

7742 processor. Finally, NVIDIA-A100-Tensor-Core GPUs,

built for deep learning, accelerate various ML workloads

in their HPC system. Material Modelling involves below

mentioned steps: Gathering information: Multiple groups

provide standardized datasets with visual representations for

pre-processing. Researchers keep their modeling data in the

sources achieves this. After using raw parameters/features

from antecedent models, fine-tuning is done to incorporate

these features into the forecast model. Supervised learning

models identify a function that can uncover new material

models based on known and existing materials. Regression

problems involve continuous properties. ANNs, Kriging, and

SVMs are used in such cases. Decision trees and random

forests can classify classified outputs. This step requires

high computing power and can vary depending on the model

used to find the most accurate model for researchers to use

candidates for further study. Inverse design uses machine

can validate the best candidates. Neural networks reveal

hidden trends in datasets. Instead of using the network

for prediction, this method extracts relevant data insights.

Similar to combinatorial material science experiments, Ansys

Rocky/Fluent simulation software and CNN-based neural

network models mimic complex data in multiple dimensions.

The model analyzes main data relationship prediction output.

This model interprets the relationships with human expertise

to gain a fundamental comprehension. This model is trained

using machine learning techniques that require GPUs for

CNN models and Ansys Rocky simulation software. [7]

Discovered materials are saved and used to teach models.

materials until desired qualities are achieved. Researchers will

need more advanced storage and high-performance computing

tools to manage and analyze data as it grows, enabling them

exploration. The current High-Performance Computing

The Dell EMC-PowerEdge-R740 infrastructure server, Dell

EMC-PowerEdge-C6525 compute nodes, NVIDIA-A100-

Tensor-Core GPUs, Dell EMC-PowerScale-OneFS storage,

HPC systems. The study facility’s HPC system meets high-

performance computing workloads. The infrastructure server,

compute nodes, GPUs, storage, and networking components

were carefully chosen to provide significant processing power,

extensive memory capacity, scalability, and fast connectivity,

but as researchers use more advanced methods and workloads,

it is becoming harder for the research facility to conduct effi-

erties for various uses. [8] Discovering new materials requires

dation, and inverse design. Discovering new materials requires

complex calculations that require HPC systems. The research

facility’s HPC system includes servers, storage, networking,

and GPUs. This study examines the on-site research infrastruc-

ture and HPC system design used to find new materials. This

novel material modeling use cases. The study facility’s HPC

system is designed to handle high-performance computing

workloads needed for material modeling and discovery. The

computing power, large memory capacity, scalability, and fast

connectivity [Fig 2]:

all system resources, including compute units, GPUs, storage,

and networking, and schedules jobs. Dell EMC PowerEdge

R740 servers with Intel-Xeon Silver 4210 processors are this

system’s backbone servers. With 10 cores and 20 threads,

this processor can handle large-scale machine learning tasks.

control. The Dell EMC PowerEdge R740 server has a 2.2

GHz Intel Xeon Silver 4210 processor that can turbo boost

to 3.2 GHz. It has 10 cores and 20 threads. CPU IPC is 2.

One CPU per unit. The 24-DIMM system has 3 terabytes of

core. The Dell EMC PowerEdge C6525 server with 4

AMD-EPYC-7742 processors has a total of 256 cores .

CPU benchmark result. [10]

(HPC) system uses deep learning-optimized NVIDIA-A100-

Tensor-Core [11] Graphics Processing Units to accelerate the

research facility’s varied Machine Learning (ML) workloads.

These GPUs can speed up Machine Learning (ML) workloads,

improving system efficiency. Tensor Cores in NVIDIA A100

GPUs speed matrix calculations, making them ideal for Ma-

chine Learning (ML) workloads. It has 6,912 CUDA cores and

40 GB of HBM2 RAM with 1.6 TB/s bandwidth. The GPU

operates at 1.41 GHz and 1.54 GHz. The Ampere architecture

datasets in the HPC system. PowerScale’s OneFS storage

high-capacity, high-performance storage for the system. Dell

EMC Isilon scale-out Network Attached Storage [3] is ideal

for machine learning (ML) tasks due to its high throughput and

IOPS. Dell EMC PowerScale OneFS can store machine learn-

ing data. The device can handle 50 petabytes of data. It also

provides a machine-learning-optimized computing platform.

OneFS supports NFS, SMB, and HDFS. It also compresses,

deletes, and protects data.

connects the HPC system’s components quickly. This network-

ing solution optimizes data transfer between the infrastructure

server, compute nodes, GPUs, and storage components, im-

proving system efficiency. Mellanox InfiniBand HDR network-

ing technology has 200Gb/s HDR adapters with low latency

has peak node efficiency of CPU Speed (2.2)x No. of CPU

cores(10)x CPU IPC (2) x No. of CPU node (2) = 88 GFLOPS

IPC (2) x No. of CPU node (2) = 576 GFLOPS X 100

(nodes) = 57600 GFLOPS The processing capability of the

NVIDIA A100 Tensor Core GPU is reported to be up to 19.5

TFLOPS. We can assume that the storage and networking

the system. The computation of the overall TFLOPS capacity

TFLOPS) x 50 (GPU Nodes) = 975 TFLOPS Total Peak

performance of existing HPC = 975 + 57.688 TFLOPS =

1032.688 = 1.032 PFLOPS The High-Performance Computing

(HPC) architecture employed for the purpose of material

discovery involves an infrastructure node, 100 compute nodes,

combined capacity of the system is 1.032 PFLOPS

allocation and eliminating the need for dedicated hardware

hances resource allocation. Virtualization allows managers to

quickly allocate and deallocate virtual machines, improving

HPC ecosystem adaptability and responsiveness. By using

containers for each task, virtualization can improve security

and isolation across high-performance computing workloads.

The above strategy reduces security breaches and workload

interference.

computing environment for virtualized tasks. These include:

allowing workload management across VMs and con-

tainers. It also supports simulated NVIDIA graphics

processing units for data analysis and AI.

It provides a highly available, easy-to-manage hyper-

native software for Artificial Intelligence and data analyt-

ics by NVIDIA for operation solely on VMware vSphere

systems. The Tanzu platform includes the NVIDIA GPU

Operator and numerous AI and data science frameworks

tailored to its needs.Perpetual licenses allow unlimited

Enterprise requires perpetual licenses and 1-, 3-, or 5-

year support packages. [16]

• NVIDIA GPU Operator and Network Operator simplify

worker node provisioning of NVIDIA drivers and the

Container Runtime. These operators setup ConnectX net-

• The Tanzu Ecosystem encompasses a variety of essen-

tial components necessary for Tanzu Kubernetes clus-

ters, including but not limited to Harbor, Prometheus,

container image storage, metrics gathering and analy-

sis, and multicloud load balancing, respectivelyHarbor,

tem, which supports Kubernetes groups. These compo-

nents store container images, gather and analyze data,

and balance multicloud loads. Container entry is provided

by the NSX Advanced Load Balancer (Avi) [14] [17] .

Basic, Standard, and Advanced variants are available.

Fig. 3. Proposed Virtualized Peta Scale Architecture

C. Architecture Components:

A research center will use VMware and NVIDIA. VMware

vSphere, Tanzu, and NVIDIA AI Enterprise manage Kuber-

netes container groups and conventional virtual machines on

pre-existing servers with few new additions using the above

architectural design. Virtualization allows full lifecycle control

of computing and storage resources. [Fig 3]

1) Infrastructure/Management Node: : The Management

management containers and nodes. The management cluster

pre-existing management cluster was capacity analyzed and

adjusted to support HPC management components.

single points of failure, VMware vSphere Enterprise Plus

to advanced features.

workloads on compute clusters. VMware’s vSphere Scale-Out

computing tasks at a low cost. The researchers can use the

groups to be managed by Tanzu Mission Control. NVIDIA

PowerEdge servers.

servers can replace many CPU servers by delegating compu-

tationally intensive jobs to them. VMware vSphere configures

virtual machine storage. vSAN’s CSI driver automatically

creates a preset Storage Class. vSAN persistently stores pods

and Docker files. PowerScale can act as an NFS cache for large

datasets from external object storage, saving time. Kubernetes

for neural network training on a single node with two GPUs.

GPU partitioning-based model development and inference are

worker pods on Kubernetes worker nodes with CPU and GPU

specification limit pods to specific nodes. CPU, GPU model

creation, GPU training, and GPU inference worker pods exist.

Each division has a node pool and worker nodes. To maxi-

memory, and GPU allocations. Resource management and pod

assignment are improved. We propose a Tanzu Kubernetes

GPU resources to Virtual Machine (VM) classes by specifying

their memory sizes.

building machine learning models, fast proofs of concept,

and pipelines. GPUs are fully allocated in pods tied to high-

performance worker nodes to train complex CNN models that

have the requirement of accelerated architectural components

and scalable performance. Ansys Rocky Simulation illustrates

such tasks. GPU inference pods tied to small GPU MIG

partitions simplify model deployment and prediction API

endpoints. [4]

D. Theoretical peak performance Calculation

The proposed system comprises of a 5 x infrastructure node,

(2) = 88 GFLOPS + 4x the configuration comprises of Dell-

EMC-PowerEdge-R940 servers, each equipped with four Intel

Xeon Platinum 8380 processors. CPU Speed (2.3)x No. of

CPU cores(40)x CPU IPC (2) x No. of CPU node (4)= 2944

GFLOPS Total= 88GFLOPS + 2944GFLOPS =3032 GFLOPS

2) Theoretical peak performance Calculation of GPU and

Worker Node: 100 x Dell EMC PowerEdge C6525 servers,

each equipped with an AMD EPYC 7742 Processor, has

been initiated that exhibit processing capabilities as calculated

CPU Speed (2.4)x No. of CPU cores(64)x CPU IPC (2)

x No. of CPU node (4) = 1228.8 GFLOPS x 400 (nodes) =

As the proposed system includes 50 x NVIDIA A100 Tensor

Core GPUs and 100 x DGX A100s. The processing capability

of the NVIDIA DGX A100s is reported to be up to 39

TFLOPS [23]. We can assume that the storage and networking

components do not contribute to the TFLOPS capacity of

the system. 50 x 19.5 TFLOPS + 100 x 39TFLOPS = 975

TFLOPS + 3900 TFLOPS

Total Peak performance of existing HPC = 975 + 3900

TFLOPS = 4875 TFLOPS

(HPC)system involves 5 infrastructure node, 500 compute

the fraction of a program that cannot be executed in parallel

limits its possible acceleration.Amdahl’s law’s math is:

erEdge C6525 servers with an AMD EPYC 7742 Processor,

while in the new architecture, the compute nodes have 100

Dell EMC PowerEdge C6525 servers with an AMD EPYC

with 4 AMD EPYC 7763 Processors. This represents a 4x

of speedup can be determined through the application of

Amdahl’s law.

problem size with the addition of processors, resulting in a

and the serial proportion as 0.1, the speedup can be computed

utilizing Gustafson’s law.

450.01

potential for speedup than the old architecture when the

problem size is scaled up.

that the novel architecture possesses the capability to achieve

been specifically crafted to facilitate the processing of Material

Modelling AI workloads, all the while incorporating ethical

and data governance factors. The utilization of virtualization

and Tanzu Kubernetes clusters can potentially offer enhanced

scalability and redundancy. Furthermore, the incorporation of

parallelism, thereby supporting expedited processing of AI

workloads.

• The utilization of virtualization in facilitating artificial in-

telligence workloads has the potential to improve security

and isolation.

• The utilization of virtualization technology enables the

efficient management of workloads through the dynamic

allocation of resources.

• The architecture that has been implemented recently has

been designed to enhance scalability and redundancy.

This has been achieved by integrating additional compute

nodes and GPUs, which enables the system to effec-

hardware has the potential to enhance the deployment and

administration of the underlying infrastructure, thereby

optimizing the overall process.

• The proposed architecture aims to improve the processing

of AI workloads by incorporating additional processors,

cores, and nodes to enable increased parallelism. This

enhancement is expected to result in expedited processing

of AI workloads with improved speedup.

• The implementation of a novel architecture with supple-

mentary redundancy can significantly reduce downtime

I have included some common cost components such as

hardware, maintenance, electricity, cooling, network, software

licenses, and personnel. The total cost is calculated for a 3-

year, 5-year, and 7-year period, and the cost per core per year

is also calculated for comparison purposes.

For the old infrastructure, I assumed a hardware cost of

is assumed to be 1percent of the hardware cost per year, the

cooling cost is assumed to be 50,000 euros per year, the

network cost is assumed to be 150,000 euros per year, the

software licenses cost is assumed to be 300,000 euros per year,

Fig. 4. TCO Analysis

and the personnel cost is assumed to be 500,000 euros per year.

million euros, which includes the Dell EMC PowerEdge R740

infrastructure server, 4 Dell EMC PowerEdge R940 servers,

100 Dell EMC PowerEdge C6525 compute nodes, 400 Dell

EMC PowerEdge C6525 compute nodes, 50 NVIDIA A100

Tensor Core GPUs, and 350 NVIDIA DGX A100 GPUs with

network cost is assumed to be 250,000 euros per year, the

old infrastructure, which indicates that the new infrastructure

is more cost-efficient in terms of processing power. One

interesting aspect of the TCO analysis is that despite the

significant increase in the number of compute nodes and

GPUs in the new infrastructure, the total cost of ownership

is lower compared to the old infrastructure. This is due to

several factors such as the more efficient hardware and the use

of open-source software components which reduces licensing

costs. Another interesting aspect is the impact of workload

on TCO. In the example provided, the workload was assumed

be higher due to the need for additional compute nodes or

is essential to consider workload variability when evaluating

the TCO of different infrastructure options.

Upon contemplation of this undertaking, I have acquired

knowledge and discerned the advantages of employing vir-

tualization technology within High-Performance Computing

(HPC) settings. Research institutions can enhance the effi-

(HPC) environments, encompassing the diverse software of-

ferings within the Tanzu ecosystem. Furthermore, an inquiry

into the potential utilization of virtualization to augment the

security and governance of the High-Performance Computing

(HPC) infrastructure is warranted. Additionally, alternative

methods to amplify resilience and redundancy should be

explored.

With additional time, I would expand upon this project

by incorporating heightened security measures and isolation

protocols, including the utilization of distinct containers for

enhancing the scalability and fault tolerance of the virtualized

High-Performance Computing (HPC) infrastructure. One such

approach could involve the automation of workflow restarts

on alternative physical servers situated within the cluster.

In summary, the advantages of virtualization technology

in enhancing productivity, efficacy, and security within HPC

settings are acknowledged by myself as an HPC administrator.

The implementation of virtualization technology within High

Performance Computing (HPC) settings has enabled a level of

scalability and fault tolerance that was previously unachievable

[1] Imran, Qayyum F, Kim DH, Bong SJ, Chi SY, Choi YH. A Survey of

Datasets, Preprocessing, Modeling Mechanisms, and Simulation Tools

Based on AI for Material Analysis and Discovery. Materials (Basel).

2022 Feb 15;15(4):1428. doi: 10.3390/ma15041428. PMID: 35207968;

PMCID: PMC8875409.

[2] Design guide-virtualizing gpus for AI with vmware and Nvidia based

on Dell Infrastructure (no date) Dell Technologies Info Hub. Avail-

gpus-for-ai-with-vmware-and-nvidia-based-on-dell-infrastructure-1.

[3] Unstructured data – Data Storage (no date) – Data Storage —

Dell Canada. Available at: https://www.dell.com/en-ca/dt/learn/data-

[4] Dunn, A. et al. (2020) Benchmarking materials property predic-

https://www.nature.com/articles/s41524-020-00406-3.

chine learning models of ..... Available at:

[9] CPU Benchmark (2023) Intel Xeon Silver 4210 - benchmark, test and

[11] Nvidia A100 gpus power the modern data center (no date) NVIDIA.

infiniband-solutions.pdf, https://network.nvidia.com/files/doc-2020/ocp-

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/certification/vmw-

with-tanzu/GUID-70CAF0BB-1722-4526-9CE7-D5C92C15D7D0.html

[16] Nvidia AI Enterprise Licensing and packaging guide (no date)

[21] Poweredge C6525 - Dell (no date). Available at: