How to use Kubeflow and the MPI Operator on OpenShift

This repository provides an example of using Kubeflow and its MPI operator on top of OpenShift. The examples specifically target running OpenFOAM CFD simulations with CPUs and GPUs. It uses OpenShift Data Foundation for providing RWX storage.

Background

The growing adoption of Kubernetes provides a new opportunity to shed legacy HPC infrastructures. Kubernetes is effectively a general purpose scheduling system for containers. As many MPI-based workloads are already written on Linux, they can be easily containerized. The Kubeflow project has an early-stage operator that handles MPI applications.

OpenFOAM is an application suite used for computational fluid dynamics (CFD) analysis. It is capable of processing large jobs in parallel using MPI. These jobs frequently involve large numbers of processors (CPUs). For an organization with a large Kubernetes cluster at its disposal, making use of these large processor pools to perform MPI jobs in their spare time seems logical. Further, in cloud-based environments like AWS (where this example is designed to run), organizations can make use of the autoscaling features inside Kubernetes to simply create the capacity required to fulfill the MPI job’s requirements.

Deeper descriptions of MPI, workers, processors, and etc. is outside of the scope of this example. Knowledge of CFD and OpenFOAM is also outside of the scope of this example. Some understanding of both MPI and OpenFOAM is assumed, but not required. A solid grasp of Kubernetes and, to a degree, OpenShift, is assumed.

Learn more: https://cloud.redhat.com/blog/how-to-use-kubeflow-and-the-mpi-operator-on-openshift

--

--

--

CLOUD TECHNOLOGY STRATEGY ADVISORY

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Why You Should Be Using Django

Getting into Web Hosting Environments — Static, Dynamic and Local Hosting

Database introduction

Modernizing and scaling our authentication with Firebase

How to swap two numbers without using a temporary variable

CS373 Spring 2021: Jack Raney (Week 10)

Improving Fault-Tolerance of System in Micro-services Environment.

Traces of the Icebreakers — hidden solutions of a vanilla Dynamics version 9

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
cloudstrata GmbH

cloudstrata GmbH

CLOUD TECHNOLOGY STRATEGY ADVISORY

More from Medium

Using the Apache APISIX proxy gRPC service

Simple Message Transforms with Kafka Connect

Avoid These Mistakes When Deploying EdTech in a Public Cloud

How to Bring your own Scheduler into OpenShift with the Secondary Scheduler Operator