Overview of Red Hat's Multi Cloud Gateway (Noobaa)

OpenShift

- Toni Schmidbauer Toni Schmidbauer ( Lastmod: 2024-05-05 ) - 5 min read

This is my personal summary of experimenting with Red Hat's Multi Cloud Gateway (MCG) based on the upstream Noobaa project. MCG is part of Red Hat's OpenShift Data Foundation (ODF). ODF bundles the upstream projects Ceph and Noobaa.

Overview

Noobaa, or the Multicloud Gateway (MCG), is a S3 based data federation tool. It allows you to use S3 backends from various sources and

  • sync
  • replicate
  • or simply use existing

S3 buckets. Currently the following sources, or backing stores are supported:

  • AWS S3
  • Azure Blob
  • Google Cloud Storage

  • Any other S3 compatible storage, for example

    • Ceph
    • Minio

Noobaa also supports using a local file system as a backing store for S3.

The main purpose is to provide a single API endpoint for applications using various S3 backends.

One of the main features of Noobaa is the storage pipeline. With a standard Noobaa S3 bucket, when storing a new Object Noobaa executes the following steps:

  • Chunking of the Object
  • De-duplication
  • Compression
  • and Encryption

This means that data stored in public cloud S3 offerings is automatically encrypted. Noobaa also supports using Hashicorp Vault for storing and retrieving encryption keys.

If you need to skip the storage pipeline, Noobaa also supports namespace buckets. For example these type of buckets allow you to write directly to AWS S3 and retrieve Objects via Noobaa. Or it could be used to migrate buckets from one cloud provider to another.

Noobaa also has support for triggering JavaScript based function when

  • creating new objects
  • reading existing objects
  • deleting objects

Setup

With OpenShift Plus or an OpenShift Data Foundation subscription you can use the OpenShift Data Foundation Operator.

For testing Noobaa we used the standalone installation method without setting up Ceph storage (see here). OpenShift was running in AWS for testing.

If you would like to use the upstream version you can use the Noobaa operator (https://github.com/noobaa/noobaa-operator). This is what the OpenShift Data Foundation (ODF) is using as well.

Command line interface

Noobaa also comes with a command line interface noobaa. It's available via an ODF subscription or can be installed separately. See the noobaa-operator readme for more information.

Resources

Before using an S3 object store with Noobaa we need to create so called Resources. This can be done via the Noobaa user interface or via the command line. For example the following commands create a new Resource using an AWS S3 bucket as a backing store

# create an S3 bucket in eu-north-1
aws s3api create-bucket \
    --region eu-north-1 \
    --bucket tosmi-eu-north-1 \
    --create-bucket-configuration LocationConstraint=eu-north-1

# create an S3 bucket in eu-north-1
aws s3api create-bucket \
    --region eu-west-1 \
    --bucket tosmi-eu-west-1 \
    --create-bucket-configuration LocationConstraint=eu-west-1

# create Noobaa backing store using the tosmi-eu-north-1 bucket above
noobaa backingstore create aws-s3 \
       --region eu-north-1 \
       --target-bucket tosmi-eu-north-1 aws-eu-north

# create Noobaa backing store using the tosmi-eu-west-1 bucket above
noobaa backingstore create aws-s3 \
       --region eu-west-1 \
       --target-bucket tosmi-eu-west-1 aws-eu-west

Or if we would like to use Azure blob

# create two resource groups for storage
az group create --location northeurope -g mcg-northeurope

# create two storage accounts
az storage account create --name mcgnortheurope -g mcg-northeurope --location northeurope --sku Standard_LRS --kind StorageV2

# create containers for storing blobs
az storage container create --account-name mcgnortheurope -n mcg-northeurope

# list storage account keys for noobaa
az storage account list
az storage account show -g mcg-northeurope -n mcgnortheurope
az storage account keys list -g mcg-westeurope -n mcgwesteurope
az storage account keys list -g mcg-northeurope -n mcgnortheurope

noobaa backingstore create \
       azure-blob azure-northeurope \
       --account-key="<the key>" \
       --account-name=mcgnortheurope \
       --target-blob-container=mcg-northeurope

Using

noobaa backingstore list

we are able to confirm that our stores were created successfully.

Buckets

After creating the backend stores we are able to create Buckets and define the layout of backends.

There are two ways how to create buckets, either directly via the Noobaa UI, or using Kubernetes (K8s) objects.

We will focus on using K8s objects in this post.

Required K8s objects

The Noobaa operator provides the following Custom Resource Definitions:

  • BackingStore: we already created BackingStores in the Resources section
  • BucketClass: a bucket class defines the layout of our bucket (single, mirrored or tiered)
  • StorageClass: a standard K8s StorageClass referencing the BucketClass
  • ObjectBucketClaim: A OBC or ObjectBucketClaim creates the bucket for us in Noobaa. Additionally the Noobaa operator creates a ConfigMap and a Secret with the same name as the Bucket, storing access details (ConfigMap) and credentials (Secret) for accessing the bucket.

BucketClass

Let's create a example BucketClass which mirrors objects between the AWS S3 buckets eu-west-1 and eu-north-1.

apiVersion: noobaa.io/v1alpha1
kind: BucketClass
metadata:
  labels:
    app: noobaa
  name: aws-mirrored-bucket-class
  namespace: openshift-storage
spec:
  placementPolicy:
    tiers:
    - backingStores:
      - aws-eu-north
      - aws-eu-west
      placement: Mirror

So we are defining a BucketClass aws-mirrored-bucket-class that has the following placement policy:

  • A single tier with one backing store

  • The backing store uses two AWS buckets

    • aws-eu-north
    • aws-eu-west
  • The placement policy is mirror, so all objects uploaded to buckets using this BucketClass will be mirrored between aws-eu-north and aws-eu-west.

A BucketClass could have multiple tiers, moving cold data transparently to a lower tier, but let's keep this simple.

StorageClass

After creating our BucketClass we are now able to define a standard K8s StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    description: Provides Mirrored Object Bucket Claims (OBCs) in AWS
  name: aws-mirrored-openshift-storage.noobaa.io
parameters:
  bucketclass: aws-mirrored-bucket-class
provisioner: openshift-storage.noobaa.io/obc
reclaimPolicy: Delete
volumeBindingMode: Immediate

This StorageClass uses our BucketClass aws-mirrored-bucket-class as a backend. All buckets created leveraging this StorageClass will mirror data between aws-eu-north and aws-eu-west (see the previous chapter).

ObjectBucketClaim

Finally we are able to create ObjectBucketClaims for projects requiring object storage. An ObjectBucketClaim is similar to an PersistentVolumeClaim. Every time a claim is created the Noobaa operator will create a corresponding S3 bucket for us.

Let's start testing this out by creating a new OpenShift project

oc new-project obc-test

Now we define a ObjectBucketClaim to create a new bucket for our application:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  labels:
    app: noobaa
  name: aws-mirrored-claim
spec:
  generateBucketName: aws-mirrored
  storageClassName: aws-mirrored-openshift-storage.noobaa.io

We use the StorageClass created in the previous step. This will create

  • a S3 Bucket in the requested StorageClass
  • a ConfigMap storing access information
  • a Secret storing credentials for accessing the S3 Bucket

For testing we will upload some data via s3cmd and use a pod to monitor data within the bucket.

Let's do the upload with s3cmd, we need the following config file:

[default]
check_ssl_certificate = False
check_ssl_hostname = False
access_key = <access key>
secret_key = <secret key>
host_base = s3-openshift-storage.apps.ocp.aws.tntinfra.net
host_bucket = %(bucket).s3-openshift-storage.apps.ocp.aws.tntinfra.net

Of course you must change host-base according to your cluster name. It's a route in the openshift-storage namespace:

oc get route -n openshift-storage s3 -o jsonpath='{.spec.host}'

You can extract the access and secret key from the K8s secret via:

oc extract secret/aws-mirrored-claim --to=-

Copy the access key and the secret key to the s3 command config file (we've called our config noobaa-s3.cfg). Now we can list all available buckets via:

$ s3cmd ls -c noobaa-s3.cfg
2022-04-22 13:56  s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4

Now we are going to upload a sample file:

$ s3cmd -c noobaa-s3.cfg put simple-aws-mirrored-obc.yaml s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4
upload: 'simple-aws-mirrored-obc.yaml' -> 's3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4/simple-aws-mirrored-obc.yaml'  [1 of 1]
 226 of 226   100% in    0s   638.18 B/s  done

We can also list available files via

s3cmd -c noobaa-s3.cfg ls s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4
2022-04-22 13:57          226  s3://aws-mirrored-c1087a17-5c84-4c62-9f36-29081a6cf5a4/simple-aws-mirrored-obc.yaml

Our we could use a Pod to list available files from within OpenShift. Note how we use the ConfigMap and the Secret the Noobaa operater created for us, when we created the ObjectBucketClaim:

apiVersion: batch/v1
kind: Job
metadata:
  name: s3-test-job
spec:
  template:
    metadata:
      name: s3-pod
    spec:
      containers:
      - image: d3fk/s3cmd:latest
        name: s3-pod
        env:
        - name: BUCKET_NAME
          valueFrom:
            configMapKeyRef:
              name: aws-mirrored-claim
              key: BUCKET_NAME
        - name: BUCKET_HOST
          valueFrom:
            configMapKeyRef:
              name: aws-mirrored-claim
              key: BUCKET_HOST
        - name: BUCKET_PORT
          valueFrom:
            configMapKeyRef:
              name: aws-mirrored-claim
              key: BUCKET_PORT
        - name: AWS_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              name: aws-mirrored-claim
              key: AWS_ACCESS_KEY_ID
        - name: AWS_SECRET_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: aws-mirrored-claim
              key: AWS_SECRET_ACCESS_KEY
        command:
        - /bin/sh
        - -c
        - 's3cmd --host $BUCKET_HOST --host-bucket "%(bucket).$BUCKET_HOST" --no-check-certificate ls s3://$BUCKET_NAME'
      restartPolicy: Never

That's all for now. If time allows we are going to write a follow up blog post on

  • Replicating Buckets and
  • Functions

See Also:

  • Introducing AdminNetworkPolicies –

    Classic Kubernetes/OpenShift offer a feature called NetworkPolicy that allows users to control the traffic to and from their assigned Namespace. NetworkPolicies are designed to give project owners or tenants the ability to protect their own namespace. Sometimes, however, I worked with customers where the cluster administrators or a dedicated (network) team need to enforce these policies. Since the NetworkPolicy API is namespace-scoped, it is not possible to enforce policies across namespaces. The only solution was to create custom (project) admin and edit roles, and remove the ability of creating, modifying or deleting NetworkPolicy objects. Technically, this is possible and easily done. But shifts the whole network security to cluster administrators. Luckily, this is where AdminNetworkPolicy (ANP) and BaselineAdminNetworkPolicy (BANP) comes into play.

  • Using Kustomize to post render a Helm Chart –

    Lately I came across several issues where a given Helm Chart must be modified after it has been rendered by Argo CD. Argo CD does a helm template to render a Chart. Sometimes, especially when you work with Subcharts or when a specific setting is not yet supported by the Chart, you need to modify it later …​ you need to post-render the Chart. In this very short article, I would like to demonstrate this on a real-live example I had to do. I would like to inject annotations to a Route objects, so that the certificate can be injected. This is done by the cert-utils operator. For the post-rendering the Argo CD repo pod will be extended with a sidecar container, that is watching for the repos and patches them if required.

  • Managing Certificates using GitOps approach –

    The article SSL Certificate Management for OpenShift on AWS explains how to use the Cert-Manager Operator to request and install a new SSL Certificate. This time, I would like to leverage the GitOps approach using the Helm Chart cert-manager I have prepared to deploy the Operator and order new Certificates. I will use an ACME Letsencrypt issuer with a DNS challenge. My domain is hosted at AWS Route 53. However, any other integration can be easily used.

  • Update Cluster Version using GitOps approach –

    During a GitOps journey at one point, the question arises, how to update a cluster? Nowadays it is very easy to update a cluster using CLI or WebUI, so why bother with GitOps in that case? The reason is simple: Using GitOps you can be sure that all clusters are updated to the correct, required version and the version of each cluster is also managed in Git. All you need is the channel you want to use and the desired cluster version. Optionally, you can define the exact image SHA. This might be required when you are operating in a restricted environment.

  • Multiple Sources for Applications in Argo CD –

    Argo CD or OpenShift GitOps uses Applications or ApplicationSets to define the relationship between a source (Git) and a cluster. Typically, this is a 1:1 link, which means one Application is using one source to compare the cluster status. This can be a limitation. For example, if you are working with Helm Charts and a Helm repository, you do not want to re-build (or re-release) the whole chart just because you made a small change in the values file that is packaged into the repository. You want to separate the configuration of the chart with the Helm package. The most common scenarios for multiple sources are (see: Argo CD documentation): Your organization wants to use an external/public Helm chart You want to override the Helm values with your own local values You don’t want to clone the Helm chart locally as well because that would lead to duplication and you would need to monitor it manually for upstream changes. This small article describes three different ways with a working example and tries to cover the advantages and disadvantages of each of them. They might be opinionated but some of them proved to be easier to use and manage.

Copyright © 2020 - 2024 Toni Schmidbauer & Thomas Jungbauer