ocp | Spagno's Blog

OCS 4.2 in OCP 4.2.14 - UPI installation in RHV

Tue, 28 Jan 2020 00:00:00 +0000

When OCS 4.2 GA was released days ago, I was thrilled to finally test and deploy it in my lab. I read the documentation and saw that only vSphere and AWS installations are currently supported. My lab is installed in an RHV environment following the UPI Bare Metal documentation so, in the beginning, I was a bit disappointed. I realized that it could be an interesting challenge to find a different way to use it and, well, I found it during my day by day late night fun. All the following procedures are unsupported.

Prerequisites

An OCP 4.2.x cluster installed (the current latest version is 4.2.14)
The possibility to create new local disks inside the VMs (if you are using a virtualized environment) or servers with disks that can be used

Issues

The official OCS 4.2 installation in vSphere requires a minimum of 3 nodes which use 2TB volume each (a PVC using the default “thin” storage class) for the OSD volumes + 10GB for each mon POD (3 in total using always a PVC). It also requires 16 CPU and 64GB RAM for node.

Use case scenario

bare-metal installations
vSphere cluster
- without a shared datastore
- you don’t want to use the vSphere dynamic provisioner
- without enough space in the datastore
- without enough RAM or CPU
other virtualized installation (for example RHV which is the one used for this article)

Challenges

create a PVC using local disks
change the default 2TB volumes size
define a different StorageClass (without using a default one) for the mon PODs and the OSD volumes
define different limits and requests per component

Solutions

use the local storage operator
create the ocs-storagecluster resource using a YAML file instead of the new interface. That means also add the labels to the worker nodes that are going to be used by OCS

Procedures

Add the disks in the VMs. Add 2 disks for each node. 10GB disk for mon POD and 100GB disk for the OSD volume

Create 10GB disk

Create 100GB disk

Repeat for the other 2 nodes

The disks MUST be in the same order and have the same device name in all the nodes. For example, /dev/sdb MUST be the 10GB disk and /dev/sdc the 100GB disk in all the nodes.

[root@utility ~]# for i in {1..3} ; do ssh core@worker-${i}.ocp42.ssa.mbu.labs.redhat.com lsblk | egrep "^sdb.*|sdc.*$" ; done
sdb      8:16   0   10G  0 disk
sdc      8:32   0  100G  0 disk
sdb      8:16   0   10G  0 disk
sdc      8:32   0  100G  0 disk
sdb      8:16   0   10G  0 disk
sdc      8:32   0  100G  0 disk
[root@utility ~]#

Install the Local Storage Operator. Here the official documentation

Create the namespace

[root@utility ~]# oc new-project local-storage‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Then install the operator from the OperatorHub

LSO Operator

Wait for the operator POD up&running

[root@utility ~]# oc get pod -n local-storage
NAME                                     READY   STATUS    RESTARTS   AGE
local-storage-operator-ccbb59b45-nn7ww   1/1     Running   0          57s
[root@utility ~]#

The Local Storage Operator works using the devices as reference. The LocalVolume resource scans the nodes which match the selector and creates a StorageClass for the device.

Do not use different StorageClass names for the same device.

We need the Filesystem type for these volumes. Prepare the LocalVolume YAML file to create the resource for the mon PODs which use /dev/sdb

[root@utility ~]# cat <<EOF > local-storage-filesystem.yaml
apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
  name: "local-disks-fs"
  namespace: "local-storage"
spec:
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-1.ocp42.ssa.mbu.labs.redhat.com
          - worker-2.ocp42.ssa.mbu.labs.redhat.com
          - worker-3.ocp42.ssa.mbu.labs.redhat.com
  storageClassDevices:
    - storageClassName: "local-sc"
      volumeMode: Filesystem
      devicePaths:
        - /dev/sdb
EOF

Then create the resource

[root@utility ~]# oc create -f local-storage-filesystem.yaml
localvolume.local.storage.openshift.io/local-disks-fs created
[root@utility ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Check if all the PODs are up&running and if the StorageClass and the PVs exist

[root@utility ~]# oc get pod -n local-storage
NAME                                     READY   STATUS    RESTARTS   AGE
local-disks-fs-local-diskmaker-2bqw4     1/1     Running   0          106s
local-disks-fs-local-diskmaker-8w9rz     1/1     Running   0          106s
local-disks-fs-local-diskmaker-khhm5     1/1     Running   0          106s
local-disks-fs-local-provisioner-g5dgv   1/1     Running   0          106s
local-disks-fs-local-provisioner-hkj69   1/1     Running   0          106s
local-disks-fs-local-provisioner-vhpj8   1/1     Running   0          106s
local-storage-operator-ccbb59b45-nn7ww   1/1     Running   0          15m
[root@utility ~]# oc get sc
NAME       PROVISIONER                    AGE
local-sc   kubernetes.io/no-provisioner   109s
[root@utility ~]# oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
local-pv-68faed78   10Gi       RWO            Delete           Available           local-sc                84s
local-pv-780afdd6   10Gi       RWO            Delete           Available           local-sc                83s
local-pv-b640422f   10Gi       RWO            Delete           Available           local-sc                9s
[root@utility ~]#

The PVs were created.

Prepare the LocalVolume YAML file to create the resource for the OSD volumes which use /dev/sdc

We need the Block type for these volumes.

[root@utility ~]# cat <<EOF > local-storage-block.yaml
apiVersion: "local.storage.openshift.io/v1"
kind: "LocalVolume"
metadata:
  name: "local-disks"
  namespace: "local-storage"
spec:
  nodeSelector:
    nodeSelectorTerms:
    - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - worker-1.ocp42.ssa.mbu.labs.redhat.com
          - worker-2.ocp42.ssa.mbu.labs.redhat.com
          - worker-3.ocp42.ssa.mbu.labs.redhat.com
  storageClassDevices:
    - storageClassName: "localblock-sc"
      volumeMode: Block
      devicePaths:
        - /dev/sdc
EOF

Then create the resource

[root@utility ~]# oc create -f local-storage-block.yaml
localvolume.local.storage.openshift.io/local-disks created
[root@utility ~]#

Check if all the PODs are up&running and if the StorageClass and the PVs exist

[root@utility ~]# oc get pod -n local-storage
NAME                                     READY   STATUS    RESTARTS   AGE
local-disks-fs-local-diskmaker-2bqw4     1/1     Running   0          6m33s
local-disks-fs-local-diskmaker-8w9rz     1/1     Running   0          6m33s
local-disks-fs-local-diskmaker-khhm5     1/1     Running   0          6m33s
local-disks-fs-local-provisioner-g5dgv   1/1     Running   0          6m33s
local-disks-fs-local-provisioner-hkj69   1/1     Running   0          6m33s
local-disks-fs-local-provisioner-vhpj8   1/1     Running   0          6m33s
local-disks-local-diskmaker-6qpfx        1/1     Running   0          22s
local-disks-local-diskmaker-pw5ql        1/1     Running   0          22s
local-disks-local-diskmaker-rc5hr        1/1     Running   0          22s
local-disks-local-provisioner-9qprp      1/1     Running   0          22s
local-disks-local-provisioner-kkkcm      1/1     Running   0          22s
local-disks-local-provisioner-kxbnn      1/1     Running   0          22s
local-storage-operator-ccbb59b45-nn7ww   1/1     Running   0          19m
[root@utility ~]# oc get sc
NAME            PROVISIONER                    AGE
local-sc        kubernetes.io/no-provisioner   6m36s
localblock-sc   kubernetes.io/no-provisioner   25s
[root@utility ~]# oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS    REASON   AGE
local-pv-5c4e718c   100Gi      RWO            Delete           Available           localblock-sc            10s
local-pv-68faed78   10Gi       RWO            Delete           Available           local-sc                 6m13s
local-pv-6a58375e   100Gi      RWO            Delete           Available           localblock-sc            10s
local-pv-780afdd6   10Gi       RWO            Delete           Available           local-sc                 6m12s
local-pv-b640422f   10Gi       RWO            Delete           Available           local-sc                 4m58s
local-pv-d6db37fd   100Gi      RWO            Delete           Available           localblock-sc            5s
[root@utility ~]#

All the PVs were created.

Install OCS 4.2. Here the official documentation

Create the namespace “openshift-storage”

[root@utility ~]# cat <<EOF > ocs-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: openshift-storage
  labels:
    openshift.io/cluster-monitoring: "true"
EOF
[root@utility ~]# oc create -f ocs-namespace.yaml
namespace/openshift-storage created
[root@utility ~]#

Add the labels to the workers

oc label node worker-1.ocp42.ssa.mbu.labs.redhat.com "cluster.ocs.openshift.io/openshift-storage=" --overwrite
oc label node worker-1.ocp42.ssa.mbu.labs.redhat.com "topology.rook.io/rack=rack0" --overwrite
oc label node worker-2.ocp42.ssa.mbu.labs.redhat.com "cluster.ocs.openshift.io/openshift-storage=" --overwrite
oc label node worker-2.ocp42.ssa.mbu.labs.redhat.com "topology.rook.io/rack=rack1" --overwrite
oc label node worker-3.ocp42.ssa.mbu.labs.redhat.com "cluster.ocs.openshift.io/openshift-storage=" --overwrite
oc label node worker-3.ocp42.ssa.mbu.labs.redhat.com "topology.rook.io/rack=rack3" --overwrite‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Install the operator from the web interface

OCS Operator

Check on the web interface if the operator is Up to date

Installed Operators

And wait for the PODs up&running

[root@utility ~]# oc get pod -n openshift-storage
NAME                                  READY   STATUS    RESTARTS   AGE
noobaa-operator-85d86479fc-n8vp5      1/1     Running   0          106s
ocs-operator-65cf57b98b-rk48c         1/1     Running   0          106s
rook-ceph-operator-59d78cf8bd-4zcsz   1/1     Running   0          106s
[root@utility ~]#

Create the OCS Cluster Service YAML file

[root@utility ~]# cat <<EOF > ocs-cluster-service.yaml
apiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-storagecluster
  namespace: openshift-storage
spec:
  manageNodes: false
  monPVCTemplate:
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: 'local-sc'
      volumeMode: Filesystem
  storageDeviceSets:
  - count: 1
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: 'localblock-sc'
        volumeMode: Block
    name: ocs-deviceset
    placement: {}
    portable: true
    replica: 3
    resources: {}
EOF

You can notice the “monPVCTemplate” section in which we define the StorageClass “local-sc” and in the section “storageDeviceSets” the different storage sizes and the StorageClass “localblock-sc” used by OSD volumes.

Now we can create the resource

[root@utility ~]# oc create -f ocs-cluster-service.yaml
storagecluster.ocs.openshift.io/ocs-storagecluster created
[root@utility ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

During the creation of the resources, we can see how the PVCs created are bounded with the Local Storage PVs

[root@utility ~]# oc get pvc -n openshift-storage
NAME              STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS   AGE
rook-ceph-mon-a   Bound    local-pv-68faed78   10Gi       RWO            local-sc       13s
rook-ceph-mon-b   Bound    local-pv-b640422f   10Gi       RWO            local-sc       8s
rook-ceph-mon-c   Bound    local-pv-780afdd6   10Gi       RWO            local-sc       3s
[root@utility ~]# oc get pv
NAME                CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                               STORAGECLASS    REASON   AGE
local-pv-5c4e718c   100Gi      RWO            Delete           Available                                       localblock-sc            28m
local-pv-68faed78   10Gi       RWO            Delete           Bound       openshift-storage/rook-ceph-mon-a   local-sc                 34m
local-pv-6a58375e   100Gi      RWO            Delete           Available                                       localblock-sc            28m
local-pv-780afdd6   10Gi       RWO            Delete           Bound       openshift-storage/rook-ceph-mon-c   local-sc                 34m
local-pv-b640422f   10Gi       RWO            Delete           Bound       openshift-storage/rook-ceph-mon-b   local-sc                 33m
local-pv-d6db37fd   100Gi      RWO            Delete           Available                                       localblock-sc            28m
[root@utility ~]#

And now we can see the OSD PVCs and the PVs bounded

[root@utility ~]# oc get pvc -n openshift-storage
NAME                      STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS    AGE
ocs-deviceset-0-0-7j2kj   Bound    local-pv-6a58375e   100Gi      RWO            localblock-sc   3s
ocs-deviceset-1-0-lmd97   Bound    local-pv-d6db37fd   100Gi      RWO            localblock-sc   3s
ocs-deviceset-2-0-dnfbd   Bound    local-pv-5c4e718c   100Gi      RWO            localblock-sc   3s‍‍‍‍‍
[root@utility ~]# oc get pv | grep localblock-sc
local-pv-5c4e718c                          100Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-2-0-dnfbd   localblock-sc                          31m
local-pv-6a58375e                          100Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-0-0-7j2kj   localblock-sc                          31m
local-pv-d6db37fd                          100Gi      RWO            Delete           Bound    openshift-storage/ocs-deviceset-1-0-lmd97   localblock-sc                          31m
[root@utility ~]#

This is the first PVC created inside the OCS cluster used by noobaa

[root@utility ~]# oc get pvc -n openshift-storage
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
db-noobaa-core-0          Bound    pvc-d8dbb86f-3d83-11ea-ac51-001a4a16017d   50Gi       RWO            ocs-storagecluster-ceph-rbd   72s‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Wait for all the PODs up&running

[root@utility ~]# oc get pod -n openshift-storage
NAME                                                              READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-2qkl8                                            3/3     Running     0          5m31s
csi-cephfsplugin-4pbvl                                            3/3     Running     0          5m31s
csi-cephfsplugin-j8w82                                            3/3     Running     0          5m31s
csi-cephfsplugin-provisioner-647cd6996c-6mw9t                     4/4     Running     0          5m31s
csi-cephfsplugin-provisioner-647cd6996c-pbrxs                     4/4     Running     0          5m31s
csi-rbdplugin-9nj85                                               3/3     Running     0          5m31s
csi-rbdplugin-jmnqz                                               3/3     Running     0          5m31s
csi-rbdplugin-provisioner-6b8ff67dc4-jk5lm                        4/4     Running     0          5m31s
csi-rbdplugin-provisioner-6b8ff67dc4-rxjhq                        4/4     Running     0          5m31s
csi-rbdplugin-vrzjq                                               3/3     Running     0          5m31s
noobaa-core-0                                                     1/2     Running     0          2m34s
noobaa-operator-85d86479fc-n8vp5                                  1/1     Running     0          13m
ocs-operator-65cf57b98b-rk48c                                     0/1     Running     0          13m
rook-ceph-drain-canary-worker-1.ocp42.ssa.mbu.labs.redhat.w2cqv   1/1     Running     0          2m41s
rook-ceph-drain-canary-worker-2.ocp42.ssa.mbu.labs.redhat.whv6s   1/1     Running     0          2m40s
rook-ceph-drain-canary-worker-3.ocp42.ssa.mbu.labs.redhat.ll8gj   1/1     Running     0          2m40s
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-d7d64976d8cm7   1/1     Running     0          2m28s
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-864fdf78ppnpm   1/1     Running     0          2m27s
rook-ceph-mgr-a-5fd6f7578c-wbsb6                                  1/1     Running     0          3m24s
rook-ceph-mon-a-bffc546c8-vjrfb                                   1/1     Running     0          4m26s
rook-ceph-mon-b-8499dd679c-6pzm9                                  1/1     Running     0          4m11s
rook-ceph-mon-c-77cd5dd54-64z52                                   1/1     Running     0          3m46s
rook-ceph-operator-59d78cf8bd-4zcsz                               1/1     Running     0          13m
rook-ceph-osd-0-b46fbc7d7-hc2wz                                   1/1     Running     0          2m41s
rook-ceph-osd-1-648c5dc8d6-prwks                                  1/1     Running     0          2m40s
rook-ceph-osd-2-546d4d77fb-qb68j                                  1/1     Running     0          2m40s
rook-ceph-osd-prepare-ocs-deviceset-0-0-7j2kj-s72g4               0/1     Completed   0          2m56s
rook-ceph-osd-prepare-ocs-deviceset-1-0-lmd97-27chl               0/1     Completed   0          2m56s
rook-ceph-osd-prepare-ocs-deviceset-2-0-dnfbd-s7z8v               0/1     Completed   0          2m56s
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-d7b4b5b6hnpr   1/1     Running     0          2m12s

Our installation is now complete and OCS fully operative.

Now we can browse the noobaa management console (for now it only works in Chrome) and create a new user to test the S3 object storage

User Page

New User

Permission

Secret Page

Get the endpoint for the S3 object server

[root@utility ~]# oc get route s3 -o jsonpath='{.spec.host}' -n openshift-storage
s3-openshift-storage.apps.ocp42.ssa.mbu.labs.redhat.com‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Test it with your preferred S3 client (I use Cyberduck in my windows desktop which I’m using to write this article)

List Buckets

Create something to check if you can write

New file

It works!

Set the ocs-storagecluster-cephfs StorageClass as the default one

[root@utility ~]# oc patch storageclass ocs-storagecluster-cephfs -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/ocs-storagecluster-cephfs patched
[root@utility ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Test the ocs-storagecluster-cephfs StorageClass adding persistent storage to the registry

 [root@utility ~]# oc edit configs.imageregistry.operator.openshift.io
storage:
  pvc:
    claim:‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Check the PVC created and wait for the new POD up&running

[root@utility ~]# oc get pvc -n openshift-image-registry
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                AGE
image-registry-storage   Bound    pvc-ba4a07c1-3d86-11ea-ad40-001a4a1601e7   100Gi      RWX            ocs-storagecluster-cephfs   12s
[root@utility ~]# oc get pod -n openshift-image-registry
NAME                                               READY   STATUS    RESTARTS   AGE
cluster-image-registry-operator-655fb7779f-pn7ms   2/2     Running   0          36h
image-registry-5bdf96556-98jbk                     1/1     Running   0          105s
node-ca-9gbxg                                      1/1     Running   1          35h
node-ca-fzcrm                                      1/1     Running   0          35h
node-ca-gr928                                      1/1     Running   1          35h
node-ca-jkfzf                                      1/1     Running   1          35h
node-ca-knlcj                                      1/1     Running   0          35h
node-ca-mb6zh                                      1/1     Running   0          35h
[root@utility ~]#

Test it in a new project test

[root@utility ~]# oc new-project test
Now using project "test" on server "https://api.ocp42.ssa.mbu.labs.redhat.com:6443".
You can add applications to this project with the 'new-app' command. For example, try:
    oc new-app django-psql-example
to build a new example application in Python. Or use kubectl to deploy a simple Kubernetes application:
    kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
[root@utility ~]# podman pull alpine
Trying to pull docker.io/library/alpine...Getting image source signatures
Copying blob c9b1b535fdd9 doneCopying config e7d92cdc71 doneWriting manifest to image destination
Storing signaturese7d92cdc71feacf90708cb59182d0df1b911f8ae022d29e8e95d75ca6a99776a
[root@utility ~]# podman login -u $(oc whoami) -p $(oc whoami -t) $REGISTRY_URL --tls-verify=false
Login Succeeded!
[root@utility ~]# podman tag alpine $REGISTRY_URL/test/alpine
[root@utility ~]# podman push $REGISTRY_URL/test/alpine --tls-verify=false
Getting image source signatures
Copying blob 5216338b40a7 done
Copying config e7d92cdc71 done
Writing manifest to image destination
Storing signatures
[root@utility ~]# oc get is -n test
NAME     IMAGE REPOSITORY                                                                        TAGS     UPDATED
alpine   default-route-openshift-image-registry.apps.ocp42.ssa.mbu.labs.redhat.com/test/alpine   latest   3 minutes ago
[root@utility ~]#

The registry works!

Other Scenario

If your cluster is deployed in vSphere and uses the default “thin” StorageClass but your datastore isn’t big enough, you can start from the OCS installation. When it comes to creating the OCS Cluster Service, create a YAML file with your desired sizes and without storageClassName (it will use the default one). You can also remove the “monPVCTemplate” if you are not interested in changing the storage size.

[root@utility ~]# cat <<EOF > ocs-cluster-service.yaml
apiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-storagecluster
  namespace: openshift-storage
spec:
  manageNodes: false
  monPVCTemplate:
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: ''
      volumeMode: Filesystem
  storageDeviceSets:
  - count: 1
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: ''
        volumeMode: Block
    name: ocs-deviceset
    placement: {}
    portable: true
    replica: 3
    resources: {}
EOF

Limits and Requests

Limits and Requests, by default, are set like that

[root@utility ~]# oc describe node worker-1.ocp42.ssa.mbu.labs.redhat.com
...
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
openshift-storage noobaa-core-0 4 (25%) 4 (25%) 8Gi (12%) 8Gi (12%) 13m
openshift-storage rook-ceph-mgr-a-676d4b4796-54mtk 1 (6%) 1 (6%) 3Gi (4%) 3Gi (4%) 12m
openshift-storage rook-ceph-mon-b-7d7747d8b4-k9txg 1 (6%) 1 (6%) 2Gi (3%) 2Gi (3%) 13m
openshift-storage rook-ceph-osd-1-854847fd4c-482bt 1 (6%) 2 (12%) 4Gi (6%) 8Gi (12%) 12m
...

We can create our new YAML file to change those settings in the ocs-storagecluster StorageCluster resource

[root@utility ~]# cat <<EOF > ocs-cluster-service-modified.yaml
apiVersion: ocs.openshift.io/v1
kind: StorageCluster
metadata:
  name: ocs-storagecluster
  namespace: openshift-storage
spec:
resources:
mon:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
mgr:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
noobaa-core:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
noobaa-db:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
  manageNodes: false
  monPVCTemplate:
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: 'local-sc'
      volumeMode: Filesystem
  storageDeviceSets:
  - count: 1
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 100Gi
        storageClassName: 'localblock-sc'
        volumeMode: Block
    name: ocs-deviceset
    placement: {}
    portable: true
    replica: 3
    resources:
limits:
cpu: 1
memory: 4Gi
requests:
cpu: 1
memory: 4Gi
EOF

And apply

[root@utility ~]# oc apply -f ocs-cluster-service-modified.yaml
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
storagecluster.ocs.openshift.io/ocs-storagecluster configured

We have to wait for the operator which reads the new configs and applies them

[root@utility ~]# oc describe node worker-1.ocp42.ssa.mbu.labs.redhat.com
...
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
openshift-storage noobaa-core-0 2 (12%) 2 (12%) 2Gi (3%) 2Gi (3%) 23s
openshift-storage rook-ceph-mgr-a-54f87f84fb-pm4rn 1 (6%) 1 (6%) 1Gi (1%) 1Gi (1%) 56s
openshift-storage rook-ceph-mon-b-854f549cd4-bgdb6 1 (6%) 1 (6%) 1Gi (1%) 1Gi (1%) 46s
openshift-storage rook-ceph-osd-1-ff56d545c-p7hvn 1 (6%) 1 (6%) 4Gi (6%) 4Gi (6%) 50s
...

And now we have our PODs with the new configurations applied.

The OSD PODs won’t start if you choose too low values.

Sections:

mon for rook-ceph-mon
mgr for rook-ceph-mgr
noobaa-core and noobaa-db for the 2 containers in the pod noobaa-core-0
mds for rook-ceph-mds-ocs-storagecluster-cephfilesystem
rgw for rook-ceph-rgw-ocs-storagecluster-cephobjectstore
the resources section in the end for rook-ceph-osd

rgw and mds sections work only the first time we create the resource.

---
spec:
resources:
mds:
limits:
cpu: 2
memory: 4Gi
requests:
cpu: 2
memory: 4Gi
rgw:
limits:
cpu: 1
memory: 2Gi
requests:
cpu: 1
memory: 2Gi
---

Conclusions

Now you can enjoy your brand-new OCS 4.2 in OCP 4.2.x
Things changed if you think about OCS 3.x, for example, the use of the PVCs instead of using directly the disks attached and for now, there are a lot of limitations for sustainability and supportability reasons.
We will wait for a fully supported installation for these scenarios.

UPDATES

The cluster used to write this article has been updated from 4.2.14 to 4.2.16 and then from 4.2.16 to 4.3.0.

The current OCS setup is still working

Upgrade

Added Requests and Limits configurations.

Heketi Integrated Metrics with Prometheus and Grafana in OCP 3.11

Wed, 01 Jan 2020 00:00:00 +0000

Since I started using OCP with GlusterFS one of the bigger blocker was the lack of metrics for GlusterFS. Now we have GlusterFS 3.4.0 with heketi 7 which ships the integrated metrics endpoint for Prometheus. Searching in our documentation, I found Architects – Red Hat Storage but it doesn’t work for OCP 3.11 because the entire Prometheus framework has changed. I found https://bugzilla.redhat.com/show_bug.cgi?id=1644665 which brought me to an internal Redhat document and from here I started ordering all the pieces of the puzzle.

In my lab I installed OCP 3.11.43 with RHGS 3.4.0 with 2 separate GlusterFS

glusterfs

to provision PVC for the apps

glusterfs_registry

to provision PVC for the infrastructure components

[glusterfs]
ocp-node-gluster1.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'
ocp-node-gluster2.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'
ocp-node-gluster3.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'
[glusterfs_registry]
ocp-node-gluster4.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'
ocp-node-gluster5.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'
ocp-node-gluster6.example.com glusterfs_devices='[ "/dev/sdc", "/dev/sdd" ]'‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

I checked the Heketi metrics endpoint

[root@ocp-master1 ~]# oc get svc -n ocs-infra
NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
heketi-db-registry-endpoints   ClusterIP   172.30.34.253   <none>        1/TCP      10h
heketi-registry                ClusterIP   172.30.17.135   <none>        8080/TCP   10h
[root@ocp-master1 ~]# curl 172.30.17.135:8080/metrics -s | head -n1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
[root@ocp-master1 ~]# oc get svc -n ocs-app
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
heketi-db-storage-endpoints   ClusterIP   172.30.227.21    <none>        1/TCP      10h
heketi-storage                ClusterIP   172.30.138.116   <none>        8080/TCP   10h
[root@ocp-master1 ~]# curl 172.30.138.116:8080/metrics -s | head -n1
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
[root@ocp-master1 ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Prometheus uses servicemonitors, new resources introduced by the Prometheus Operator which describe the set of targets to be monitored in OCP 3.11 (more information about Prometheus Operator here), so I had to create those objects:

[root@ocp-master1 ~]# cat heketi-infra-sm.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: heketi-infra
  labels:    k8s-app: heketi-infra
  namespace: openshift-monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: heketi
    scheme: http
    targetPort: 0
  namespaceSelector:
    matchNames:
    - ocs-infra
  selector:
    matchLabels:
      heketi: registry-service
[root@ocp-master1 ~]# oc create -f heketi-infra-sm.yaml -n openshift-monitoring
servicemonitor.monitoring.coreos.com/heketi-infra created
[root@ocp-master1 ~]#
[root@ocp-master1 ~]# cat heketi-app-sm.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: heketi-app
  labels:    k8s-app: heketi-app
  namespace: openshift-monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: heketi
    scheme: http
    targetPort: 0
  namespaceSelector:
    matchNames:
    - ocs-app
  selector:
    matchLabels:
      heketi: storage-service‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
[root@ocp-master1 ~]# oc create -f heketi-app-sm.yaml -n openshift-monitoring
servicemonitor.monitoring.coreos.com/heketi-app created
[root@ocp-master1 ~]#

The two selectors at line had been found in the heketi svc:

[root@ocp-master1 ~]# oc project ocs-infra
Now using project "ocs-infra"
[root@ocp-master1 ~]# oc describe svc heketi-registry
Name:              heketi-registry
Namespace:         ocs-infra
Labels:            glusterfs=heketi-registry-service
                  heketi=registry-service
...‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍
[root@ocp-master1 ~]# oc project ocs-app
Now using project "ocs-app"
[root@ocp-master1 ~]# oc describe svc heketi-storage
Name:              heketi-storage
Namespace:         ocs-app
Labels:            glusterfs=heketi-storage-service
                   heketi=storage-service
...‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Final, add the cluster role to the prometheus-k8s service account:

[root@ocp-master1 ~]# oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-monitoring:prometheus-k8s -n openshift-monitoring
cluster role "cluster-reader" added: "system:serviceaccount:openshift-monitoring:prometheus-k8s"
[root@ocp-master1 ~]#‍‍‍‍‍‍

After about 1 minute, Prometheus loaded the new servicemonitors:

targets

In the Grafana shipped with OCP 3.11, to have admin privileges you MUST have an user “admin” with cluster-admin cluster role. I created the user (htpasswd Identity Provider):

[root@ocp-master1 ~]# htpasswd /etc/origin/master/htpasswd admin
New password:Re-type new password:
Updating password for user admin
[root@ocp-master1 ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And added it to the cluster role:

[root@ocp-master1 ~]# oc adm policy add-cluster-role-to-user cluster-admin admin
cluster role "cluster-admin" added: "admin"
[root@ocp-master1 ~]#‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

In the previous gdrive document you can find this mail http://post-office.corp.redhat.com/archives/sme-storage/2018-October/msg00388.html which contains a Grafana Dashboard for these metrics. I added some variables to manage more than 3 nodes and both GlusterFS clusters. Finally the new dashboard was imported:

grafana

Please Note

Grafana uses ephemeral storage: if the pod is destroyed you MUST re-import this dashboard.

Enjoy your metrics!

Google OAuth as Identity Provider with Red Hat login in OCP 3.11

Wed, 13 Mar 2019 00:00:00 +0000

When I was in Red Hat, I needed to grant access to my lab to some of my colleagues.
The lab uses httpasswd IdentityProvider and It was really painful to add new users to the file each time.
So, an idea popped up: could I use the google oauth Identity Provider with our Red Hat login?
Well, it can be done! This is a detailed how-to

I logged in https://console.developers.google.com/apis/dashboard with my Red Hat credentials.

At the top of the page, click the select box next to the google APIs logo

google apis

Choose REDHAT.COM in the Select from box and then click NEW PROJECT

location

Choose your Project Name and be sure that the Location is redhat.com. Then click CREATE

new project

On the left, you’ll find the credentials section: click on it

credentials

Under credentials, click on the tab OAuth Consent Screen

Now we have to configure the Application type as internal and add your ocp domain in Authorized Domain and your Application Name. Then click save and you’ll be redirected in the credentials configuration

application type

authorized domain

Click Create credentials and select Oauth Client ID

create credentials

Select Web application in Application type and choose the Name. In Authorized JavaScript origins add the URI of your ocp webconsole. In Authorized redirect URIs add your callback uri. In OCP 3.11 your callback uri should be: https://<master>/oauth2callback/<identityProviderName>. The IdentityProviderName must have the same name as the one we’ll configure in OpenShift. Then click create. A popup will be shown giving you the client ID and the client secret. Save that information because we’ll need them later to setup OpenShift

web application

credentials

Now it’s time to configure our OpenShift. The following procedure must be done in **ALL **the masters of the cluster.

You must log in the master server and modify the /etc/origin/master/master-config.yml file adding this snippet under the section identityProviders

  - name: RedHat
    challenge: false
    login: true
    mappingMethod: claim
    provider:
      apiVersion: v1
      kind: GoogleIdentityProvider
      clientID: "xxx"
      clientSecret: "xxx"
      hostedDomain: "redhat.com"‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

name must be the same as the IdentityProviderName we have configured in the callback URI

clientID and clientSecret are the info we got in the credentials setup in google.

After that, restart api and controllers

[root@ocp-master1 ~]# master-restart api api
2
[root@ocp-master1 ~]# master-restart controllers controllers
2
[root@ocp-master1 ~]# ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Now we can check if all works good. Open in your browser your OCP webconsole and select the RedHat identity provider

oauth page

You’ll be redirect into the RED HAT INTERNAL SSO

sso page

And finally you’ll have access to your OpenShift

webconsole

[root@ocp-master1 ~]# oc get user
NAME                  UID                                    FULL NAME         IDENTITIES
aspagnol@redhat.com   f2e04e82-40d9-11e9-ac71-005056a802f7   Andrea Spagnolo   RedHat:108476506439924310236
[root@ocp-master1 ~]# oc get identity
NAME                           IDP NAME   IDP USER NAME           USER NAME             USER UID
RedHat:108476506439924310236   RedHat     108476506439924310236   aspagnol@redhat.com   f2e04e82-40d9-11e9-ac71-005056a802f7
[root@ocp-master1 ~]# ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

Now you can manage your users directly in OpenShift and, for example, create an admin group and add the users

[root@ocp-master1 ~]# oc adm groups new admins
group.user.openshift.io/admins created
[root@ocp-master1 ~]# oc adm policy add-cluster-role-to-group cluster-admin admins
cluster role "cluster-admin" added: "admins"
[root@ocp-master1 ~]# oc adm groups add-users admins aspagnol@redhat.com
group "admins" added: "aspagnol@redhat.com"
[root@ocp-master1 ~]# oc describe groups admins
Name:          admins
Created:     About a minute ago
Labels:          <none>
Annotations:     <none>
Users:          aspagnol@redhat.com
[root@ocp-master1 ~]# ‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍‍

And we can check in the webconsole

webconsole

You can find the complete documentation about the Identity Providers in OCP here. You can also configure the inventory to add the GoogleIdentityProvider directly during the installation of OCP

openshift_master_identity_providers=[{'name': 'RedHat', 'challenge': 'false', 'login': 'true', 'kind': 'GoogleIdentityProvider', 'clientID': 'xxx', 'clientSecret': 'xxx', 'hostedDomain': 'redhat.com'}]‍‍

Please Note

If yours OpenShift masters need a proxy to go to internet, the proxy MUST have https://www.googleapis.com in allow because it’s needed by the server to get the oauth2 token

That’s All!

ocp | Spagno's Blog

OCS 4.2 in OCP 4.2.14 - UPI installation in RHV

Table of Contents

Prerequisites

Issues

Use case scenario

Challenges

Solutions

Procedures

Other Scenario

Limits and Requests

Conclusions

UPDATES

Heketi Integrated Metrics with Prometheus and Grafana in OCP 3.11

Please Note

Google OAuth as Identity Provider with Red Hat login in OCP 3.11

Please Note