24. 05. 2024 Alessandro Valentini DevOps

OpenShift: How to Check and Reset Ceph Storage in Warning State

Every so often it may happen (in particular after a cluster update or hardware issues) that you see your storage in a warning state on OpenShift.

The first thing to do is to check what’s wrong with your cluster: sometimes you may have a real issue, in other cases it may just be a temporary problem which is already resolved.

The first place to check is in the GUI: Storage -> Data Foundation → Topology and check the alerts.

In this case we can see the error Storage cluster is in warning state for more than 15m. and the official documentation suggests collecting a must-gather and sending it to RedHat Support.

You can gain additional details for investigation from the CLI using ceph-tools:

Deploy the toolbox manually with the patch command:

oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

Then connect to the new pod toolbox:

TOOLS_POD=$(oc get pods -n openshift-storage -l app=rook-ceph-tools -o name)
oc rsh -n openshift-storage $TOOLS_POD

And run these commands to get more info about the health state:

ceph -s
ceph health detail

You’ll have to investigate the issue and fix it, or else contact official support.

If everything is okay and the problem has been automatically recovered, then at this point you can list and check the crashes:

ceph crash ls
ceph crash info <ID>

and finally you can archive them to reset the warning:

ceph crash archive 2024-01-04T14:24:04.160485Z_...

and you will then have see everything in green.

These Solutions are Engineered by Humans

Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth Phoenix.