Troubleshooting Guide

Cluster not ready

Cluster may be showing as not ready in following cases

  • Kubeconfig credentails supplied when connecting a cluster may have been expired. Try exporting kubeconfig manually and try connecting
export KUBECONFIG=your.kubeconfig
kubectl get nodes
  • Kubeconfig credentails may be using custom cloud provider auth, use following method to Quick ref to generate generic kubeconfigs from service account

  • Cluster may be down or not reachable from outside, check connectivity

Backup component not ready

  • Check if velero installation is failing on member cluster
export KUBECONFIG=your.kubeconfig
kubectl get pods -n cape

Cleanup CAPE installation

Be careful with following steps as they may remove any stored config/data of workloads.

On Member cluster

for type in 'restores' 'schedules' 'backups' 'veleros'
do
for obj in $(kubectl get $type | awk '{print $1}'); do
echo "deleting $type $obj";
kubectl -n velero patch $type $obj -p '{"metadata":{"finalizers": []}}'
done
kubectl -n velero delete $type --all --force --grace-period=0;
done

On CAPE cluster

for crd in 'backuplocationoperations' 'components' 'backuplocations' 'federations' 'clusters' 'users' 'organizations' 'deployments.operator.cape.sh' 'applications' 'repositories' 'configs'
do
for ns in $(kubectl get ns --no-headers -l heritage=cape | awk '{print $1}'); do
echo "deleting $crd in $ns";
kubectl -n $ns delete $crd --all --timeout=2s;
kubectl -n $ns patch $crd -p '{"metadata":{"finalizers": []}}' --type=merge
kubectl -n $ns delete $crd --all --timeout=2s;
kubectl -n $ns patch $crd -p '{"metadata":{"finalizers": []}}' --type=merge
kubectl -n $ns delete $crd --all --timeout=2s;
done
done
kubectl delete organizations --all -n cape
kubectl get ns -l heritage=cape | awk '{print $1}' | xargs kubectl delete ns

Clean up Installation

kubectl delete ns cape

delete crd's and roles

kubectl get crd | grep cape.sh | awk '{print $1}' | xargs kubectl delete crd
kubectl delete clusterrole,clusterrolebinding -l heritage=cape

Verify

for crd in 'deployments.operator.cape.sh' 'applications' 'repositories' 'configs' 'components' 'backuplocations' 'federations' 'clusters' 'users' 'organizations' 'backuplocationoperations'
do
for ns in $(kubectl get ns -l heritage=cape | awk '{print $1}'); do
kubectl -n $ns get $crd
done
done

Terminating namespace issue

port="$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1])')";
echo "Unused Port: $port"
kubectl proxy --port=$port &
for ns in `kubectl get ns --field-selector status.phase=Terminating -o name | cut -d/ -f2`;
do
export NS=$ns
kubectl create ns $NS --dry-run -ojson > /tmp/ns.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @/tmp/ns.json http://localhost:$port/api/v1/namespaces/$NS/finalize
done