MountVolume.SetUp failed for volume secret not found Issue

Today I found that some pods in kubernetes cluster are failed, the status is Waiting: ContainerCreating. The pod events:

MountVolume.SetUp failed for volume "xxxxx" : secret "xxxxx" not found
kubelet aks-agentpool-xxx-vmss000001

Unable to attach or mount volumes: unmounted volumes=[xxxxx], unattached volumes=[xxxxx]: timed out waiting for the condition

I remember that about one week ago I delete some secretes in this cluster. Therefore, the problem becomes how to recover the deleted secret "xxxxx"?

Refer the following documents: # To create additional API tokens

According to # Service Account Admission Controller:

if the pod does not have a ServiceAccount set, it sets the ServiceAccount to default.

Since the missing secret format is "default-token-xxxxx", the ServiceAccount must be default.

Then we create a service.json file:

{
    "kind": "Secret",
    "apiVersion": "v1",
    "metadata": {
        "name": "default-token-xxxxx",
        "annotations": {
            "kubernetes.io/service-account.name": "default"
        }
    },
    "type": "kubernetes.io/service-account-token"
}

There are two fields in metadata need to be modified:

• name: change to your missing secret name which is included in the warning message
• kubernetes.io/service-account.name: default

Execute the following command:

kubectl create -f ./secret.json

Check if the secret is there:

kubectl get secrets

Done. Waiting for the pod to start!

If you deleted the default kubernetes secret by mistake, hope this artible helps. :-)