### What happened?

For example, we had a pod which had:

```yaml
 securityContext:
   fsGroup: 1001
   runAsGroup: 1001
   runAsUser: 1001
```

This — unbeknownst to us for quite some time — led to the pod taking _hours_ to start, with it being otherwise apparently stuck in `Pending`.

Now, _in this particular example_, the "root" cause is that the `fsGroup` there, combined with `fsGroupChangePolicy: Always` — _**which is the default**_, and unspecified by us here — causes the `kubelet`, prior to starting the pod's containers, to recursively `chown` the Pod's volumes to the GID in `fsGroup`. *This is a potentially enormous piece of I/O*, as in our case, where it took _hours._ (The volume in question had 2.1M files on it!)

### What did you expect to happen?

A clear way to tell *why* a pod is not transitioning out of `Pending` in a timely manner. A simple event in the `describe pod` output for the pod would have done it.

### How can we reproduce it (as minimally and precisely as possible)?

Have a volume w/ ~2M files, and an `fsGroup`.

### Anything else we need to know?

The same thing goes for pod termination, too. For example, see [this bug](https://github.com/kubernetes/kubernetes/issues/51835), which is full of people who cannot determine why a pod isn't capable of termination. Sadly, that bug was never addressed, and was dismissed with:

> there are a lot of different issues being discussed in this thread. There are many possible reasons why a Pod could get stuck in terminating. We know this is a common issue with many possible root causes!

The same pod (large volume + `fsGroup`) is also failing to terminate cleanly; hence I stumbled into that bug. And similarly, there is no output in `describe pod` as to why the pod isn't making progress. Users of Kubernetes need to be able to tell what operations are pending against a pod that the kubelet is still working on, that prevent it from transitioning, e.g., to `Terminated` or to `CreatingContainer`, etc.

Cf. https://github.com/Azure/AKS/issues/3865

Cf. https://github.com/Azure/AKS/issues/3681

We didn't have someone who knew this particular facet of k8s by heart on hand, and thus spent years dealing with this. Azure's support spent years with this too, similarly making little to no progress. (Until that issues 3865 above finally cracked the case.) That is what makes a useful diagnostic all the more important: to clue the user in on this default behavior that is probably the right thing for most use cases, but not something the user ever asked for, and thus, not something they're going to guess.

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

Azure (AKS), GCP (GKE)


### OS version

GKE:

(We've since left Azure/AKS for this, mostly. But given that we have replicated the behavior on GKE, I'm pretty sure [the linked bug](https://github.com/Azure/AKS/issues/3865) is correct in that this is base k8s behavior, not specific to a vendor's implementation.)

<details>

```console
# cat /etc/lsb-release
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu
CHROMEOS_RELEASE_BRANCH_NUMBER=66
CHROMEOS_RELEASE_BUILD_NUMBER=17800
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=109
CHROMEOS_RELEASE_DESCRIPTION=17800.66.78 (Official Build) stable-channel lakitu
CHROMEOS_RELEASE_KEYSET=v10
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=78
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=17800.66.78
DEVICETYPE=OTHER
GOOGLE_RELEASE=17800.66.78
HWID_OVERRIDE=LAKITU DEFAULT
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

`containerd://1.7.10`

### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>

Potentially lengthy I/O by kubelet during pod startup needs to be visible to the user
This is a GitHub Issue
repo:kubernetes
owner:kubernetes
Title : Potentially lengthy I/O by kubelet during pod startup needs to be visible to the user
Issue date:

--- start body ---
### What happened?

For example, we had a pod which had:

```yaml
 securityContext:
   fsGroup: 1001
   runAsGroup: 1001
   runAsUser: 1001
```

This — unbeknownst to us for quite some time — led to the pod taking _hours_ to start, with it being otherwise apparently stuck in `Pending`.

Now, _in this particular example_, the "root" cause is that the `fsGroup` there, combined with `fsGroupChangePolicy: Always` — _**which is the default**_, and unspecified by us here — causes the `kubelet`, prior to starting the pod's containers, to recursively `chown` the Pod's volumes to the GID in `fsGroup`. *This is a potentially enormous piece of I/O*, as in our case, where it took _hours._ (The volume in question had 2.1M files on it!)

### What did you expect to happen?

A clear way to tell *why* a pod is not transitioning out of `Pending` in a timely manner. A simple event in the `describe pod` output for the pod would have done it.

### How can we reproduce it (as minimally and precisely as possible)?

Have a volume w/ ~2M files, and an `fsGroup`.

### Anything else we need to know?

The same thing goes for pod termination, too. For example, see [this bug](https://github.com/kubernetes/kubernetes/issues/51835), which is full of people who cannot determine why a pod isn't capable of termination. Sadly, that bug was never addressed, and was dismissed with:

> there are a lot of different issues being discussed in this thread. There are many possible reasons why a Pod could get stuck in terminating. We know this is a common issue with many possible root causes!

The same pod (large volume + `fsGroup`) is also failing to terminate cleanly; hence I stumbled into that bug. And similarly, there is no output in `describe pod` as to why the pod isn't making progress. Users of Kubernetes need to be able to tell what operations are pending against a pod that the kubelet is still working on, that prevent it from transitioning, e.g., to `Terminated` or to `CreatingContainer`, etc.

Cf. https://github.com/Azure/AKS/issues/3865

Cf. https://github.com/Azure/AKS/issues/3681

We didn't have someone who knew this particular facet of k8s by heart on hand, and thus spent years dealing with this. Azure's support spent years with this too, similarly making little to no progress. (Until that issues 3865 above finally cracked the case.) That is what makes a useful diagnostic all the more important: to clue the user in on this default behavior that is probably the right thing for most use cases, but not something the user ever asked for, and thus, not something they're going to guess.

### Kubernetes version

<details>

```console
$ kubectl version
# paste output here
```

</details>


### Cloud provider

Azure (AKS), GCP (GKE)


### OS version

GKE:

(We've since left Azure/AKS for this, mostly. But given that we have replicated the behavior on GKE, I'm pretty sure [the linked bug](https://github.com/Azure/AKS/issues/3865) is correct in that this is base k8s behavior, not specific to a vendor's implementation.)

<details>

```console
# cat /etc/lsb-release
CHROMEOS_AUSERVER=https://tools.google.com/service/update2
CHROMEOS_BOARD_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}
CHROMEOS_DEVSERVER=
CHROMEOS_RELEASE_APPID={76E245CF-C0D0-444D-BA50-36739C18EB00}
CHROMEOS_RELEASE_BOARD=lakitu
CHROMEOS_RELEASE_BRANCH_NUMBER=66
CHROMEOS_RELEASE_BUILD_NUMBER=17800
CHROMEOS_RELEASE_BUILD_TYPE=Official Build
CHROMEOS_RELEASE_CHROME_MILESTONE=109
CHROMEOS_RELEASE_DESCRIPTION=17800.66.78 (Official Build) stable-channel lakitu
CHROMEOS_RELEASE_KEYSET=v10
CHROMEOS_RELEASE_NAME=Chrome OS
CHROMEOS_RELEASE_PATCH_NUMBER=78
CHROMEOS_RELEASE_TRACK=stable-channel
CHROMEOS_RELEASE_VERSION=17800.66.78
DEVICETYPE=OTHER
GOOGLE_RELEASE=17800.66.78
HWID_OVERRIDE=LAKITU DEFAULT
```

</details>


### Install tools

<details>

</details>


### Container runtime (CRI) and version (if applicable)

`containerd://1.7.10`

### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>

</details>

--- end body ---
