### What happened?

We bumped an aggregated apiserver library based on the generic apiserver library to 1.29 in https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/170 and users mentioned that after pulling the new dependencies, the aggregated apiservers started failing on Kubernetes 1.28 clusters with the following errors:

```
W0425 14:25:27.110980       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.111035       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.571189       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.571265       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:39.776616       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:39.776683       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:58.606055       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:58.606080       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:26:01.846754       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:26:01.846786       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:27.211376       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.211394       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.498183       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.498228       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:35.545134       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:35.545287       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:49.677632       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:49.677657       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:51.325292       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:51.325325       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
```

### What did you expect to happen?

I expected 1.29 libraries to still be compatible with Kubernetes 1.28 based on the kube-apiserver version skew.

### How can we reproduce it (as minimally and precisely as possible)?

I haven't tested it with the sample-apiserver yet, but I assume that running its 1.29 version on a 1.28 kind cluster will result in the same failure.

If not, feel free to reach out to me I can share a reproducer with prometheus-adapter, and aggregated apiserver where the problem exists.

### Anything else we need to know?

The error appeared with the graduation of P&F to stable in 1.29. I chatted briefly with @tkashem about it and he mentioned that the controller logic was moved from v1beta3 to v1 in 1.29 which is also the version that introduces the v1 API. Meaning that retro compatibility with 1.28 was broken.
1.29-apiserver based aggregated APIServer does not work against 1.28 kube-apiserver
This is a GitHub Issue
repo:kubernetes
owner:kubernetes
Title : 1.29-apiserver based aggregated APIServer does not work against 1.28 kube-apiserver
Issue date:

--- start body ---
### What happened?

We bumped an aggregated apiserver library based on the generic apiserver library to 1.29 in https://github.com/kubernetes-sigs/custom-metrics-apiserver/pull/170 and users mentioned that after pulling the new dependencies, the aggregated apiservers started failing on Kubernetes 1.28 clusters with the following errors:

```
W0425 14:25:27.110980       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.111035       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.571189       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.571265       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:39.776616       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:39.776683       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:58.606055       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:58.606080       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:26:01.846754       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:26:01.846786       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:27.211376       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:27.211394       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:34.498183       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:34.498228       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
W0425 14:25:35.545134       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:35.545287       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:49.677632       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E0425 14:25:49.677657       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
W0425 14:25:51.325292       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: failed to list *v1.FlowSchema: the server could not find the requested resource
E0425 14:25:51.325325       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.4/tools/cache/reflector.go:229: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource
```

### What did you expect to happen?

I expected 1.29 libraries to still be compatible with Kubernetes 1.28 based on the kube-apiserver version skew.

### How can we reproduce it (as minimally and precisely as possible)?

I haven't tested it with the sample-apiserver yet, but I assume that running its 1.29 version on a 1.28 kind cluster will result in the same failure.

If not, feel free to reach out to me I can share a reproducer with prometheus-adapter, and aggregated apiserver where the problem exists.

### Anything else we need to know?

The error appeared with the graduation of P&F to stable in 1.29. I chatted briefly with @tkashem about it and he mentioned that the controller logic was moved from v1beta3 to v1 in 1.29 which is also the version that introduces the v1 API. Meaning that retro compatibility with 1.28 was broken.
--- end body ---
