/kind feature
/sig scheduling

Discussion with sig-scalability: https://kubernetes.slack.com/archives/C09QZTRH7/p1715262959575039

## What

We have scheduler-perf, and it'd be great if we could have an alert-ish stuff based on the result.

Based on the discussion with sig-scalability, the easiest way is to change scheduler_perf so that it can fail if the results show degradation, and monitor/alert the failures via testgrid.

"_if the results show degradation_" > for this, we probably have to define reasonable thresholds per test case.

## Why

The current pain point is that [perf-dash](https://perf-dash.k8s.io/#/?jobname=scheduler-perf-benchmark&metriccategoryname=Sched[…]20applicable&result=not%20applicable&plugin=not%20applicable) visualizes it, but no one actually doesn't care much, and consequently we've overlooked degradation several times actually.

scheduler_perf: define thresholds per test case and set up alerts for results
This is a GitHub Issue
repo:kubernetes
owner:kubernetes
Title : scheduler_perf: define thresholds per test case and set up alerts for results
Issue date:

--- start body ---
/kind feature
/sig scheduling

Discussion with sig-scalability: https://kubernetes.slack.com/archives/C09QZTRH7/p1715262959575039

## What

We have scheduler-perf, and it'd be great if we could have an alert-ish stuff based on the result.

Based on the discussion with sig-scalability, the easiest way is to change scheduler_perf so that it can fail if the results show degradation, and monitor/alert the failures via testgrid.

"_if the results show degradation_" > for this, we probably have to define reasonable thresholds per test case.

## Why

The current pain point is that [perf-dash](https://perf-dash.k8s.io/#/?jobname=scheduler-perf-benchmark&metriccategoryname=Sched[…]20applicable&result=not%20applicable&plugin=not%20applicable) visualizes it, but no one actually doesn't care much, and consequently we've overlooked degradation several times actually.

--- end body ---
