title: How to get kafka consumer lag for spark structured streaming application
body: <p>I am building monitoring for my spark structured streaming application and need to get consumer lag of a certain topic consumed by the spark application. I believe the spark driver must be aware of this lag as it has all the metadata of the executors. I don't see any way to get this metrics from any existing spark docs or resources. I checked out <code>streaminQueryListener</code> interface but it also has limited capability as we can get only per query metrics from it.  </p>