title: How to add labels to an existing Google Dataflow job?
body: <p>I am using the Java GAPI client to work with Google Cloud Dataflow (v1b3-rev197-1.22.0). I am running a pipeline from template and the method for doing that (<code>com.google.api.services.dataflow.Dataflow.Projects.Templates#create</code>) does not allow me to set labels for the job. However I get the <code>Job</code> object back when I execute the pipeline, so I updated the labels and tried to call <code>com.google.api.services.dataflow.Dataflow.Projects.Jobs#update</code> to persist that information in Dataflow. But the labels do not get updated. </p>
  
  <p>I also tried updating labels on finished jobs (which I also need to do), which didn't work either, so I thought it's because the job is in a terminal state. But updating labels seems to do nothing regardless of the state.</p>
  
  <p>The documentation does not say anything about labels not being mutable on running or terminated pipelines, so I would expect things to work. Am I doing something wrong and if not what is the rationale behing the decision no to allow label updates? (And how are template users supposed to set the initial label set when executing the template?)</p>
  
  <p>Background: I want to mark terminated pipelines that have been "processed", i.e. those that our automated infrastructure already sent notification about to appropriate places. Labels seemed as a good approach that would shield me from having to use some kind of local persitence to track stuff (big complexity jump). Any suggestions on how to approach this if labels are not the right tool? Sadly, Stackdriver cannot monitor finished pipelines, only failed ones. And sending a notification from within the pipeline code doesn't seem as a good idea to me (wrong?).</p>