title: How to include path information in data extracted from text files with FileSource
body: <p>We’ve got time-stamped directories containing text files, stored in HDFS.</p>
  <p>We can regularly get new files added, so we’re using a <code>FileSource</code> (Flink 1.14.4, and a streaming job) with a monitoring duration, so that it continuously picks up any new files.</p>
  <p>The challenge is that we need to include the parent directory’s timestamp in the output, for doing time-window joins of this enrichment data with another stream.</p>
  <p>Previously I could extend the input format to extract path information, and emit a <code>Tuple2&lt;LongWritable, Text&gt;</code> (see my <a href="https://stackoverflow.com/a/68764550/231762">SO answer</a> to a question about doing that).</p>
  <p>But with the new FileSource architecture, I’m really not sure if it’s possible, or if so, the right way to go about doing it.</p>
  <p>I’ve wandered through the source code (FileSource, AbstractFileSource, SourceReader, FileSourceReader, FileSourceSplit, ad nauseam) but haven’t seen any happy path to making that all work.</p>
  <p>There might be a way using some really ugly hacks to TextLineFormat, where it would reverse engineer the FSDataInputStream to try to find information about the original file, but feels very fragile.</p>
  <p>Any suggestions?</p>