tensorflow/core/grappler/optimizers/data/auto_shard.cc
// Recursively walk the dataset graph from sink to source, searching for
// the first (i.e. closest to the sink) occurrence of a ReaderDataset, such as
// CSVDataset, TFRecordDataset, etc. We then insert a ShardDataset op before
// that nodes input, so that each worker only reads a subset of files.
// Additionally, we remove sources of randomness (e.g. ShuffleDataset) that
// occur upstream of the ShardDataset transformation to ensure that sharding
// returns a sensible result.
Recursively walk the dataset graph from sink to source, searching for
the first  occurrence of a ReaderDataset, such as
CSVDataset, TFRecordDataset, etc. We then insert a ShardDataset op before
that nodes input, so that each worker only reads a subset of files.
Additionally, we remove sources of randomness  that
occur upstream of the ShardDataset transformation to ensure that sharding
returns a sensible result.
Recursively walk the dataset graph from sink to source searching for the first occurrence of a ReaderDataset such as CSVDataset TFRecordDataset etc . We then insert a ShardDataset op before that nodes input so that each worker only reads a subset of files . Additionally we remove sources of randomness that occur upstream of the ShardDataset transformation to ensure that sharding returns a sensible result .
Status ShardByFile(const NodeDef& sink_node, int64_t num_workers, int64_t index,
                   FunctionLibraryDefinition* flib, MutableGraphView* graph) {
  absl::flat_hash_set<string> nodes_to_delete;
  TF_RETURN_IF_ERROR(RecursivelyHandleOp(sink_node, num_workers, index, flib,
                                         graph, &nodes_to_delete));
  return graph->DeleteNodes(nodes_to_delete);
}
Status ShardByFile ( const NodeDef & sink_node , int64_t num_workers , int64_t index , FunctionLibraryDefinition * flib , MutableGraphView * graph ) { absl :: flat_hash_set < string > nodes_to_delete ; TF_RETURN_IF_ERROR ( RecursivelyHandleOp ( sink_node , num_workers , index , flib , graph , & nodes_to_delete ) ) ; return graph -> DeleteNodes ( nodes_to_delete ) ; }
Recursively walk the dataset graph from sink to source, searching for
the first (i.e.
Recursively walk the dataset graph from sink to source searching for the first ( i . e .