original_docstring: Decorator which makes a DynamicItem and specifies its argkeys.
  
      If the wrapped object is a generator function (has a yield statement),
      Creates a GeneratorDynamicItem. If the object is already a DynamicItem,
      just specifies the argkeys for that. Otherwise creates a new regular
      DynamicItem, with argkeys specified.
  
      The args are always passed to the function at the start. Generators could
      support sending new arguments, but for such use cases, simply create a new
      dynamic item. The GeneratorDynamicItem class is meant for pipelines which
      take in an input and transform it in multiple ways, where the intermediate
      representations may be needed for e.g. fitting a BPE segmenter.
  
      Example
      -------
      @takes("text")
      ... def tokenize(text):
      ...     return text.strip().lower().split()
      tokenize.provides = ["tokenized"]
      tokenize('\tThis Example gets tokenized')
      ['this', 'example', 'gets', 'tokenized']
      
docstring: Decorator which makes a DynamicItem and specifies its argkeys.
  If the wrapped object is a generator function (has a yield statement),
  Creates a GeneratorDynamicItem. If the object is already a DynamicItem,
  just specifies the argkeys for that. Otherwise creates a new regular
  DynamicItem, with argkeys specified.
  
  The args are always passed to the function at the start. Generators could
  support sending new arguments, but for such use cases, simply create a new
  dynamic item. The GeneratorDynamicItem class is meant for pipelines which
  take in an input and transform it in multiple ways, where the intermediate
  representations may be needed for e.g. fitting a BPE segmenter.
  
  Example
  
  
docstring_tokens: Decorator which makes a DynamicItem and specifies its argkeys . If the wrapped object is a generator function ( has a yield statement ) Creates a GeneratorDynamicItem . If the object is already a DynamicItem just specifies the argkeys for that . Otherwise creates a new regular DynamicItem with argkeys specified . The args are always passed to the function at the start . Generators could support sending new arguments but for such use cases simply create a new dynamic item . The GeneratorDynamicItem class is meant for pipelines which take in an input and transform it in multiple ways where the intermediate representations may be needed for e . g . fitting a BPE segmenter . Example
code: def takes(*argkeys):
      """Decorator which makes a DynamicItem and specifies its argkeys.
  
      If the wrapped object is a generator function (has a yield statement),
      Creates a GeneratorDynamicItem. If the object is already a DynamicItem,
      just specifies the argkeys for that. Otherwise creates a new regular
      DynamicItem, with argkeys specified.
  
      The args are always passed to the function at the start. Generators could
      support sending new arguments, but for such use cases, simply create a new
      dynamic item. The GeneratorDynamicItem class is meant for pipelines which
      take in an input and transform it in multiple ways, where the intermediate
      representations may be needed for e.g. fitting a BPE segmenter.
  
      Example
      -------
      @takes("text")
      ... def tokenize(text):
      ...     return text.strip().lower().split()
      tokenize.provides = ["tokenized"]
      tokenize('\tThis Example gets tokenized')
      ['this', 'example', 'gets', 'tokenized']
      """
  
      def decorator(obj):
          if isinstance(obj, DynamicItem):
              if obj.takes:
                  raise ValueError("Can't overwrite DynamicItem.takes")
              obj.takes = argkeys
              return obj
          elif inspect.isgeneratorfunction(obj):
              return GeneratorDynamicItem(takes=argkeys, func=obj)
          else:
              return DynamicItem(takes=argkeys, func=obj)
  
      return decorator
code_tokens: def takes ( * argkeys ) : def decorator ( obj ) : if isinstance ( obj , DynamicItem ) : if obj . takes : raise ValueError ( "Can't overwrite DynamicItem.takes" ) obj . takes = argkeys return obj elif inspect . isgeneratorfunction ( obj ) : return GeneratorDynamicItem ( takes = argkeys , func = obj ) else : return DynamicItem ( takes = argkeys , func = obj ) return decorator
short_docstring: Decorator which makes a DynamicItem and specifies its argkeys.
short_docstring_tokens: Decorator which makes a DynamicItem and specifies its argkeys .
comment: """Decorator which makes a DynamicItem and specifies its argkeys.
  
      If the wrapped object is a generator function (has a yield statement),
      Creates a GeneratorDynamicItem. If the object is already a DynamicItem,
      just specifies the argkeys for that. Otherwise creates a new regular
      DynamicItem, with argkeys specified.
  
      The args are always passed to the function at the start. Generators could
      support sending new arguments, but for such use cases, simply create a new
      dynamic item. The GeneratorDynamicItem class is meant for pipelines which
      take in an input and transform it in multiple ways, where the intermediate
      representations may be needed for e.g. fitting a BPE segmenter.
  
      Example
      -------
      @takes("text")
      ... def tokenize(text):
      ...     return text.strip().lower().split()
      tokenize.provides = ["tokenized"]
      tokenize('\tThis Example gets tokenized')
      ['this', 'example', 'gets', 'tokenized']
      """