original_docstring: 
      Create a learning rate scheduler if specified in config and
      determine when a scheduler step should be executed.
  
      Current options:
          - "plateau": see `torch.optim.lr_scheduler.ReduceLROnPlateau`
          - "decaying": see `torch.optim.lr_scheduler.StepLR`
          - "exponential": see `torch.optim.lr_scheduler.ExponentialLR`
          - "noam": see `joeynmt.builders.NoamScheduler`
          - "warmupexponentialdecay": see
            `joeynmt.builders.WarmupExponentialDecayScheduler`
  
      If no scheduler is specified, returns (None, None) which will result in
      a constant learning rate.
  
      :param config: training configuration
      :param optimizer: optimizer for the scheduler, determines the set of
          parameters which the scheduler sets the learning rate for
      :param scheduler_mode: "min" or "max", depending on whether the validation
          score should be minimized or maximized.
          Only relevant for "plateau".
      :param hidden_size: encoder hidden size (required for NoamScheduler)
      :return:
          - scheduler: scheduler object,
          - scheduler_step_at: either "validation" or "epoch"
      
docstring: Create a learning rate scheduler if specified in config and
  determine when a scheduler step should be executed.
  
  
  
  If no scheduler is specified, returns (None, None) which will result in
  a constant learning rate.
docstring_tokens: Create a learning rate scheduler if specified in config and determine when a scheduler step should be executed . If no scheduler is specified returns ( None None ) which will result in a constant learning rate .
code: def build_scheduler(config: dict, optimizer: Optimizer, scheduler_mode: str,
                      hidden_size: int = 0) \
          -> (Optional[_LRScheduler], Optional[str]):
      """
      Create a learning rate scheduler if specified in config and
      determine when a scheduler step should be executed.
  
      Current options:
          - "plateau": see `torch.optim.lr_scheduler.ReduceLROnPlateau`
          - "decaying": see `torch.optim.lr_scheduler.StepLR`
          - "exponential": see `torch.optim.lr_scheduler.ExponentialLR`
          - "noam": see `joeynmt.builders.NoamScheduler`
          - "warmupexponentialdecay": see
            `joeynmt.builders.WarmupExponentialDecayScheduler`
  
      If no scheduler is specified, returns (None, None) which will result in
      a constant learning rate.
  
      :param config: training configuration
      :param optimizer: optimizer for the scheduler, determines the set of
          parameters which the scheduler sets the learning rate for
      :param scheduler_mode: "min" or "max", depending on whether the validation
          score should be minimized or maximized.
          Only relevant for "plateau".
      :param hidden_size: encoder hidden size (required for NoamScheduler)
      :return:
          - scheduler: scheduler object,
          - scheduler_step_at: either "validation" or "epoch"
      """
      scheduler, scheduler_step_at = None, None
      if "scheduling" in config.keys() and \
              config["scheduling"]:
          if config["scheduling"].lower() == "plateau":
              # learning rate scheduler
              scheduler = ReduceLROnPlateau(optimizer=optimizer,
                                            mode=scheduler_mode,
                                            verbose=False,
                                            threshold_mode='abs',
                                            factor=config.get(
                                                "decrease_factor", 0.1),
                                            patience=config.get("patience", 10))
              # scheduler step is executed after every validation
              scheduler_step_at = "validation"
          elif config["scheduling"].lower() == "decaying":
              scheduler = StepLR(optimizer=optimizer,
                                 step_size=config.get("decaying_step_size", 1))
              # scheduler step is executed after every epoch
              scheduler_step_at = "epoch"
          elif config["scheduling"].lower() == "exponential":
              scheduler = ExponentialLR(optimizer=optimizer,
                                        gamma=config.get("decrease_factor", 0.99))
              # scheduler step is executed after every epoch
              scheduler_step_at = "epoch"
          elif config["scheduling"].lower() == "noam":
              factor = config.get("learning_rate_factor", 1)
              warmup = config.get("learning_rate_warmup", 4000)
              scheduler = NoamScheduler(hidden_size=hidden_size,
                                        factor=factor,
                                        warmup=warmup,
                                        optimizer=optimizer)
  
              scheduler_step_at = "step"
          elif config["scheduling"].lower() == "warmupexponentialdecay":
              min_rate = config.get("learning_rate_min", 1.0e-5)
              decay_rate = config.get("learning_rate_decay", 0.1)
              warmup = config.get("learning_rate_warmup", 4000)
              peak_rate = config.get("learning_rate_peak", 1.0e-3)
              decay_length = config.get("learning_rate_decay_length", 10000)
              scheduler = WarmupExponentialDecayScheduler(
                  min_rate=min_rate,
                  decay_rate=decay_rate,
                  warmup=warmup,
                  optimizer=optimizer,
                  peak_rate=peak_rate,
                  decay_length=decay_length)
              scheduler_step_at = "step"
      return scheduler, scheduler_step_at
code_tokens: def build_scheduler ( config : dict , optimizer : Optimizer , scheduler_mode : str , hidden_size : int = 0 ) -> ( Optional [ _LRScheduler ] , Optional [ str ] ) : scheduler , scheduler_step_at = None , None if "scheduling" in config . keys ( ) and config [ "scheduling" ] : if config [ "scheduling" ] . lower ( ) == "plateau" : scheduler = ReduceLROnPlateau ( optimizer = optimizer , mode = scheduler_mode , verbose = False , threshold_mode = 'abs' , factor = config . get ( "decrease_factor" , 0.1 ) , patience = config . get ( "patience" , 10 ) ) scheduler_step_at = "validation" elif config [ "scheduling" ] . lower ( ) == "decaying" : scheduler = StepLR ( optimizer = optimizer , step_size = config . get ( "decaying_step_size" , 1 ) ) scheduler_step_at = "epoch" elif config [ "scheduling" ] . lower ( ) == "exponential" : scheduler = ExponentialLR ( optimizer = optimizer , gamma = config . get ( "decrease_factor" , 0.99 ) ) scheduler_step_at = "epoch" elif config [ "scheduling" ] . lower ( ) == "noam" : factor = config . get ( "learning_rate_factor" , 1 ) warmup = config . get ( "learning_rate_warmup" , 4000 ) scheduler = NoamScheduler ( hidden_size = hidden_size , factor = factor , warmup = warmup , optimizer = optimizer ) scheduler_step_at = "step" elif config [ "scheduling" ] . lower ( ) == "warmupexponentialdecay" : min_rate = config . get ( "learning_rate_min" , 1.0e-5 ) decay_rate = config . get ( "learning_rate_decay" , 0.1 ) warmup = config . get ( "learning_rate_warmup" , 4000 ) peak_rate = config . get ( "learning_rate_peak" , 1.0e-3 ) decay_length = config . get ( "learning_rate_decay_length" , 10000 ) scheduler = WarmupExponentialDecayScheduler ( min_rate = min_rate , decay_rate = decay_rate , warmup = warmup , optimizer = optimizer , peak_rate = peak_rate , decay_length = decay_length ) scheduler_step_at = "step" return scheduler , scheduler_step_at
short_docstring: Create a learning rate scheduler if specified in config and
  determine when a scheduler step should be executed.
short_docstring_tokens: Create a learning rate scheduler if specified in config and determine when a scheduler step should be executed .
comment: """
      Create a learning rate scheduler if specified in config and
      determine when a scheduler step should be executed.
  
      Current options:
          - "plateau": see `torch.optim.lr_scheduler.ReduceLROnPlateau`
          - "decaying": see `torch.optim.lr_scheduler.StepLR`
          - "exponential": see `torch.optim.lr_scheduler.ExponentialLR`
          - "noam": see `joeynmt.builders.NoamScheduler`
          - "warmupexponentialdecay": see
            `joeynmt.builders.WarmupExponentialDecayScheduler`
  
      If no scheduler is specified, returns (None, None) which will result in
      a constant learning rate.
  
      :param config: training configuration
      :param optimizer: optimizer for the scheduler, determines the set of
          parameters which the scheduler sets the learning rate for
      :param scheduler_mode: "min" or "max", depending on whether the validation
          score should be minimized or maximized.
          Only relevant for "plateau".
      :param hidden_size: encoder hidden size (required for NoamScheduler)
      :return:
          - scheduler: scheduler object,
          - scheduler_step_at: either "validation" or "epoch"
      """$SEP$# learning rate scheduler$SEP$# scheduler step is executed after every validation$SEP$# scheduler step is executed after every epoch$SEP$# scheduler step is executed after every epoch