ation file,

# and it is evaluated to `production`@pipeline

def my_pipeline(environment: str):

...

if __name__=="__main__":

my_pipeline.with_options(config_paths="config.yaml")()

There might be conflicting settings for step or pipeline inputs, while working with YAML configuration files. Such situations happen when you define a step or a pipeline parameter in the configuration file and override it from the code later on. Don't worry - once it happens you will be informed with details and instructions how to fix. Example of such a conflict:

# config.yaml

parameters:

some_param: 24

steps:

my_step:

parameters:

input_2: 42

# run.py

from zenml import step, pipeline

@step

def my_step(input_1: int, input_2: int) -> None:

pass

@pipeline

def my_pipeline(some_param: int):

# here an error will be raised since `input_2` is

# `42` in config, but `43` was provided in the code

my_step(input_1=42, input_2=43)

if __name__=="__main__":

# here an error will be raised since `some_param` is

# `24` in config, but `23` was provided in the code

my_pipeline(23)

Parameters and caching

When an input is passed as a parameter, the step will only be cached if all parameter values are exactly the same as for previous executions of the step.

Artifacts and caching

When an artifact is used as a step function input, the step will only be cached if all the artifacts are exactly the same as for previous executions of the step. This means that if any of the upstream steps that produce the input artifacts for a step were not cached, the step itself will always be executed.

See Also:

Use configuration files to set parameters

How caching works and how to control it

PreviousBuild a pipeline

NextStep output typing and annotation

Last updated 15 days ago