prefect.utilities.pydantic

PartialModel

Bases: Generic[M]

A utility for creating a Pydantic model in several steps.

Fields may be set at initialization, via attribute assignment, or at finalization when the concrete model is returned.

Pydantic validation does not occur until finalization.

Each field can only be set once and a ValueError will be raised on assignment if a field already has a value.

Example

class MyModel(BaseModel): x: int y: str z: float

partial_model = PartialModel(MyModel, x=1) partial_model.y = "two" model = partial_model.finalize(z=3.0)

Source code in src/prefect/utilities/pydantic.py

class PartialModel(Generic[M]):
    """
    A utility for creating a Pydantic model in several steps.

    Fields may be set at initialization, via attribute assignment, or at finalization
    when the concrete model is returned.

    Pydantic validation does not occur until finalization.

    Each field can only be set once and a `ValueError` will be raised on assignment if
    a field already has a value.

    Example:
        >>> class MyModel(BaseModel):
        >>>     x: int
        >>>     y: str
        >>>     z: float
        >>>
        >>> partial_model = PartialModel(MyModel, x=1)
        >>> partial_model.y = "two"
        >>> model = partial_model.finalize(z=3.0)
    """

    def __init__(self, __model_cls: Type[M], **kwargs: Any) -> None:
        self.fields = kwargs
        # Set fields first to avoid issues if `fields` is also set on the `model_cls`
        # in our custom `setattr` implementation.
        self.model_cls = __model_cls

        for name in kwargs.keys():
            self.raise_if_not_in_model(name)

    def finalize(self, **kwargs: Any) -> M:
        for name in kwargs.keys():
            self.raise_if_already_set(name)
            self.raise_if_not_in_model(name)
        return self.model_cls(**self.fields, **kwargs)

    def raise_if_already_set(self, name):
        if name in self.fields:
            raise ValueError(f"Field {name!r} has already been set.")

    def raise_if_not_in_model(self, name):
        if name not in self.model_cls.model_fields:
            raise ValueError(f"Field {name!r} is not present in the model.")

    def __setattr__(self, __name: str, __value: Any) -> None:
        if __name in {"fields", "model_cls"}:
            return super().__setattr__(__name, __value)

        self.raise_if_already_set(__name)
        self.raise_if_not_in_model(__name)
        self.fields[__name] = __value

    def __repr__(self) -> str:
        dsp_fields = ", ".join(
            f"{key}={repr(value)}" for key, value in self.fields.items()
        )
        return f"PartialModel(cls={self.model_cls.__name__}, {dsp_fields})"

add_cloudpickle_reduction(__model_cls=None, **kwargs)

add_cloudpickle_reduction(__model_cls: Type[M]) -> Type[M]

add_cloudpickle_reduction(**kwargs: Any) -> Callable[[Type[M]], Type[M]]

Adds a __reducer__ to the given class that ensures it is cloudpickle compatible.

Workaround for issues with cloudpickle when using cythonized pydantic which throws exceptions when attempting to pickle the class which has "compiled" validator methods dynamically attached to it.

We cannot define this utility in the model class itself because the class is the type that contains unserializable methods.

Any model using some features of Pydantic (e.g. Path validation) with a Cython compiled Pydantic installation may encounter pickling issues.

See related issue at https://github.com/cloudpipe/cloudpickle/issues/408

Source code in src/prefect/utilities/pydantic.py

def add_cloudpickle_reduction(__model_cls: Optional[Type[M]] = None, **kwargs: Any):
    """
    Adds a `__reducer__` to the given class that ensures it is cloudpickle compatible.

    Workaround for issues with cloudpickle when using cythonized pydantic which
    throws exceptions when attempting to pickle the class which has "compiled"
    validator methods dynamically attached to it.

    We cannot define this utility in the model class itself because the class is the
    type that contains unserializable methods.

    Any model using some features of Pydantic (e.g. `Path` validation) with a Cython
    compiled Pydantic installation may encounter pickling issues.

    See related issue at https://github.com/cloudpipe/cloudpickle/issues/408
    """
    if __model_cls:
        __model_cls.__reduce__ = _reduce_model
        __model_cls.__reduce_kwargs__ = kwargs
        return __model_cls
    else:
        return cast(
            Callable[[Type[M]], Type[M]],
            partial(
                add_cloudpickle_reduction,
                **kwargs,
            ),
        )

add_type_dispatch(model_cls)

Extend a Pydantic model to add a 'type' field that is used as a discriminator field to dynamically determine the subtype that when deserializing models.

This allows automatic resolution to subtypes of the decorated model.

If a type field already exists, it should be a string literal field that has a constant value for each subclass. The default value of this field will be used as the dispatch key.

If a type field does not exist, one will be added. In this case, the value of the field will be set to the value of the __dispatch_key__. The base class should define a __dispatch_key__ class method that is used to determine the unique key for each subclass. Alternatively, each subclass can define the __dispatch_key__ as a string literal.

The base class must not define a 'type' field. If it is not desirable to add a field to the model and the dispatch key can be tracked separately, the lower level utilities in prefect.utilities.dispatch should be used directly.

Source code in src/prefect/utilities/pydantic.py

def add_type_dispatch(model_cls: Type[M]) -> Type[M]:
    """
    Extend a Pydantic model to add a 'type' field that is used as a discriminator field
    to dynamically determine the subtype that when deserializing models.

    This allows automatic resolution to subtypes of the decorated model.

    If a type field already exists, it should be a string literal field that has a
    constant value for each subclass. The default value of this field will be used as
    the dispatch key.

    If a type field does not exist, one will be added. In this case, the value of the
    field will be set to the value of the `__dispatch_key__`. The base class should
    define a `__dispatch_key__` class method that is used to determine the unique key
    for each subclass. Alternatively, each subclass can define the `__dispatch_key__`
    as a string literal.

    The base class must not define a 'type' field. If it is not desirable to add a field
    to the model and the dispatch key can be tracked separately, the lower level
    utilities in `prefect.utilities.dispatch` should be used directly.
    """
    defines_dispatch_key = hasattr(
        model_cls, "__dispatch_key__"
    ) or "__dispatch_key__" in getattr(model_cls, "__annotations__", {})

    defines_type_field = "type" in model_cls.model_fields

    if not defines_dispatch_key and not defines_type_field:
        raise ValueError(
            f"Model class {model_cls.__name__!r} does not define a `__dispatch_key__` "
            "or a type field. One of these is required for dispatch."
        )

    elif not defines_dispatch_key and defines_type_field:
        field_type_annotation = model_cls.model_fields["type"].annotation
        if field_type_annotation != str:
            raise TypeError(
                f"Model class {model_cls.__name__!r} defines a 'type' field with "
                f"type {field_type_annotation.__name__!r} but it must be 'str'."
            )

        # Set the dispatch key to retrieve the value from the type field
        @classmethod
        def dispatch_key_from_type_field(cls):
            return cls.model_fields["type"].default

        model_cls.__dispatch_key__ = dispatch_key_from_type_field

    else:
        raise ValueError(
            f"Model class {model_cls.__name__!r} defines a `__dispatch_key__` "
            "and a type field. Only one of these may be defined for dispatch."
        )

    cls_init = model_cls.__init__
    cls_new = model_cls.__new__

    def __init__(__pydantic_self__, **data: Any) -> None:
        type_string = (
            get_dispatch_key(__pydantic_self__)
            if type(__pydantic_self__) != model_cls
            else "__base__"
        )
        data.setdefault("type", type_string)
        cls_init(__pydantic_self__, **data)

    def __new__(cls: Type[M], **kwargs: Any) -> M:
        if "type" in kwargs:
            try:
                subcls = lookup_type(cls, dispatch_key=kwargs["type"])
            except KeyError as exc:
                raise ValidationError(errors=[exc], model=cls)
            return cls_new(subcls)
        else:
            return cls_new(cls)

    model_cls.__init__ = __init__
    model_cls.__new__ = __new__

    register_base_type(model_cls)

    return model_cls

get_class_fields_only(model)

Gets all the field names defined on the model class but not any parent classes. Any fields that are on the parent but redefined on the subclass are included.

Source code in src/prefect/utilities/pydantic.py

def get_class_fields_only(model: Type[BaseModel]) -> set:
    """
    Gets all the field names defined on the model class but not any parent classes.
    Any fields that are on the parent but redefined on the subclass are included.
    """
    subclass_class_fields = set(model.__annotations__.keys())
    parent_class_fields = set()

    for base in model.__class__.__bases__:
        if issubclass(base, BaseModel):
            parent_class_fields.update(base.__annotations__.keys())

    return (subclass_class_fields - parent_class_fields) | (
        subclass_class_fields & parent_class_fields
    )

parse_obj_as(type_, data, mode='python')

Parse a given data structure as a Pydantic model via TypeAdapter.

Read more about TypeAdapter here.

Parameters:

Name	Type	Description	Default
type_	type[T]	The type to parse the data as.	required
data	Any	The data to be parsed.	required
mode	Literal['python', 'json', 'strings']	The mode to use for parsing, either python, json, or strings. Defaults to python, where data should be a Python object (e.g. dict).	'python'

Returns:

Type	Description
T	The parsed data as the given type_.

Example

Basic Usage of parse_as

from prefect.utilities.pydantic import parse_as
from pydantic import BaseModel

class ExampleModel(BaseModel):
    name: str

# parsing python objects
parsed = parse_as(ExampleModel, {"name": "Marvin"})
assert isinstance(parsed, ExampleModel)
assert parsed.name == "Marvin"

# parsing json strings
parsed = parse_as(
    list[ExampleModel],
    '[{"name": "Marvin"}, {"name": "Arthur"}]',
    mode="json"
)
assert all(isinstance(item, ExampleModel) for item in parsed)
assert parsed[0].name == "Marvin"
assert parsed[1].name == "Arthur"

# parsing raw strings
parsed = parse_as(int, '123', mode="strings")
assert isinstance(parsed, int)
assert parsed == 123

Source code in src/prefect/utilities/pydantic.py

def parse_obj_as(
    type_: type[T],
    data: Any,
    mode: Literal["python", "json", "strings"] = "python",
) -> T:
    """Parse a given data structure as a Pydantic model via `TypeAdapter`.

    Read more about `TypeAdapter` [here](https://docs.pydantic.dev/latest/concepts/type_adapter/).

    Args:
        type_: The type to parse the data as.
        data: The data to be parsed.
        mode: The mode to use for parsing, either `python`, `json`, or `strings`.
            Defaults to `python`, where `data` should be a Python object (e.g. `dict`).

    Returns:
        The parsed `data` as the given `type_`.


    Example:
        Basic Usage of `parse_as`
        ```python
        from prefect.utilities.pydantic import parse_as
        from pydantic import BaseModel

        class ExampleModel(BaseModel):
            name: str

        # parsing python objects
        parsed = parse_as(ExampleModel, {"name": "Marvin"})
        assert isinstance(parsed, ExampleModel)
        assert parsed.name == "Marvin"

        # parsing json strings
        parsed = parse_as(
            list[ExampleModel],
            '[{"name": "Marvin"}, {"name": "Arthur"}]',
            mode="json"
        )
        assert all(isinstance(item, ExampleModel) for item in parsed)
        assert parsed[0].name == "Marvin"
        assert parsed[1].name == "Arthur"

        # parsing raw strings
        parsed = parse_as(int, '123', mode="strings")
        assert isinstance(parsed, int)
        assert parsed == 123
        ```

    """
    adapter = TypeAdapter(type_)

    if get_origin(type_) is list and isinstance(data, dict):
        data = next(iter(data.values()))

    parser: Callable[[Any], T] = getattr(adapter, f"validate_{mode}")

    return parser(data)