Using Annotated in attrs

[Dr-ZeeD]

I have been playing with Annotated recently, and was wondering whether this is something that might be used within attrs. My first instinct was "maybe this could be used for validators?". Maybe even field()?

[euresti]

Full disclosure I haven't had a chance to use Annotated yet, though I'm excited about it for my own projects.

Are you thinking of something like:

@attr.define
class MyClass:
   x: int
   y: Annotated[int, Field(validator=...)] = 15

My big concerns of doing it for validators are:

1. Type checking those validators. I don't know that the validator can actually be type-checked by mypy against the int. Though maybe that doesn't matter too much.
2. In 3.10 those will be "string annotations" and so you're more likely to have the get_type_hints issues.

We'd probably not just want to stick attr.ib as the annotation since we probably don't want people sticking default values etc in there.

Reading the Pep a little more it looks like one idea is to have a series of classes rather than one class for the Annotations.

So you could do something like this

But it could be nice to stick the bool arguments in there oh you can even do it as separate "Annotations"

from attr import NoInit, Comparable, define
@attr.define
class MyClass:
   x: Annotated[int, NoInit(), Comparable()]
   y: Annotated[str, Comparable()]  = "wheee"

I can't decide if that's worse or better though.

[uSpike]

Just wanted to chime-in that I have a use-case for adding field configuration in Annotated types. Currently I'm using Annotated to denote parental relationship between objects:

from typing import Annotated, TypeVar
import attr

T = TypeVar("T", bound="Driver")

Parent = Annotated[T, "parent"]

@attr.s
class Driver:
    @property
    def parents(self) -> list[Driver]:
        def _is_parent_type(field: attr.Attribute, value: Any) -> bool:
            return type(field.type) is type(Parent) and "parent" in getattr(field.type, "__metadata__")
        return list(attr.asdict(self, recurse=False, filter=_is_parent_type).values())

@attr.s
class Bus(Driver):
   pass
   
@attr.s
class Device(Driver):
   bus: Parent[Bus] = attr.ib()

bus = Bus()
dev = Device(bus)
assert dev.parents == [bus]

What would be nice is to be able to add extra metadata into Parent so that things like repr=False could be passed into the attr.ib() field:

Parent = Annotated[T, "parent", Field(repr=False)]

[uSpike]

That said, it could even be improved to allow insertion of field metadata via Annotated:

Parent = Annotated[T, Field(repr=False, metadata={"parent": True})]

@attr.s
class Driver:
    @property
    def parents(self) -> list[Driver]:
        def _is_parent_type(field: attr.Attribute, value: Any) -> bool:
            return field.metadata.get("parent", False)
        return list(attr.asdict(self, recurse=False, filter=_is_parent_type).values())

[AdrianSosic]

I just also wanted to briefly express my interest in the possibility of using Annotated. As mentioned in #1275, I oftentimes find myself in situations where I have several attributes that have the exact same type hint + field definition, i.e. they more or less represent the same semantic meaning. In these cases, it would be nice if one could simply define this "meaning" as a type upfront and reuse the type across the different contexts where it is needed. AFAIK that's the primary use case for which pydantic uses Annotated.

[hynek]

Skimming the comments, do I see it correctly that there's somewhat a consensus to get rid of the special-cased x: int = field(validator=le_validator) / = attr.ib and use x: Annotated[int, Field(validator=le_validator))]?

Generally speaking, that seems like a good way to make the typing situation less about lying to the type checker.

Open questions for me are:

• is Annotated meant for that? I've seen some rescinding of feature in Pyright around that
• is it possible to express any of this using dataclass transforms?

cc @Tinche

[AdrianSosic]

Hi @hynek, just to make my point clear: I'm not suggesting to get rid of the good old field assignment approach. But using typing.Annotated in addition would allow to not only assign a "concept" to an attribute of one particular class but rather encapsulate this "concept" in the form of its own reusable semantic object.

Concrete example

Ideally, I would like to capture the semantics of a PrimeNumber in its own type and then outsource the parts of the field definition that are necessary to describe that specific meaning (i.e., the corresponding validators, converters, ...) to Annotated.

So instead if having to repeat the following field assignment

@define
class NeedsPrimeNumber:
    prime: int = field(default=3, validator=is_prime, converter=le_converter)


@define
class AlsoNeedsPrimeNumber:
    prime: int = field(default=5, validator=is_prime, converter=le_converter)

I'd rather want to write

PrimeNumber = Annotated[int, Field(validator=is_prime, converter=le_converter)]


@define
class UsingAnnotated:
    prime: PrimeNumber = field(default=3)

@define
class AlsoUsingAnnotated:
    prime: PrimeNumber = field(default=5)

where in this case only the default (which is class specific) remains part of the regular assignment.

Perhaps this was already clear to you, just thought a specific example could help 🙃

[dlax]

@AdrianSosic, in your example, I believe the right-side field(default=...) is superfluous and the example should rather lean towards:

PrimeNumber = Annotated[int, Field(validator=is_prime, converter=le_converter)]

@define
class UsingAnnotated:
    prime: PrimeNumber = 3

because, otherwise, the library has to tell type checkers that field(default=...) means the field has a default value; whereas, as written above, there's nothing to do on our side.

---

@hynek

do I see it correctly that there's somewhat a consensus to get rid of the special-cased x: int = field(validator=le_validator) / = attr.ib and use x: Annotated[int, Field(validator=le_validator))]?

IMHO, yes!

[AdrianSosic]

Hi @dlax 👋🏼 Probably, setting a default was not the best example from my side. But what I meant is that there could be other field-related setting that do not belong the the concept of a PrimeNumber and which I would like to set outside the PrimeNumber annotation.

For example:

PrimeNumber = Annotated[int, Field(validator=is_prime, converter=le_converter)]

@define
class RequiresPrimeLargerThanFive:
    prime: PrimeNumber = field(default=7, validator=gt(5))

So the specific class here needs a prime larger than five, but primes in general don't have this requirement.

Or are you suggesting you use a second layer of Annotated in this case, a la:

PrimeNumber = Annotated[int, Field(validator=is_prime, converter=le_converter)]

@define
class RequiresPrimeLargerThanFive:
    prime: Annotated[PrimeNumber, validator=gt(5)] = 7

with the general idea of moving every functionality of field to Annotated, i.e. also stuff like kw_only, eq, on_setattr, etc?

[dlax]

Yes, one key aspect of Annotated is that you can combine multiple pieces of metadata. But in your example, I might be strange to allow several Field values in Annotated, so perhaps we'd need dedicated metadata for validation/conversion:

PrimeNumber = Annotated[int, Validator(is_prime), Converter(le_converter)]

@define
class RequiresPrimeLargerThanFive:
    prime: Annotated[PrimeNumber, Validator(gt(5))] = 7

and keep the Field for things that are meant to be specified only once (like eq, kw_only, etc.)?

[LecrisUT]

It seems in the meantime cattrs got support for Annotated first 🙂.

Anyway, let's talks shop a bit more. Afaiu in order for attrs to know how to handle the metadata fields, it would need to be a defined class so that both first-party metadata Field, Validator, Converter etc. and third-party metadata would be put in the right place in attrib. This might actually be a good time for some consolidation and unification of the various function and classes. One potential interface is like

@attrs.define
class AttrsMetadata:
    _metadata_key: ClassVar[key]
    """The metadata variable where this eventually lands"""

@attrs.define
class CustomMetadata(AttrsMetadata):
    _metadata_key = "my_extension"
    factor: int

@attrs.define
class AttrsFieldMetadata:
    _metadata_key = "_attrs.field"
    default: Any = NOTHING
    validator: Callable[..., Any] | None = None
    ...

@attrs.define
class AttrsValidatorMetadata:
    _metadata_key = "_attrs.validator"
    validator: Callable[..., Any] | None = None

    @classmethod
    def from_func(cls, func: Callable[..., Any], pass_self: bool = False, pass_field: bool = false): ...

@attrs.define
class Example:
    var: Annotated[int, AttrsFieldMetadata(default=0), AttrsValidatorMetadata(lambda x: x>0), CustomMetadata(factor=10)]

Some important points to be discussed:

• How to unify duplicate metadata that point to the same key. One option would be to use the dict.update strategy and override for some special behavior, e.g. the validators being chained with and_.
• Type-checking support of the metadata itself. The way forward seems to be PEP746. It is not fool-proof though, e.g. it would not work for Converter that chain the return types
• Should this be documented as a recommended way forward? I believe yes because it offers a simpler way for handling the typechecking, particularly for third-party extensions
• Naming is hard, better suggestions welcome
• How to start handling this? From the perspective of a third-party extender, I would like this code path to be a bit clearer for them, so my suggestion here is to start small and first figure out how to handled custom metadata field, then we could work on gluing the attrs metadata on top of it. What is nice is that cattrs already seems to have done it, but a definition on the first-party interface is missing to be more generic

[astrojuanlu]

Given that stdlib dataclasses does not support this pattern, would be nice if attrs gained it 👍🏼