Support generic type hints

[danielgafni]

What's the use case?

Currently, Dagster's internal typing does not support generics.
At least, it's not possible to automatically resolve DagsterType objects from type hints with generics.

I am working on integrating pandera validation library with dagster-polars. Pandera provides generic type annotations for DataFrames, but Dagster can't accept them without manually constructed DagsterType objects, which makes our life harder.

Example of what I want to be able to write:

from dagster import asset
import pandera
import pandera.typing.polars
import polars as pl

class MySchema(pandera.DataFrameModel):
    foo: str
    bar: int


@asset(io_manager_key="polars_parquet_io_manager")
def upstream() -> pandera.typing.polars.DataFrame[MySchema]:
    return pl.DataFrame({"foo": ["a", "b", "c"], "bar": [1, 2, 3]})


@asset(io_manager_key="polars_parquet_io_manager")
def downstream(upstream: pandera.typing.polars.LazyFrame[MySchema]):
    ...

Currently, this core produces the following error:

E           dagster._core.errors.DagsterInvalidDefinitionError: Problem using type 'pandera.typing.polars.LazyFrame[dagster_polars_tests.test_upath_io_managers.MySchema]' from type annotation for argument 'upstream', correct the issue or explicitly set the dagster_type via In().

Ideas of implementation

Apart from simple changes to automatic DagsterType creation, we need to figure out if generics should be a first-class citizen (e.g. be aware of the generic type parameter, represent it in the UI, etc) or rather rely on some work-around and build on top of one of existing types (like Any). I'm not sure how exactly this can be done.

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.