Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add source & stage decorators #1364

Merged
merged 116 commits into from
Nov 29, 2023

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Nov 15, 2023

Description

  • Adds two new decorators source and stage allowing stages to be created from python functions.
  • Accept & return types are inferred by inspecting the type annotations of the decorated function.
  • This removes the need for several lines of boilerplate code needed for simple Python-only stages.
  • Includes unrelated docstring fix to morpheus/pipeline/stage_base.py

example:

from morpheus.pipeline.stage_decorator import source
from morpheus.pipeline.stage_decorator import stage

@source
def source_gen(dataframes: list[cudf.DataFrame]) -> collections.abc.Iterator[MessageMeta]:
    for df in dataframes:
        yield MessageMeta(df)


@stage
def on_data(message: MessageMeta) -> MessageMeta:
    with message.mutable_dataframe() as df:
        print(df)
    return message

Closes #1253

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

…ith MultiMessage however the Python impl accepts typing.Any. Re-work example to only use the C++ impl when the input type is a MultiMessage [no ci]
… first where the Python impl will be returned, the second where the C++ impl is returned [no ci]
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good start but we need some changes to be forward looking with these decorators. General changes:

  1. The classes used by the decorators should be "dumber" and move most of the implementation into the decorator instead of the class. See my comments for more details.
  2. Need some tests validating the parameters used at construction time
  3. Needs documentation showing how this can be used in the developer guide.

morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Show resolved Hide resolved
morpheus/pipeline/stage_decorator.py Outdated Show resolved Hide resolved
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now. Like the changes. This will be super useful in the future.

@mdemoret-nv
Copy link
Contributor

/merge

1 similar comment
@mdemoret-nv
Copy link
Contributor

/merge

@rapids-bot rapids-bot bot merged commit ba28cff into nv-morpheus:branch-23.11 Nov 29, 2023
7 checks passed
@dagardner-nv dagardner-nv deleted the david-stage-decorator branch November 30, 2023 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[FEA]: Create a stage decorator allowing stages to be created from Python functions
3 participants