Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update TimeSeries stage to also work with Production DFP #1121

Merged
merged 17 commits into from
Aug 30, 2023

Conversation

efajardo-nv
Copy link
Contributor

@efajardo-nv efajardo-nv commented Aug 4, 2023

Description

  • Update TimeSeries stage to also work with Production DFP pipeline instead of Starter DFP. Now accepts MultiMessage instead of MultiResponseMessage. Remove use of hard-coded event_dt and replace with config.ae.timestamp_column_name.
  • Tested updated TimeSeries stage with Duo and Azure production DFP pipelines.
  • As part of a Starter CloudTrail DFP pipeline, TimeSeries stage runs into issues when processing multiple batches, i.e. when repeat option in CloudTrailSourceStage is increased (increase input data size) or pipeline batch size is decreased. Pipeline either errors out or not all data is processed. Remove TimeSeries stage from AutoEncoder E2E benchmark for now and will create separate issue for this.

Fixes #1107

Checklist

[x] I am familiar with the Contributing Guidelines.
[x] New or existing tests cover these changes.
[x] The documentation is up to date with these changes.

@efajardo-nv efajardo-nv added bug Something isn't working non-breaking Non-breaking change labels Aug 4, 2023
@efajardo-nv efajardo-nv requested a review from a team as a code owner August 4, 2023 22:27
@efajardo-nv efajardo-nv marked this pull request as draft August 8, 2023 13:55
@efajardo-nv efajardo-nv changed the title Remove TimeSeries stage from AutoEncoder benchmark Remove TimeSeries stage from Starter DFP Aug 8, 2023
@efajardo-nv efajardo-nv changed the title Remove TimeSeries stage from Starter DFP Update TimeSeries stage to work with Production DFP instead of Starter DFP Aug 8, 2023
@efajardo-nv efajardo-nv self-assigned this Aug 9, 2023
@efajardo-nv efajardo-nv marked this pull request as ready for review August 9, 2023 19:49
@efajardo-nv efajardo-nv requested a review from a team as a code owner August 9, 2023 19:49
tests/test_dfp_kafka.py Outdated Show resolved Hide resolved
@efajardo-nv efajardo-nv marked this pull request as draft August 10, 2023 20:26
@efajardo-nv efajardo-nv changed the title Update TimeSeries stage to work with Production DFP instead of Starter DFP Update TimeSeries stage to also work with Production DFP Aug 10, 2023
@efajardo-nv efajardo-nv marked this pull request as ready for review August 11, 2023 12:36
morpheus/stages/postprocess/timeseries_stage.py Outdated Show resolved Hide resolved
morpheus/stages/postprocess/timeseries_stage.py Outdated Show resolved Hide resolved
@copy-pr-bot
Copy link

copy-pr-bot bot commented Aug 29, 2023

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@efajardo-nv
Copy link
Contributor Author

/ok to test

@efajardo-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 4d4a7ee into nv-morpheus:branch-23.11 Aug 30, 2023
2 checks passed
@efajardo-nv efajardo-nv deleted the update-ae-benchmark branch July 29, 2024 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working non-breaking Non-breaking change
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

[BUG]: Error from E2E autoencoder benchmark after increasing input size
3 participants