Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch incoming DOCA raw packet data #1731

Merged
merged 103 commits into from
Aug 1, 2024
Merged

Batch incoming DOCA raw packet data #1731

merged 103 commits into from
Aug 1, 2024

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Jun 7, 2024 •
edited
Loading

Description

  • Fix bug in DocaSourceStage ring buffer initialization ([BUG]: DocaSourceStage missing and duplicate packets #1820)
  • Incoming packet data is buffered into a mrc::BufferedChannel, limiting the amount of times the convert stage needs to acquire the GIL.
  • Launch the _packet_gather_payload_kernel kernel with a 2D grid, treating the byte-offset as the second axis.
  • No longer convert 32bit int IP data into strings in the gather kernel, instead buffer 32bit int data, and let cuDF convert the data all in one big pass prior to constructing the cuDF DataFrame.
  • No longer outputs fixed data sizes.
  • Perform the sizes to offsets calculation with MatX rather than using a deprecated cuDF method.
  • Add PacketDataBuffer struct to wrap the three pieces of data needed to be buffered: header, payload and payload sizes.
  • Split doca_stages.hpp into doca_source_stage.hpp and doca_convert_stage.hpp
  • Rename doca_convert.cpp->doca_convert_stage.cpp and doca_source.cpp->doca_source_stage.cpp
  • Add command line flags to examples/doca/run_udp_convert.py and examples/doca/vdb_realtime/sender/send.py
  • Move constants in morpheus/_lib/doca/include/morpheus/doca/common.hpp into the morpheus::doca namespace (was in global).
  • Fix CI checks for DOCA based code
  • Remove unused code

Closes #1820

This PR punts on fixing #1827

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

@dagardner-nv
Set delayed_start=True for MonitorStage, add a debug stage to print t…
8dff5a2 
…he number of rows in each MessageMeta
@dagardner-nv
Incoming RawPacket messages often contain a small number of packets, …
9032a3f 
…typically only a single packet. Change the convert stage to build up incoming packet data prior to building a MessageMeta
@dagardner-nv
Hack send.py to continously send, and to send packets in bulk
cee1a33
@dagardner-nv
Merge branch 'branch-24.06' of https://github.com/nv-morpheus/Morpheus …
9a551dc 
…into david-doca-batched-tables
@dagardner-nv
Remove concurrency setting
d5ceba2
@dagardner-nv
Merge branch 'david-doca-batched-tables' of github.com:dagardner-nv/M…
6aee2fc 
…orpheus into david-doca-batched-tables
@dagardner-nv
WIP
2fa6f93
@dagardner-nv
WIP
97af015
@dagardner-nv
WIP
4ef8650
@dagardner-nv
WIP
81f58f8
@dagardner-nv
WIP
821a246
@dagardner-nv
sizes to offsets not working as expected
273c6e8
@dagardner-nv
Fix buffering
ab014ec
@dagardner-nv dagardner-nv added non-breaking Non-breaking change DO NOT MERGE PR should not be merged; see PR for details skip-ci Optionally Skip CI for this PR labels Jun 7, 2024
@dagardner-nv dagardner-nv self-assigned this Jun 7, 2024
@dagardner-nv dagardner-nv requested a review from a team as a code owner June 7, 2024 02:06
@copy-pr-bot copy-pr-bot
Copy link

copy-pr-bot bot commented Jun 7, 2024

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@dagardner-nv dagardner-nv changed the base branch from branch-24.06 to branch-24.10 July 5, 2024 15:42
@dagardner-nv dagardner-nv removed the skip-ci Optionally Skip CI for this PR label Jul 31, 2024
@dagardner-nv dagardner-nv marked this pull request as ready for review July 31, 2024 16:44
@dagardner-nv
Copy link
Contributor Author

/ok to test

@dagardner-nv
Copy link
Contributor Author

/ok to test

mdemoret-nv
mdemoret-nv approved these changes Jul 31, 2024
View reviewed changes
morpheus/_lib/doca/CMakeLists.txt Outdated Show resolved Hide resolved
morpheus/_lib/doca/CMakeLists.txt Outdated Show resolved Hide resolved
morpheus/_lib/doca/src/doca_convert_stage.cpp Show resolved Hide resolved
morpheus/stages/doca/doca_convert_stage.py Outdated Show resolved Hide resolved
morpheus/stages/doca/doca_convert_stage.py Outdated Show resolved Hide resolved
@dagardner-nv
Copy link
Contributor Author

/ok to test

@dagardner-nv
Copy link
Contributor Author

/ok to test

@dagardner-nv
Copy link
Contributor Author

/ok to test

@dagardner-nv
Copy link
Contributor Author

/ok to test

@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 68bbaa9 into nv-morpheus:branch-24.10 Aug 1, 2024
11 checks passed
@dagardner-nv dagardner-nv deleted the david-doca-batched-tables-p2 branch August 1, 2024 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG]: DocaSourceStage missing and duplicate packets
2 participants
@dagardner-nv @mdemoret-nv