Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Strip HTML & XML tags from RSS feed input #1670

Merged
merged 45 commits into from
May 1, 2024
Merged

Strip HTML & XML tags from RSS feed input #1670

merged 45 commits into from
May 1, 2024

Conversation

dagardner-nv
Copy link
Contributor

@dagardner-nv dagardner-nv commented Apr 29, 2024 •
edited
Loading

Description

  • Optionally strip HTML & XML tags embedded in RSS feeds

Requires PR #1665 to be merged first
Closes #1666

By Submitting this PR I confirm:

  • I am familiar with the Contributing Guidelines.
  • When the PR is ready for review, new or existing tests cover these changes.
  • When the PR is ready for review, the documentation is up to date with these changes.

dagardner-nv and others added 30 commits April 23, 2024 14:23
@dagardner-nv
First pass at a truncate strings method
ab7eaf4
@dagardner-nv
Merge branch 'branch-24.06' of github.com:nv-morpheus/Morpheus into d…
54e7681 
…avid-truncate-milvus-1650
@dagardner-nv
Tests for new truncate_string_cols_by_bytes method and the _cudf_need…
2f14478 
…s_truncate helper
@dagardner-nv
Log a warning when we truncate a column
83ad8ad
@dagardner-nv
Refactor such that each column has it's own max length
f91d9ac
@dagardner-nv
Expand tests
10a68f9
@dagardner-nv
Add in truncating of long string fields.
c80fc27 
In insert_dataframe, perform pandas conversion as late as possible to perform
as many operations in cudf as possible, also only pay the cost of converting
the columns we need to pandas.
@dagardner-nv
Use DataFrameType alias
4b771e8
@dagardner-nv
Cleanup
3b6408e
@dagardner-nv
Exclude string type from max_length checking
9c5435c
@dagardner-nv
Ensure truncate_long_strings parameter is set in configs and passed a…
dbe34dc 
…long to MilvusVectorDBResourceService
@dagardner-nv
Add docstring for truncate_long_strings
9acea9e
@dagardner-nv
Add docstring for warn_on_truncate
078cd19
@dagardner-nv
Merge branch 'branch-24.06' into david-truncate-milvus-1650
00f8170
@dagardner-nv
Add type-alias for Series type
2b72313
@dagardner-nv
Refactor cudf_string_cols_exceed_max_bytes and truncate_string_cols_b…
f26ce68 
…y_bytes per PR feedback [no ci]
@dagardner-nv
Refactor to call cudf_string_cols_exceed_max_bytes prior to convertin…
4a6de1b 
…g to pandas, and call truncate_string_cols_by_bytes after converting to pandas [no ci]
@dagardner-nv
Merge branch 'branch-24.06' of github.com:nv-morpheus/Morpheus into d…
e121b33 
…avid-truncate-milvus-1650 [no ci]
@dagardner-nv
Test WIP [no ci]
5f3804f
@dagardner-nv
Fix bug where truncate was always enabled for cudf dataframes
701e1f9
@dagardner-nv
Finish up tests
38d3125
@dagardner-nv
Remove parametarization based on num_rows, not important for this tes…
a5f05e1 
…t that is already slow
@dagardner-nv
Remove old param
35dc567
@dagardner-nv
Lint fixes
a6cb131
@dagardner-nv
Remove stray print method
148d398
@dagardner-nv
Don't hard-code the name of the probabilities tensor, don't assume it…
d352930 
…s the only tensor
@dagardner-nv
Re-work hard-coded probs->embeddings copy that used to exist in infer…
bc1e3c2 
…ence_stage.py as a stage in the pipeline
@dagardner-nv
Lint fix
078780f
@dagardner-nv
Re-enable C++ mode support
f3d0334
@dagardner-nv
Remove the two issues this PR should resolve
abeb811
@dagardner-nv dagardner-nv added bug Something isn't working dependencies Pull requests that update a dependency file non-breaking Non-breaking change labels Apr 29, 2024
@dagardner-nv dagardner-nv self-assigned this Apr 29, 2024
@dagardner-nv dagardner-nv requested review from a team as code owners April 29, 2024 17:50
@dagardner-nv dagardner-nv marked this pull request as draft April 29, 2024 17:50
@dagardner-nv dagardner-nv added the Merge After Dependencies PR is completed and reviewed but depends on another PR; do not merge out of order label Apr 29, 2024
@dagardner-nv dagardner-nv marked this pull request as ready for review April 29, 2024 18:55
mdemoret-nv
mdemoret-nv approved these changes May 1, 2024
View reviewed changes
morpheus/controllers/rss_controller.py Outdated Show resolved Hide resolved
@dagardner-nv
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 9d3de8a into nv-morpheus:branch-24.06 May 1, 2024
12 checks passed
@dagardner-nv dagardner-nv deleted the david-vdb_upload-strip-tags-1666 branch May 1, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mdemoret-nv mdemoret-nv mdemoret-nv approved these changes

Assignees

@dagardner-nv dagardner-nv

Labels
bug Something isn't working dependencies Pull requests that update a dependency file Merge After Dependencies PR is completed and reviewed but depends on another PR; do not merge out of order non-breaking Non-breaking change
Projects
Morpheus Boards
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[BUG]: vdb_upload pipeline should be stripping html tags from content
2 participants
@dagardner-nv @mdemoret-nv