v24.06.00

Morpheus 24.06.00 (03 Jul 2024)

🚨 Breaking Changes

• Introduce multi-endpoint servers and health check endpoints to HttpServerSourceStage (#1734) @jadu-nv
• Update devcontainer to use latest build utils (#1658) @cwharris
• Update CI to install DOCA and build Morpheus DOCA components. (#1622) @cwharris
• Support non-json serializable objects in LLMContext (#1589) @dagardner-nv

🐛 Bug Fixes

• Fix LLM Agents Kafka pipeline (#1793) @dagardner-nv
• Add fetch data command in build_container_release.sh (#1787) @ifengw-nv
• Add cuda and cudf to link targets for C++ examples (#1777) @dagardner-nv
• Release container fixes (#1766) @dagardner-nv
• Remove setting of prog_name, this implies that an executable named morpheus_llm exists (#1759) @dagardner-nv
• Provide a default set of questions for the standalone RAG pipeline (#1758) @dagardner-nv
• Disable shared memory by default, and fix --stop_after flag for vdb_upload example (#1755) @dagardner-nv
• Fix conda errors in release container (#1750) @dagardner-nv
• Fixes for C++ impl for DeserializeStage and add missing get_info overloads to SlicedMessageMeta (#1749) @dagardner-nv
• Add GIT_CLONE_PROTECTION_ACTIVE env config to fix build script (#1748) @jadu-nv
• Fix triton multi threading when using the C++ stage (#1739) @mdemoret-nv
• resolve rapids-dependency-file-generator warning (#1735) @jameslamb
• Updating all uses of the secrets.PROJECT_MANAGEMENT_PAT to use a registered Github App (#1730) @mdemoret-nv
• DOCA: fix optional deps + remove PreallocatorMixin from source stage (#1729) @e-ago
• Remove pyarrow_hotfix import from __init__.py (#1692) @efajardo-nv
• Support the filter_null parameter in the C++ impl of the FileSourceStage (#1689) @dagardner-nv
• Enable C++ mode for abp_pcap_detection example (#1687) @dagardner-nv
• Strip HTML & XML tags from RSS feed input (#1670) @dagardner-nv
• Truncate strings exceeding max_length when inserting to Milvus (#1665) @dagardner-nv
• Fix a typo in the devcontainer base image (#1638) @cwharris
• Fix tests to detect issue #1626 (#1629) @dagardner-nv
• Fix cupy_to_tensor to also infer uint8 and int8 dtypes (#1621) @efajardo-nv
• Add struct column support to cudf_helpers (#1538) @efajardo-nv

📖 Documentation

• Cleanup docs so that each as a single H1 title (#1794) @dagardner-nv
• Mark the LLM Agents Kafka pipeline as broken (#1792) @dagardner-nv
• Document supported environments for each example (#1786) @dagardner-nv
• Removes unused environment variables from Morpheus build docs (#1784) @yczhang-nv
• Remove documentation for yaml config files in vdb_upload pipeline until #1752 is resolved (#1778) @dagardner-nv
• Remove out of date instructions from contributing.md (#1774) @dagardner-nv
• Add troubleshooting reference for unsuccessful documentation builds (#1768) @ifengw-nv
• Remove pre-built container section from getting_started.md (#1764) @yczhang-nv
• Clarify Documentation: Run fetch_data.py Outside Docker Container (#1762) @ifengw-nv
• Add function return documentation for LLMService (#1721) @acaklovic-nv
• Fix description for cache_mode option of DFP Rolling Window module (#1707) @efajardo-nv
• Update root-cause-analysis-model-card.md (#1684) @HesAnEasyCoder
• Update abp-model-card.md (#1683) @HesAnEasyCoder
• Update dfp-model-card.md (#1682) @HesAnEasyCoder
• Update gnn-fsi-model-card.md (#1681) @HesAnEasyCoder
• Update phishing-model-card.md (#1680) @HesAnEasyCoder
• Update examples to execute from the root of the repo (#1674) @dagardner-nv
• Update dfp-model-card.md (#1644) @HesAnEasyCoder

🚀 New Features

• Add ransomware model to devcontainer (#1785) @yczhang-nv
• Introduce multi-endpoint servers and health check endpoints to HttpServerSourceStage (#1734) @jadu-nv
• Support ControlMessage for Preprocess and PostProcess stages (#1623) @yczhang-nv
• Update CI to install DOCA and build Morpheus DOCA components. (#1622) @cwharris
• DOCA stage split: source + convert (#1617) @e-ago
• ControlMessage support in TritonInferenceStage and PreallocatorMixin (#1610) @cwharris

🛠️ Improvements

• Merge Agent Morpheus changes (#1760) @dagardner-nv
• Fix RabbitMQ output path (#1756) @yczhang-nv
• Misc improvements for sid_visualization example (#1751) @dagardner-nv
• Auditing the dependencies and syncing dependencies.yaml with meta.yaml (#1743) @mdemoret-nv
• Always run the PR builder step even if others are cancelled (#1736) @mdemoret-nv
• Enable Python install by default in compile.sh (#1724) @dagardner-nv
• Generate deprecation warning for MultiMessage (#1719) @yczhang-nv
• Improve the logging tests and add support for resetting the logger (#1716) @mdemoret-nv
• Allow passing metadata to LangChainAgentNode._run_single (#1710) @dagardner-nv
• Support passing a custom parser to HttpServerSourceStage and HttpClientSourceStage stages (#1705) @dagardner-nv
• Use EnvConfigValue for passing env-configured arguments to services (#1704) @cwharris
• Remove unused MLflow client arg from DFP inference implementations (#1700) @efajardo-nv
• Add group by column stage (#1699) @dagardner-nv
• Fix non-deterministic output of gnn sampler (#1677) @tzemicheal
• Ensuring consistent use of the export macro MORPHEUS_EXPORT (#1672) @aserGarcia
• Update devcontainer to use latest build utils (#1658) @cwharris
• Update ControlMessage to hold arbitrary Python objects & update MessageMeta to copy & slice (#1637) @yczhang-nv
• Use conda env create --yes instead of --force (#1636) @efajardo-nv
• Misc CI improvements (#1618) @dagardner-nv
• Support non-json serializable objects in LLMContext (#1589) @dagardner-nv

[NIGHTLY] v24.10.00

🔗 Links

• Development Branch
• Compare with main branch

🐛 Bug Fixes

• Ensure proper initialization of CMAKE_INSTALL_PREFIX if needed (#1815) @dagardner-nv
• Fix merging of CLI args and Yaml configs in vdb_upload example (#1813) @dagardner-nv
• Fix log parsing undefined variable and duplicate sequence id errors (#1809) @dagardner-nv
• Remove obsolete version string from compose yamls (#1808) @dagardner-nv
• Update ci/release/update-version.sh to include missed files (#1801) @dagardner-nv

📖 Documentation

• Fix documentation links to work in both source repo and documentation builds (#1814) @dagardner-nv
• Update documentation for vdb_upload to use realistic source data with the --file_source flag (#1800) @dagardner-nv

🚀 New Features

• Create a Docker image for Morpheus models (#1804) @dagardner-nv
• Add unique column to output of the log_parsing pipeline (#1795) @dagardner-nv

🛠️ Improvements

• Install built documentation into release container (#1806) @dagardner-nv

v24.03.02

🐛 Bug Fixes

• Don't set pe_count for the C++ impl of the TritonInferenceStage (#1640) @dagardner-nv
• Fix vdb_upload runtime error (#1643) @dagardner-nv

📖 Documentation

• Document current known issues in 24.03.02 (#1656) @dagardner-nv
• Fix documentation for building examples (#1659) @dagardner-nv
• Fix type-o in documentation (#1662) @dagardner-nv
• Fix mis-spelling of examples (#1664) @dagardner-nv

⚠️ Known Issues

• TrainAEStage fails with a Segmentation fault (#1641)
• vdb_upload example pipeline triggers an internal error in Triton (#1649)
• vdb_upload example pipeline error on inserting large strings (#1650)
• vdb_upload example pipeline only works with C++ mode disabled (#1651)

v24.03.01

🚨 Breaking Changes

• Move MemoryDescriptor to the morpheus namespace (#1602) @dagardner-nv

🐛 Bug Fixes

• Switch to kafka 3.5.2 (#1612) @dagardner-nv
• Update mlflow to avoid CVE-2024-27132 and CVE-2024-27133 (#1609) @dagardner-nv
• Fix for databricks_cli import error (#1604) @dagardner-nv
• Move MemoryDescriptor to the morpheus namespace (#1602) @dagardner-nv

v24.03.00

🚨 Breaking Changes

• Updating nlohman_json to 3.11 to match MRC (#1596) @mdemoret-nv
• Add retry logic and proxy support to the NeMo LLM Service (#1544) @mdemoret-nv
• Upgrade openai version to 1.13 and langchain to version 0.1.9 (#1529) @mdemoret-nv
• Make start_async() available to source stages (#1523) @efajardo-nv
• RAPIDS 24.02 Upgrade (#1468) @cwharris
• Decouple TritonInferenceStage from pipeline mode (#1402) @dagardner-nv

🐛 Bug Fixes

• Serialize datetime objects into the module config (#1592) @dagardner-nv
• Remove the defaults channel from dependencies.yml (#1584) @mdemoret-nv
• Fix iso_date_regex_pattern config in file_batcher module and allow override (#1580) @efajardo-nv
• Update DFP MLflow ModelManager to handle model retrieval using file URI (#1578) @efajardo-nv
• Fix configure_logging in DFP benchmarks (#1553) @efajardo-nv
• Catch langchain agent errors (#1539) @dagardner-nv
• Adding missing dependency on pydantic (#1535) @yuchenz427
• Fix memory leak in the mutable dataframe checkout/checkin code (#1534) @dagardner-nv
• Fix pathlib.Path support for FileSourceStage (#1531) @dagardner-nv
• Make start_async() available to source stages (#1523) @efajardo-nv
• Update CI Containers (#1521) @cwharris
• Fix intermittent segfault on interpreter shutdown (#1513) @dagardner-nv
• Adopt updated builds of CI runners (#1503) @dagardner-nv
• Update mlflow plugin version for deployments fix (#1499) @pdmack
• Add runtime environment output to fix building the release container (#1496) @cwharris
• Fix logging of sleep time (#1493) @dagardner-nv
• Pin pytest to <8 (#1485) @dagardner-nv
• Improve pipeline stop logic to ensure join is called exactly once for all stages (#1479) @efajardo-nv
• Fix expected JSON config file extension in logger (#1471) @efajardo-nv
• Fix Loss Function to Improve Model Convergence for AutoEncoder (#1460) @hsin-c
• GNN fraud detection notebook fix (#1450) @efajardo-nv
• Eliminate Redundant Fetches in RSS Controller (#1442) @bsuryadevara
• Updating the workspace settings to remove deprecated python options (#1440) @mdemoret-nv
• Improve camouflage startup issues (#1436) @dagardner-nv
• Fixes to modular DFP examples and benchmarks (#1429) @efajardo-nv

📖 Documentation

• Update minimum compute requirements to Volta (#1594) @dagardner-nv
• Fix broken link in getting started with Morpheus doc (#1494) @edknv
• Update abp-model-card.md (#1439) @drobison00
• Update gnn-fsi-model-card.md (#1438) @drobison00
• Update phishing-model-card.md (#1437) @drobison00
• Document incompatible mlflow models issue (#1434) @dagardner-nv

🚀 New Features

• Adding retry logic to the TritonInferenceStage to allow recovering from errors (#1548) @cwharris
• Create a base mixin class for ingress & egress stages (#1473) @dagardner-nv
• RAPIDS 24.02 Upgrade (#1468) @cwharris
• Install headers & morpheus-config.cmake (#1448) @dagardner-nv

🛠️ Improvements

• Updating nlohman_json to 3.11 to match MRC (#1596) @mdemoret-nv
• DOCA 2.6 from public repo (#1588) @e-ago
• Support ControlMessage for PreProcessNLPStage PreProcessFILStage AddScoreStageBase (#1573) @yuchenz427
• Update MLflow in Production DFP example to use Python 3.10 (#1572) @efajardo-nv
• Fix environment yaml paths (#1551) @efajardo-nv
• Add retry logic and proxy support to the NeMo LLM Service (#1544) @mdemoret-nv
• Update to match new MRC function sig for AsyncioRunnable::on_data (#1541) @dagardner-nv
• Expose max_retries parameter to OpenAIChatService & OpenAIChatClient (#1536) @dagardner-nv
• Upgrade openai version to 1.13 and langchain to version 0.1.9 (#1529) @mdemoret-nv
• Update ops-bot.yaml (#1528) @AyodeAwe
• Add the ability to attach Tensor objects and timestamps to ControlMessage (#1511) @drobison00
• Fix or silence warnings emitted during tests (#1501) @dagardner-nv
• Support ControlMessage output in the C++ impl of DeserializeStage (#1478) @dagardner-nv
• DOCA Source Stage improvements (#1475) @e-ago
• Update copyright headers for 2024 (#1474) @efajardo-nv
• Add conda builds to CI (#1466) @dagardner-nv
• Grafana log monitoring and error alerting example (#1463) @efajardo-nv
• Misc Conda Improvements (#1462) @dagardner-nv
• Simplification of the streaming RAG ingest example to improve usability (#1454) @drobison00
• Replace GPUtil with pynvml for benchmark reports (#1451) @efajardo-nv
• Misc test improvements (#1447) @dagardner-nv
• Add a --manual_seed flag to the CLI (#1445) @dagardner-nv
• Optionally skip ci based on a label in the pr (#1444) @dagardner-nv
• Refactor verification of optional dependencies (#1443) @dagardner-nv
• Use dependencies.yaml as source-of-truth for environment files. (#1441) @cwharris
• Add mocked test & benchmark for LLM agents pipeline (#1424) @dagardner-nv
• Add benchmarks for stand-alone RAG & vdb upload pipelines (#1421) @dagardner-nv
• Add benchmark for completion pipeline (#1414) @dagardner-nv
• Decouple TritonInferenceStage from pipeline mode (#1402) @dagardner-nv

Full Changelog: v23.11.01...v24.03.00

v23.11.01

🐛 Bug Fixes

• Convert models/ransomware-models/ransomw-model-short-rf-20220126.sav to LFS (#1408) @mdemoret-nv

📖 Documentation

• Cloud deployment guide fixes (#1406) @dagardner-nv

Full Changelog: v23.11.00...v23.11.01

v23.11.00

🚨 Breaking Changes

• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv

🐛 Bug Fixes

• CVE-2023-47248 Mitigation (#1399) @mdemoret-nv
• Fixing the hammah and phishing validation pipelines (#1398) @mdemoret-nv
• Fix the SID Viz workflow shutdown process with the new pipeline shutdown process (#1392) @mdemoret-nv
• Fix race condition in the C++ impl for the pre-process fil stage (#1390) @dagardner-nv
• Fixing the conda-build with DOCA enabled and upgrading to CMake 3.25 (#1386) @mdemoret-nv
• Add missing milvus marker (#1385) @dagardner-nv
• Register DataBricksDeltaLakeSourceStage with the CLI (#1384) @dagardner-nv
• Guard optional dependencies in try/except blocks (#1382) @dagardner-nv
• Fix test_vdb_upload_pipe (#1381) @dagardner-nv
• DFP container updates (#1347) @efajardo-nv
• Removed Mutex Related Milvus Tests (#1325) @bsuryadevara
• Pin cuda-python to 11.8.2 as a work around for 11.8.3 incompatibility. (#1320) @drobison00
• Forward-merge branch-23.07 to branch-23.11 [resolved conflicts] (#1246) @dagardner-nv
• Fix tests to support mlflow v2.7 (#1220) @dagardner-nv
• Update DFP training model_kwargs (#1216) @efajardo-nv
• Fix Kafka offset checking test (#1212) @dagardner-nv
• Add review_requested as a trigger & increased timeouts for camouflage (#1200) @dagardner-nv
• Remove multiprocess download option (#1189) @efajardo-nv
• Update feature length for test_abp_fil_e2e benchmark (#1188) @dagardner-nv
• Make manual_seed an autouse fixture for gnn_fraud_detection_pipeline tests (#1165) @dagardner-nv
• Update Downloader utility class to use static dask cluster (#1161) @efajardo-nv
• Update to handle GitHub CLI not installed (#1157) @efajardo-nv
• Update TimeSeries stage to also work with Production DFP (#1121) @efajardo-nv
• Fix issue where DFPFileToDataFrameStage logs messages about S3 even when S3 is not in use (#1120) @dagardner-nv
• Fix Tests broken by Test Reorganization (#1118) @cwharris
• Break circular reference issue causing a memory leak (#1115) @dagardner-nv
• Fix intermittent failures in test_dfencoder_distributed_e2e test (#1113) @dagardner-nv
• Resolve forward merger conflices for branch-23.11 (#1092) @mdemoret-nv
• Fix local CI building from a specific commit (#1083) @dagardner-nv

📖 Documentation

• Grafana example readme update (#1393) @efajardo-nv
• Align model card requirements (#1388) @drobison00
• Docs update to indicate use of conda-merge to generate install files (#1387) @drobison00
• Stage documentation improvements (#1362) @dagardner-nv
• Documentation patch for Examples (#1357) @pranavm7
• Update developer documentation to reflect new compute_schema changes (#1341) @dagardner-nv
• Create LICENSE.psycopg2 (#1295) @exactlyallan
• Fix documentation for morpheus.loaders.sql_loader (#1264) @dagardner-nv
• Phishing example fix (#1215) @efajardo-nv
• ABP PCAP detection readme update (#1205) @efajardo-nv
• Command line examples for module-based DFP pipelines (#1154) @efajardo-nv
• Update DFP E2E Benchmarks README to use dev container (#1125) @efajardo-nv
• Less intrusive doc builds (#1060) @dagardner-nv

🚀 New Features

• Add source & stage decorators (#1364) @dagardner-nv
• Add a Vector Database Service to allow stages to read and write to VDBs (#1225) @bsuryadevara
• CI test stage no longer depends on build stage (#1219) @dagardner-nv
• Updates for MRC/Morpheus to build in the same RAPIDS devcontainer environment (#1171) @cwharris
• KafkaSourceStage OAuth Callback Support (#1169) @cwharris
• GitHub Project Automation and Infra Updates (#1168) @jarmak-nv
• Elasticsearch Sink Module (#1163) @bsuryadevara
• RSS Source Stage for Reading RSS Feeds (#1149) @bsuryadevara
• Add parser_kwargs to FileSourceStage to support json files (#1137) @cwharris
• Add a --viz_direction flag to CLI (#1119) @dagardner-nv
• Adds support to read and write to Databricks delta tables (#630) @pthalasta

🛠️ Improvements

• LLM C++ test and doc updates (#1379) @efajardo-nv
• Merge fea-sherlock feature branch into branch-23.11 (#1359) @drobison00
• Make dfp_azure_pipeline inference output file configurable. (#1290) @drobison00
• Loosen nodejs version restriction (#1262) @dagardner-nv
• Use conda environment yaml's for training-tuning-scripts (#1256) @efajardo-nv
• Cherry pick to pull in august DFP enhancements (#1248) @drobison00
• [DRAFT] Add model and experiment template 'click' options to dfp example pipelines, and make model names Databricks compatible. (#1245) @drobison00
• Separate Pipeline type inference/checking & MRC pipeline construction (#1233) @dagardner-nv
• Adopt updated camouflage-server & fix test_dfp_mlflow_model_writer (#1195) @dagardner-nv
• Add development deps to dependencies.yaml (#1193) @cwharris
• Update to clang-16 & boost-1.82 (#1186) @dagardner-nv
• Scope Zookeeper & Kafka fixtures to session (#1160) @dagardner-nv
• Use copy-pr-bot (#1159) @ajschmidt8
• Update dfp_training stage to support ControlMessages or MultiDFPMessages (#1155) @drobison00
• Prefer conda package over pip dependencies (#1135) @cwharris
• Add tasks and metadata properties to python ControlMessage (#1134) @cwharris
• Eliminate redundant code blocks in modules and stages (#1123) @bsuryadevara
• update devcontainer base to 23.10 (#1116) @cwharris
• Slimmed down CI runners and published artifact urls (#1112) @dagardner-nv
• Updating tests to force .pyi files to be committed into the repo (#1111) @mdemoret-nv
• ABP nvsmi sample data generation (#1108) @efajardo-nv
• Reorganize C++ Tests (#1095) @cwharris
• Improve gitutils.py by using the Github CLI when available (#1088) @mdemoret-nv
• Fixing linting errors which could not be resolved in 23.07 (#1082) @mdemoret-nv
• Move testing utilities into tests/_utils (#1065) @dagardner-nv
• Update Versions for v23.11.00 (#1059) @mdemoret-nv
• Update GNN stellargraph with DGL (#1032) @tzemicheal
• Implement rest data loader (#986) @yuchenz427
• Adding HTTP sources & sinks (#977) @dagardner-nv

New Contributors

• @pthalasta made their first contribution in #630
• @exactlyallan made their first contribution in #1295
• @pranavm7 made their first contribution in #1357

Full Changelog: v23.07.03...v23.11.00

v23.07.03

🐛 Bug Fixes

• Add pinned libwebp to resolve CVE (#1236) @drobison00
• Add libwebp to meta.yaml for CVE 2307 (#1242) @drobison00
• [BUG] Fix Control Message Utils & SQL Max Connections Exhaust (#1243) @bsuryadevara

v23.07.02

🐛 Bug Fixes

• Move data dir to models (#1099) @dagardner-nv

v23.07.01

Morpheus 23.07.01 (21 Jul 2023)

🐛 Bug Fixes

• Fixing submodules commit ID which got messed up in merge (#1086) @mdemoret-nv