unable to connect to triton server #1823

srisudha-alt · 2024-07-26T16:05:55Z

srisudha-alt
Jul 26, 2024

Hello
I am new to NVIDIA Morpheus platform and trying to bring up the triton server to run anomaly detection.

Triton server is up and running without any issues.

Facing the below exception

I am not able to connect to that when trying to run pipeline via CLI using
Launch Morpheus printing debug messages
morpheus --log_level=DEBUG
# Run a pipeline with 8 threads and a model batch size of 1024 (Must be equal or less than Triton config)
run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=1024
# Specify a NLP pipeline with 256 sequence length (Must match Triton config)
pipeline-fil --columns_file=data/columns_fil.txt
# 1st Stage: Read from file
from-file --filename=examples/data/nvsmi.jsonlines
# 2nd Stage: Deserialize from JSON strings to objects
deserialize
# 3rd Stage: Preprocessing converts the input data into BERT tokens
preprocess
# 4th Stage: Send messages to Triton for inference. Specify the model loaded in Setup
inf-triton --model_name=abp-nvsmi-xgb --server_url=localhost:8000
# 5th Stage: Monitor stage prints throughput information to the console
monitor --description "Inference Rate" --smoothing=0.001 --unit inf
# 6th Stage: Add results from inference to the messages
add-class
# 7th Stage: Convert from objects back into strings. Ignore verbose input data
serialize --include 'mining'
# 8th Stage: Write out the JSON lines to the detections.jsonlines file
to-file --filename=detections.jsonlines --overwrite

Am I missing something here ? Any help would be highly appreciated !!!

I am trying to follow this wiki https://github.com/nv-morpheus/Morpheus/tree/branch-24.10/examples/abp_nvsmi_detection .

Answered by efajardo-nv

Jul 30, 2024

@srisudha-alt You shouldn't have to build the release container since you have already pulled it down. You should only have to fetch the models from Git LFS. So the steps for 24.06 should be:

Set up Triton container

Perform these steps on host:

Install Git LFS
Clone Morpheus, checkout branch-24.06 and fetch models from Git LFS:

git clone -b branch-24.06 https://github.com/nv-morpheus/Morpheus.git
cd Morpheus
git lfs install
./scripts/fetch_data.py fetch all

Start Triton and load nvsmi model.

docker run --rm -ti --gpus=all -p8000:8000 -p8001:8001 -p8002:8002 -v $PWD/models:/models nvcr.io/nvidia/tritonserver:23.06-py3 tritonserver --model-repository=/models/triton-model-repo --exit-on…

View full answer

efajardo-nv · 2024-07-29T14:56:05Z

efajardo-nv
Jul 29, 2024
Collaborator

Hi @srisudha-alt. The command you're running is for the NVSMI example but your model is for the PCAP example.

Also, your error indicates you're trying to connect to tritonserver:8000. After loading the NVSMI model, please ensure that your command is using localhost:8000 as follows:

inf-triton --model_name=abp-nvsmi-xgb --server_url=localhost:8000

Please note that with 24.10 we now provide a Triton container that includes all the models. Until the release, you'll have to build it following these steps:
https://github.com/nv-morpheus/Morpheus/blob/branch-24.10/models/docker/README.md

0 replies

srisudha-alt · 2024-07-30T00:56:24Z

srisudha-alt
Jul 30, 2024
Author

Hi @efajardo-nv ,

Thanks for the reply ! I see below issue.

I see this issue when trying to load model abp-nvsmi-xgb.

I am actually confused with different set of commands for different versions. Can u suggest a specific release of Morpheus to use to run FIL pipeline ? My aim is to have Kafka as data source ( I am just working for a POC).

Can we have multiple release of docker images in machine or does it impact proper working ?

0 replies

efajardo-nv · 2024-07-30T14:28:23Z

efajardo-nv
Jul 30, 2024
Collaborator

@srisudha-alt Appears the models haven't been pulled from Git LFS. It's not clear to me which Morpheus version you're using:

If 24.10, verify that Git LFS is installed and rebuild the Triton container. The build should pull down the models from Git LFS if you don't already have them.
If 24.06, verify that Git LFS is installed and follow these steps to pull the models from Git LFS.

For this example, the only difference between 24.10 and the current stable release (24.06) is the Triton container. The new Triton container in 24.10 is packaged with models and makes it easier by addressing the issues described in #552.

Can we have multiple release of docker images in machine or does it impact proper working ?

Yes, you can have multiple releases of images but make sure to use correct versions of Morpheus/Triton when you run the example.

If you still have issues, could you please provide Morpheus version you're using and all your steps to reproduce?

3 replies

srisudha-alt Jul 30, 2024
Author

I have followed the steps for 24.06 version https://github.com/nv-morpheus/Morpheus/blob/branch-24.06/docs/source/getting_started.md#building-the-morpheus-container. Facing below error. Can I get some help here.

=> [runtime_conda_create 2/2] RUN --mount=type=bind,from=conda_bld_morpheus,source=/opt/conda/conda-bld 249.5s
=> ERROR [build_docs 2/2] RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=lo 359.7s
=> [runtime 1/9] COPY ./conda/environments/*.yaml ./conda/environments/ 0.1s
=> [runtime 2/9] COPY ./docker ./docker 0.1s

[build_docs 2/2] RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked --mount=type=bind,from=conda_bld_morpheus,source=/opt/conda/conda-bld,target=/opt/conda/conda-bld --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked source activate morpheus && CONDA_ALWAYS_YES=true /opt/conda/bin/mamba install -n morpheus -c local -c conda-forge -c huggingface -c rapidsai -c rapidsai-nightly -c nvidia -c nvidia/label/dev -c pytorch -c defaults morpheus && cd . && CMAKE_CONFIGURE_EXTRA_ARGS="-DMORPHEUS_BUILD_DOCS=ON -DMORPHEUS_PYTHON_BUILD_STUBS=OFF -DMORPHEUS_CUDA_ARCHITECTURES=RAPIDS" ./scripts/compile.sh --target morpheus_docs:
253.6 warning libmamba Cache file "/opt/conda/pkgs/cache/497deca9.json" was modified by another program
253.6 warning libmamba Cache file "/opt/conda/pkgs/cache/09cdf8bf.json" was modified by another program
253.6 warning libmamba Cache file "/opt/conda/pkgs/cache/1c9c8fef.json" was modified by another program
262.2 warning libmamba Cache file "/opt/conda/pkgs/cache/f5d7be02.json" was modified by another program
265.4 warning libmamba Cache file "/opt/conda/pkgs/cache/c6f2354e.json" was modified by another program
265.9 warning libmamba Cache file "/opt/conda/pkgs/cache/86b0f08d.json" was modified by another program
270.6 warning libmamba Cache file "/opt/conda/pkgs/cache/7d0a1ac5.json" was modified by another program
298.5 warning libmamba Cache file "/opt/conda/pkgs/cache/73c6abdf.json" was modified by another program
298.6 warning libmamba Cache file "/opt/conda/pkgs/cache/c9ddbd6b.json" was modified by another program
298.7 warning libmamba Cache file "/opt/conda/pkgs/cache/b121c3e7.json" was modified by another program
298.7 warning libmamba Cache file "/opt/conda/pkgs/cache/c3502f35.json" was modified by another program
303.0 warning libmamba Cache file "/opt/conda/pkgs/cache/6a391b98.json" was modified by another program
303.1 warning libmamba Cache file "/opt/conda/pkgs/cache/ee0ed9e9.json" was modified by another program
303.1 warning libmamba Cache file "/opt/conda/pkgs/cache/edb1952f.json" was modified by another program
303.2 warning libmamba Cache file "/opt/conda/pkgs/cache/47929eba.json" was modified by another program
303.3 warning libmamba Cache file "/opt/conda/pkgs/cache/3e39a7aa.json" was modified by another program
351.0 Transaction
351.0
351.0 Prefix: /opt/conda/envs/morpheus
351.0
351.0 Updating specs:
351.0
351.0 - morpheus
351.0 - ca-certificates
351.0 - certifi
351.0 - openssl
351.0
351.0
351.0 Package Version Build Channel Size
351.0 ─────────────────────────────────────────────────────────────────────────────────────────
351.0 Install:
351.0 ─────────────────────────────────────────────────────────────────────────────────────────
351.0
351.0 + morpheus 24.10.00a cuda_12.1_py3.10_ga527ee7c_13 /opt/conda/conda-bld Cached
351.0
351.0 Summary:
351.0
351.0 Install: 1 packages
351.0
351.0 Total download: 0 B
351.0
351.0 ─────────────────────────────────────────────────────────────────────────────────────────
351.0
351.0
358.4
358.4 Looking for: ['morpheus']
358.4
358.4
358.4 Pinned packages:
358.4 - python 3.10.*
358.4
358.4
358.4
358.4 Downloading and Extracting Packages: ...working... done
358.4 Preparing transaction: ...working... done
358.4 Verifying transaction: ...working... done
358.9 Executing transaction: ...working... done
359.5 + set -e
359.5 + BUILD_DIR=build
359.5 + echo 'Runing CMake configure...'
359.5 Runing CMake configure...
359.5 + cmake -S . -B build -GNinja -DCMAKE_MESSAGE_CONTEXT_SHOW=ON -DMORPHEUS_USE_CLANG_TIDY=OFF -DMORPHEUS_PYTHON_INPLACE_BUILD=ON -DMORPHEUS_PYTHON_PERFORM_INSTALL=ON -DMORPHEUS_USE_CCACHE=ON -DMORPHEUS_USE_CONDA=ON -DMORPHEUS_SUPPORT_DOCA=OFF -DCMAKE_AR=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-ar -DCMAKE_CXX_COMPILER_AR=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_C_COMPILER_AR=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-gcc-ar -DCMAKE_RANLIB=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-ranlib -DCMAKE_CXX_COMPILER_RANLIB=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_C_COMPILER_RANLIB=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-gcc-ranlib -DCMAKE_LINKER=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/opt/conda/envs/morpheus/bin/x86_64-conda-linux-gnu-strip -DMORPHEUS_BUILD_DOCS=ON -DMORPHEUS_PYTHON_BUILD_STUBS=OFF -DMORPHEUS_CUDA_ARCHITECTURES=RAPIDS
359.5 CMake Error at CMakeLists.txt:69 (include):
359.5 include could not find requested file:
359.5
359.5 morpheus_utils/load
359.5
359.5
359.5 CMake Error at CMakeLists.txt:71 (morpheus_utils_initialize_package_manager):
359.5 Unknown CMake command "morpheus_utils_initialize_package_manager".
359.5
359.5
359.5 -- Configuring incomplete, errors occurred!

Dockerfile:235

234 |
235 | >>> RUN --mount=type=cache,id=workspace_cache,target=/workspace/.cache,sharing=locked
236 | >>> --mount=type=bind,from=conda_bld_morpheus,source=/opt/conda/conda-bld,target=/opt/conda/conda-bld
237 | >>> --mount=type=cache,id=conda_pkgs,target=/opt/conda/pkgs,sharing=locked
238 | >>> source activate morpheus &&
239 | >>> CONDA_ALWAYS_YES=true /opt/conda/bin/mamba install -n morpheus
240 | >>> -c local
241 | >>> -c conda-forge
242 | >>> -c huggingface
243 | >>> -c rapidsai
244 | >>> -c rapidsai-nightly
245 | >>> -c nvidia
246 | >>> -c nvidia/label/dev
247 | >>> -c pytorch
248 | >>> -c defaults
249 | >>> morpheus &&
250 | >>> # Change to the morpheus directory and build the docs
251 | >>> cd ${MORPHEUS_ROOT_HOST} &&
252 | >>> CMAKE_CONFIGURE_EXTRA_ARGS="-DMORPHEUS_BUILD_DOCS=ON -DMORPHEUS_PYTHON_BUILD_STUBS=OFF -DMORPHEUS_CUDA_ARCHITECTURES=RAPIDS"
253 | >>> ./scripts/compile.sh --target morpheus_docs
254 |

ERROR: failed to solve: process "/bin/bash -c source activate morpheus && CONDA_ALWAYS_YES=true /opt/conda/bin/mamba install -n morpheus -c local -c conda-forge -c huggingface -c rapidsai -c rapidsai-nightly -c nvidia -c nvidia/label/dev -c pytorch -c defaults morpheus && cd ${MORPHEUS_ROOT_HOST} && CMAKE_CONFIGURE_EXTRA_ARGS="-DMORPHEUS_BUILD_DOCS=ON -DMORPHEUS_PYTHON_BUILD_STUBS=OFF -DMORPHEUS_CUDA_ARCHITECTURES=RAPIDS" ./scripts/compile.sh --target morpheus_docs" did not complete successfully: exit code: 1

efajardo-nv Jul 30, 2024
Collaborator

@srisudha-alt You shouldn't have to build the release container since you have already pulled it down. You should only have to fetch the models from Git LFS. So the steps for 24.06 should be:

Set up Triton container

Perform these steps on host:

Install Git LFS
Clone Morpheus, checkout branch-24.06 and fetch models from Git LFS:

git clone -b branch-24.06 https://github.com/nv-morpheus/Morpheus.git
cd Morpheus
git lfs install
./scripts/fetch_data.py fetch all

Start Triton and load nvsmi model.

docker run --rm -ti --gpus=all -p8000:8000 -p8001:8001 -p8002:8002 -v $PWD/models:/models nvcr.io/nvidia/tritonserver:23.06-py3 tritonserver --model-repository=/models/triton-model-repo --exit-on-error=false --model-control-mode=explicit --load-model abp-nvsmi-xgb

Run Morpheus pipeline

From new terminal on host, cd to Morpheus root directory.
Launch Morpheus 24.06 release container:

DOCKER_IMAGE_TAG="24.06-runtime" ./docker/run_container_release.sh

Run pipeline in Morpheus container:

morpheus --log_level=DEBUG \
   `# Run a pipeline with 8 threads and a model batch size of 1024 (Must be equal or less than Triton config)` \
   run --num_threads=8 --pipeline_batch_size=1024 --model_max_batch_size=1024 \
   `# Specify a NLP pipeline with 256 sequence length (Must match Triton config)` \
   pipeline-fil --columns_file=data/columns_fil.txt \
   `# 1st Stage: Read from file` \
   from-file --filename=examples/data/nvsmi.jsonlines \
   `# 2nd Stage: Deserialize from JSON strings to objects` \
   deserialize \
   `# 3rd Stage: Preprocessing converts the input data into BERT tokens` \
   preprocess \
   `# 4th Stage: Send messages to Triton for inference. Specify the model loaded in Setup` \
   inf-triton --model_name=abp-nvsmi-xgb --server_url=localhost:8000 \
   `# 5th Stage: Monitor stage prints throughput information to the console` \
   monitor --description "Inference Rate" --smoothing=0.001 --unit inf \
   `# 6th Stage: Add results from inference to the messages` \
   add-class \
   `# 7th Stage: Convert from objects back into strings. Ignore verbose input data` \
   serialize --include 'mining' \
   `# 8th Stage: Write out the JSON lines to the detections.jsonlines file` \
   to-file --filename=detections.jsonlines --overwrite

Answer selected by srisudha-alt

srisudha-alt Jul 30, 2024
Author

Thank you @efajardo-nv
These series of steps helped me to run the pipeline.

srisudha-alt · 2024-07-30T14:45:52Z

srisudha-alt
Jul 30, 2024
Author

@efajardo-nv I am trying to use 24.03 version with pre-build images. Will that cause an issue ? i tried using pre-build docker containers flow as specified here https://github.com/nv-morpheus/Morpheus/blob/branch-24.03/docs/source/getting_started.md#using-pre-built-docker-containers

I can re-try using https://github.com/nv-morpheus/Morpheus/blob/branch-24.06/docs/source/getting_started.md#building-the-morpheus-container and let u know if i still see the issue !

Thank you for being active in responding.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unable to connect to triton server #1823

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 3 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

unable to connect to triton server #1823

srisudha-alt Jul 26, 2024

Replies: 4 comments · 3 replies

efajardo-nv Jul 29, 2024 Collaborator

srisudha-alt Jul 30, 2024 Author

efajardo-nv Jul 30, 2024 Collaborator

srisudha-alt Jul 30, 2024 Author

Dockerfile:235

efajardo-nv Jul 30, 2024 Collaborator

srisudha-alt Jul 30, 2024 Author

srisudha-alt Jul 30, 2024 Author

srisudha-alt
Jul 26, 2024

Replies: 4 comments 3 replies

efajardo-nv
Jul 29, 2024
Collaborator

srisudha-alt
Jul 30, 2024
Author

efajardo-nv
Jul 30, 2024
Collaborator

srisudha-alt Jul 30, 2024
Author

efajardo-nv Jul 30, 2024
Collaborator

srisudha-alt Jul 30, 2024
Author

srisudha-alt
Jul 30, 2024
Author