[BUG]: vdb_upload example pipeline error on inserting large strings

[dagardner-nv]

Version

24.03

Which installation method(s) does this occur on?

Source

Describe the bug.

Occurs intermittently, presumably based on the content fetched via the RSS feeds.

Minimum reproducible example

python examples/llm/main.py vdb_upload pipeline --stop_after=1024

Relevant log output

Click here to see error details

Unable to insert into collection: VDBUploadExample due to <ParamError: (code=1, message=invalid input, length of string exceeds max length. length: 132886, max length: 65535)>

RPC error: [batch_insert], <ParamError: (code=1, message=invalid input, length of string exceeds max length. length: 132886, max length: 65535)>, <Time:{'RPC start': '2024-04-19 10:35:27.637451', 'RPC error': '2024-04-19 10:35:27.670419'}>

Full env printout

Click here to see environment details

[Paste the results of print_env.sh here, it will be hidden by default]

Other/Misc.

No response

Code of Conduct

• I agree to follow Morpheus' Code of Conduct
• I have searched the open bugs and have found no duplicates for this bug report

[dagardner-nv]

The issue is that Milvus has a max string length of 65535 bytes. Characters such as ñ will consume two bytes.

[dagardner-nv]

This appears to be in part a bug on the milvus side as well. If I create a string containing multi-byte characters that is two characters longer than the max of 65535 chars, I receive this exception reflecting the char-length of 65537:

RPC error: [batch_insert], <ParamError: (code=1, message=invalid input, length of string exceeds max length. length: 65537, max length: 65535)>, <Time:{'RPC start': '2024-04-23 08:46:27.520470', 'RPC error': '2024-04-23 08:46:27.520651'}>

If I then truncate the data and retry, then I receive a new exception this time reflecting the byte length or 196605:

df['content'] = df['content'].str.slice(0, MAX_STRING_LENGTH)

RPC error: [batch_insert], <MilvusException: (code=1100, message=the length (196605) of 0th string exceeds max length (65535): invalid parameter[expected=valid length string][actual=string length exceeds max length])>, <Time:{'RPC start': '2024-04-23 08:48:08.730043', 'RPC error': '2024-04-23 08:48:08.733671'}>