binascii.Error: Incorrect padding

[lobodpav]

When running

kines walk stream-name 000000000001 -t '10m'

I got the following error:

Creating shard iterator with arguments = {'StreamName': 'stream-name', 'ShardId': 'shardId-000000000001', 'ShardIteratorType': 'AT_TIMESTAMP', 'Timestamp': datetime.datetime(2021, 1, 25, 15, 56, 10)}
No records found for this api call 😔
Fetch more records? [Y/n]: y
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/kines/read_util.py", line 85, in get_parsed_data
    return base64.b64decode(kinesis_record["Data"]).decode(UTF_8)
  File "/usr/local/Cellar/[email protected]/3.9.1_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)
binascii.Error: Incorrect padding

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/kines", line 8, in <module>
    sys.exit(kines())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kines/kines_cli.py", line 100, in walk
    read_util.walk(
  File "/usr/local/lib/python3.9/site-packages/kines/read_util.py", line 56, in walk
    parsed_data = get_parsed_data(record)
  File "/usr/local/lib/python3.9/site-packages/kines/read_util.py", line 87, in get_parsed_data
    return kinesis_record["Data"].decode(UTF_8)
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 0-2: invalid continuation byte

[dinsaw]

Can you provide the sample record which you put into the stream?

[lobodpav]

I would love to but it is beyond my control 😞

The reason is that the AWS KPL library we use aggregates multiple events into one and stores this single record in Kinesis. So, I do not know what did the KPL actually store on Kinesis.

However, when KPL stores a single message only, this one was always readable by kines.