Not recovering from broken pipe

[superosku]

It has started happening to us recently that rin fails to connect to redshift with a broken pipe error. This happens every few weeks and started happening after we started scaling the redshift cluster up and down daily.

It never recovers from this and just continuously prints the whole queue content into logs. Restarting RIN temporarily resolves the issue, but it is back in a week or two.

2022/04/11 06:19:38 [info] [XX] Starting process message.
2022/04/11 06:19:38 [info] [XX] Importing event: ObjectCreated:Put s3://xx.json
2022/04/11 06:19:38 [info] Import to target s3://xx => redshift://xx:****@xx from record ObjectCreated:Put s3://xx.json
2022/04/11 06:19:38 [error] [XX] Import failed. write tcp 10.167.x.x:x->34.247.x.x:x: write: broken pipe
2022/04/11 06:19:38 [info] [XX] Aborted message. ReceiptHandle: XX==

After quick investigation it seems that it is failing to broken pipe on this line.
https://github.com/fujiwara/Rin/blob/master/redshift.go#L110

That should not happen if this ping check is working as intended. Maybe it is not?
https://github.com/fujiwara/Rin/blob/master/redshift.go#L70

Thanks!

[fujiwara]

@superosku
Thank you for the reporting!
It seems to be the same as #16

Could you try redshift.reconnect_on_error = true ? https://github.com/fujiwara/Rin#configyaml

I am considering setting reconnect_on_error default true in the next version.