Pipeline creation fails in DataProcessing due to timed out accessing the glue endpoint #94
-
SummaryPipeline creation fails in DataProcessing stack creation Steps to reproduceFollowed exact same steps as per workshop, created project from webconsole, used kinesis as sink, after putting all config and on creation of pilpeine, data processing stack fails. What is the current bug behavior?Pipeline creation fails What is the expected correct behavior?Pipeline should get created Relevant logs and/or screenshotsReceived response status [FAILED] from custom resource. Message returned: Socket timed out without establishing a connection within 5000 ms Logs: /aws/lambda/Clickstream-DataProcessin-GlueTablePartitionSyncer-LHI331oLoIf1 at Timeout._onTimeout (/var/task/index.js:13076:30) at listOnTimeout (node:internal/timers:559:17) at processTimers (node:internal/timers:502:7) (RequestId: bd2a66f8-2797-41d4-8323-fb5c0c0b0ea6) Possible fixesNot Sure This is 🐛 Bug Report |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments
-
|
Hi @snjkumar23 , the data processing stack will provisioning a glue data catalog for putting the processed clickstream data. It requires accessing the endpoint of Glue. And the job will be periodically executed for creating the partitions of glue tables. So this function is placed on the private subnets when you configured the ingestion module. And the web console does some network checking for NAT gateway if you specify the private subnets with NAT gateway or the isolated subnets with the required VPC endpoints used by the solution. But, we know an edge case might cause the error like you met,
You can check your VPC endpoint for Glue, updating the inboud rule of security group to allow the reques from the Lamba function. Then clicking the Retry action in the pipeline detail page to resume the provision of the pipeline. |
Beta Was this translation helpful? Give feedback.
-
|
Let's know if it helps. Thanks. |
Beta Was this translation helpful? Give feedback.
-
|
Issues resolved, VPC endpoint security group had to allow lamda to access it. |
Beta Was this translation helpful? Give feedback.
Hi @snjkumar23 , the data processing stack will provisioning a glue data catalog for putting the processed clickstream data.
It requires accessing the endpoint of Glue. And the job will be periodically executed for creating the partitions of glue tables. So this function is placed on the private subnets when you configured the ingestion module.
And the web console does some network checking for NAT gateway if you specify the private subnets with NAT gateway or the isolated subnets with the required VPC endpoints used by the solution. But, we know an edge case might cause the error like you met,