Skip to content

Commit

Permalink
feat(aws-lambda-opensearch): created new construct (awslabs#818)
Browse files Browse the repository at this point in the history
* created README

* fixed pattern prop name

* fixed service name in README

* updated prop description

* removed legacy elasticsearch role description

* created new construct

* fixed prop description

* removed kibana text in opensearch construct

* added type safety to opensearch prop
  • Loading branch information
mickychetta committed Oct 11, 2022
1 parent 30a5160 commit f31f59d
Show file tree
Hide file tree
Showing 40 changed files with 7,970 additions and 76 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
|![Java Logo](https://docs.aws.amazon.com/cdk/api/latest/img/java32.png) Java|`software.amazon.awsconstructs.services.lambdaelasticsearchkibana`|

## Overview
This AWS Solutions Construct implements the AWS Lambda function and Amazon Elasticsearch Service with the least privileged permissions.
This AWS Solutions Construct implements an AWS Lambda function and Amazon Elasticsearch Service with the least privileged permissions.

**Some cluster configurations (e.g VPC access) require the existence of the `AWSServiceRoleForAmazonElasticsearchService` Service-Linked Role in your account.**

Expand Down Expand Up @@ -111,7 +111,7 @@ new LambdaToElasticSearchAndKibana(this, "sample",
|esDomainProps?|[`elasticsearch.CfnDomainProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_elasticsearch.CfnDomainProps.html)|Optional user provided props to override the default props for the Elasticsearch Service|
|domainName|`string`|Domain name for the Cognito and the Elasticsearch Service|
|cognitoDomainName?|`string`|Optional Cognito Domain Name, if provided it will be used for Cognito Domain, and domainName will be used for the Elasticsearch Domain|
|createCloudWatchAlarms|`boolean`|Whether to create recommended CloudWatch alarms|
|createCloudWatchAlarms?|`boolean`|Whether to create recommended CloudWatch alarms|
|domainEndpointEnvironmentVariableName?|`string`|Optional Name for the ElasticSearch domain endpoint environment variable set for the Lambda function.|
|existingVpc?|[`ec2.IVpc`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.IVpc.html)|An optional, existing VPC into which this pattern should be deployed. When deployed in a VPC, the Lambda function will use ENIs in the VPC to access network resources. If an existing VPC is provided, the `deployVpc` property cannot be `true`. This uses `ec2.IVpc` to allow clients to supply VPCs that exist outside the stack using the [`ec2.Vpc.fromLookup()`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.Vpc.html#static-fromwbrlookupscope-id-options) method.|
|vpcProps?|[`ec2.VpcProps`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.VpcProps.html)|Optional user provided properties to override the default properties for the new VPC. `enableDnsHostnames`, `enableDnsSupport`, `natGateways` and `subnetConfiguration` are set by the pattern, so any values for those properties supplied here will be overrriden. If `deployVpc` is not `true` then this property will be ignored.|
Expand All @@ -121,13 +121,13 @@ new LambdaToElasticSearchAndKibana(this, "sample",

| **Name** | **Type** | **Description** |
|:-------------|:----------------|-----------------|
|lambdaFunction|[`lambda.Function`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html)|Returns an instance of lambda.Function created by the construct|
|userPool|[`cognito.UserPool`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.UserPool.html)|Returns an instance of cognito.UserPool created by the construct|
|userPoolClient|[`cognito.UserPoolClient`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.UserPoolClient.html)|Returns an instance of cognito.UserPoolClient created by the construct|
|identityPool|[`cognito.CfnIdentityPool`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.CfnIdentityPool.html)|Returns an instance of cognito.CfnIdentityPool created by the construct|
|elasticsearchDomain|[`elasticsearch.CfnDomain`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_elasticsearch.CfnDomain.html)|Returns an instance of elasticsearch.CfnDomain created by the construct|
|elasticsearchDomain|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_iam.Role.html)|Returns an instance of iam.Role created by the construct for elasticsearch.CfnDomain|
|cloudwatchAlarms?|[`cloudwatch.Alarm[]`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Alarm.html)|Returns a list of cloudwatch.Alarm created by the construct|
|lambdaFunction|[`lambda.Function`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_lambda.Function.html)|Returns an instance of `lambda.Function` created by the construct|
|userPool|[`cognito.UserPool`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.UserPool.html)|Returns an instance of `cognito.UserPool` created by the construct|
|userPoolClient|[`cognito.UserPoolClient`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.UserPoolClient.html)|Returns an instance of `cognito.UserPoolClient` created by the construct|
|identityPool|[`cognito.CfnIdentityPool`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cognito.CfnIdentityPool.html)|Returns an instance of `cognito.CfnIdentityPool` created by the construct|
|elasticsearchDomain|[`elasticsearch.CfnDomain`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_elasticsearch.CfnDomain.html)|Returns an instance of `elasticsearch.CfnDomain` created by the construct|
|elasticsearchRole|[`iam.Role`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_iam.Role.html)|Returns an instance of `iam.Role` created by the construct for `elasticsearch.CfnDomain`|
|cloudwatchAlarms?|[`cloudwatch.Alarm[]`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Alarm.html)|Returns a list of `cloudwatch.Alarm` created by the construct|
|vpc?|[`ec2.IVpc`](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2.IVpc.html)|Returns an interface on the VPC used by the pattern (if any). This may be a VPC created by the pattern or the VPC supplied to the pattern constructor.|

## Lambda Function
Expand All @@ -136,26 +136,26 @@ This pattern requires a lambda function that can post data into the Elasticsearc

## Default settings

Out of the box implementation of the Construct without any override will set the following defaults:
Out of the box implementation of the Construct without any overrides will set the following defaults:

### AWS Lambda Function
* Configure limited privilege access IAM role for Lambda function
* Enable reusing connections with Keep-Alive for NodeJs Lambda function
* Enable reusing connections with Keep-Alive for Node.js Lambda function
* Enable X-Ray Tracing
* Set Environment Variables
* (default) DOMAIN_ENDPOINT
* AWS_NODEJS_CONNECTION_REUSE_ENABLED (for Node 10.x and higher functions)
* AWS_NODEJS_CONNECTION_REUSE_ENABLED

### Amazon Cognito
* Set password policy for User Pools
* Enforce the advanced security mode for User Pools

### Amazon Elasticsearch Service
* Deploy best practices CloudWatch Alarms for the Elasticsearch Domain
* Deploy best practices CloudWatch Alarms for the Elasticsearch Service domain
* Secure the Kibana dashboard access with Cognito User Pools
* Enable server-side encryption for Elasticsearch Domain using AWS managed KMS Key
* Enable node-to-node encryption for Elasticsearch Domain
* Configure the cluster for the Amazon ES domain
* Enable server-side encryption for the Elasticsearch Service domain using AWS managed KMS Key
* Enable node-to-node encryption for the Elasticsearch Service domain
* Configure the cluster for the Elasticsearch Service domain

## Architecture
![Architecture Diagram](architecture.png)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,11 @@ import * as cognito from 'aws-cdk-lib/aws-cognito';
import * as defaults from '@aws-solutions-constructs/core';
// Note: To ensure CDKv2 compatibility, keep the import statement for Construct separate
import { Construct } from 'constructs';
import { Role } from 'aws-cdk-lib/aws-iam';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as ec2 from 'aws-cdk-lib/aws-ec2';

/**
* @summary The properties for the CognitoToApiGatewayToLambda Construct
* @summary The properties for the LambdaToElasticSearchAndKibana Construct
*/
export interface LambdaToElasticSearchAndKibanaProps {
/**
Expand Down Expand Up @@ -99,10 +98,10 @@ export class LambdaToElasticSearchAndKibana extends Construct {
public readonly vpc?: ec2.IVpc;

/**
* @summary Constructs a new instance of the CognitoToApiGatewayToLambda class.
* @param {cdk.App} scope - represents the scope for all the resources.
* @summary Constructs a new instance of the LambdaToElasticSearchAndKibana class.
* @param {Constructs} scope - represents the scope for all the resources.
* @param {string} id - this is a a scope-unique id.
* @param {CognitoToApiGatewayToLambdaProps} props - user provided props for the construct
* @param {LambdaToElasticSearchAndKibanaProps} props - user provided props for the construct
* @since 0.8.0
* @access public
*/
Expand Down Expand Up @@ -146,7 +145,7 @@ export class LambdaToElasticSearchAndKibana extends Construct {
let cognitoAuthorizedRole: iam.Role;

[this.userPool, this.userPoolClient, this.identityPool, cognitoAuthorizedRole] =
this.setupCognito(this, props.cognitoDomainName ?? props.domainName);
defaults.buildCognitoForSearchService(this, props.cognitoDomainName ?? props.domainName);

const buildElasticSearchProps: any = {
userpool: this.userPool,
Expand Down Expand Up @@ -174,18 +173,4 @@ export class LambdaToElasticSearchAndKibana extends Construct {
this.cloudwatchAlarms = defaults.buildElasticSearchCWAlarms(this);
}
}

setupCognito(scope: Construct, domainName: string): [cognito.UserPool, cognito.UserPoolClient, cognito.CfnIdentityPool, iam.Role] {
const userPool = defaults.buildUserPool(scope);
const userPoolClient = defaults.buildUserPoolClient(scope, userPool);
const identityPool = defaults.buildIdentityPool(scope, userPool, userPoolClient);

const cognitoAuthorizedRole: Role = defaults.setupCognitoForElasticSearch(scope, domainName, {
userpool: userPool,
identitypool: identityPool,
userpoolclient: userPoolClient
});

return [userPool, userPoolClient, identityPool, cognitoAuthorizedRole];
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -804,7 +804,7 @@
"Properties": {
"Code": {
"S3Bucket": "cdk-hnb659fds-assets-12345678-test-region",
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@
"Properties": {
"Code": {
"S3Bucket": "cdk-hnb659fds-assets-12345678-test-region",
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@
"S3Bucket": {
"Fn::Sub": "cdk-hnb659fds-assets-${AWS::AccountId}-${AWS::Region}"
},
"S3Key": "67a9971e29baab2bde3043bb70ce5b53318b95429a1ce9b189cf65223e8682db.zip"
"S3Key": "35bbbc7b04b21f225891f139adf234188f348ebad5f4bbc2c06edf8aa3784c97.zip"
},
"Role": {
"Fn::GetAtt": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function postDocumentToES(doc, context) {
var body = '';
httpResp.on('data', function (chunk) {
body += chunk;
});
});
httpResp.on('end', function (chunk) {
console.log('All movie records added to ES.');
context.succeed();
Expand All @@ -47,10 +47,13 @@ function postDocumentToES(doc, context) {

exports.handler = (event, context) => {
console.log('Received event:', JSON.stringify(event, null, 2));
postDocumentToES("{ \"title\": \"Spirited Away\" }", context);
return {
      statusCode200,
      headers{ 'Content-Type''text/plain' },
      body`Hello from Project Vesper! You've hit ${event.path}\n`
    };
postDocumentToOpenSearch("{ \"title\": \"Moby Dick\" }", context);
postDocumentToOpenSearch("{ \"title\": \"A Tale of Two Cities\" }", context);
postDocumentToOpenSearch("{ \"title\": \"The Phantom of the Opera\" }", context);

return {
statusCode: 200,
headers: { 'Content-Type': 'text/plain' },
body: `Hello from AWS Solutions Constructs!\n`
};
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
lib/*.js
test/*.js
*.d.ts
coverage
test/lambda/index.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
lib/*.js
test/*.js
!test/lambda/*
*.js.map
*.d.ts
node_modules
*.generated.ts
dist
.jsii

.LAST_BUILD
.nyc_output
coverage
.nycrc
.LAST_PACKAGE
*.snk
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Exclude typescript source and config
*.ts
tsconfig.json
coverage
.nyc_output
*.tgz
*.snk
*.tsbuildinfo

# Include javascript files and typescript declarations
!*.js
!*.d.ts

# Exclude jsii outdir
dist

# Include .jsii
!.jsii

# Include .jsii
!.jsii
Loading

0 comments on commit f31f59d

Please sign in to comment.