Connecting to a relational database from a Lambda function is challenging to get right. Your database can easily get overloaded by multiple Lambda invocations if the connection is not handled correctly and if the number of maximum concurrent connections gets exceeded. We use AWS' RDS Proxy service to make sure our Lambdas don’t run into these issues.

In the modern serverless world, it’s common to see infrastructures built as code (IaC). Multiple frameworks help us do this. Some of which are:

• Serverless Framework

• AWS Serverless Application Model (SAM)

• Terraform

Creating an RDS Proxy will be demonstrated using the Serverless Framework.

This tutorial assumes that you have basic knowledge of how Serverless Framework and CloudFormation work. Ofcourse there is a repo containing all the code for this example app.

RDS Proxy Setup

We’ll start from the serverless.yml file.

1service: serverless-framework-rds-proxy-example
2
3provider:
4  name: aws
5  region: us-east-1
6  runtime: nodejs12.x
7  memorySize: 256
8  vpc:
9    securityGroupIds:
10      - !Ref LambdaSecurityGroup
11    subnetIds:
12      - !Ref SubnetA
13      - !Ref SubnetB
14  environment:
15    NODE_ENV: ${opt:stage, 'dev'}
16    DB_NAME: ${self:custom.DB_NAME}
17    DB_USER: ${self:custom.DB_USERNAME}
18    DB_PASS: ${self:custom.DB_PASSWORD}
19    DB_PORT: ${self:custom.DB_PORT}
20    DB_HOST: ${self:custom.PROXY_ENDPOINT}
21
22custom:
23  DB_NAME: dbname
24  DB_USERNAME: dbusername
25  DB_PASSWORD: dbpassword
26  DB_PORT: !GetAtt RDSInstance.Endpoint.Port
27  PROXY_ENDPOINT: !GetAtt RDSProxy.Endpoint
28  PROXY_NAME: example-proxy-name-${opt:stage, 'dev'}
29  VPC_CIDR: 10
30
31functions:
32  proxyHealthCheck:
33    handler: lib/handlers/proxyHealthCheck.handler
34    events:
35      - http:
36          path: proxy-healthcheck
37          method: get
38
39plugins:
40  - serverless-pseudo-parameters
41
42resources:
43  - ${file(resources/VpcResources.yml)}
44  # - ${file(resources/RoutingResources.yml)}
45  - ${file(resources/RdsResources.yml)}
46  - ${file(resources/RdsProxyResources.yml)}

We put the Lambda in the same VPC as the RDS is in (line 8) to connect to it. Keep in mind that when a Lambda is inside of a VPC, it losses internet access. In the following article, you can see how to give internet access to your Lambdas when they’re in a VPC.

Note that we are using the Proxys endpoint instead of the RDS' endpoint (line 27). We later use the values from the custom section as environment variables in our proxyHealthCheck function to connect to the RDS through the Proxy.

Now we’ll take a look at the VpcResources.yml file:

1Resources:
2  VPC:
3    Type: AWS::EC2::VPC
4    Properties:
5      CidrBlock: ${self:custom.VPC_CIDR}.0.0.0/16
6      Tags:
7        - Key: "Name"
8          Value: "VPC"
9          
10  SubnetA:
11    Type: AWS::EC2::Subnet
12    Properties:
13      VpcId: !Ref VPC
14      AvailabilityZone: ${self:provider.region}a
15      CidrBlock: ${self:custom.VPC_CIDR}.0.0.0/24
16      Tags:
17        - Key: "Name"
18          Value: "SubnetA"
19
20  SubnetB:
21    Type: AWS::EC2::Subnet
22    Properties:
23      VpcId: !Ref VPC
24      AvailabilityZone: ${self:provider.region}b
25      CidrBlock: ${self:custom.VPC_CIDR}.0.1.0/24
26      Tags:
27        - Key: "Name"
28          Value: "SubnetB"
29
30  LambdaSecurityGroup:
31    Type: AWS::EC2::SecurityGroup
32    Properties:
33      VpcId: !Ref VPC
34      GroupDescription: "Security group for Lambdas"
35      Tags:
36        - Key: "Name"
37          Value: "LambdaSecurityGroup"
38
39  RDSSecurityGroup:
40    Type: AWS::EC2::SecurityGroup
41    Properties:
42      GroupDescription: "Security group for RDS"
43      VpcId: !Ref VPC
44      SecurityGroupIngress:
45        - IpProtocol: tcp
46          FromPort: 0
47          ToPort: 65535
48          CidrIp: "0.0.0.0/0"
49      Tags:
50        - Key: "Name"
51          Value: "RDSSecurityGroup"

This creates several things:

• 1 VPC

• 2 subnets which are created within the VPC

• 1 security group for our Lambdas which is also created within the VPC

• 1 security group for the RDS, which is also created within the VPC

Next, we’ll see how to create the RDS instance in the RdsResources.yml file:

1Resources:
2  RDSInstance:
3    Type: AWS::RDS::DBInstance
4    Properties:
5      DBName: ${self:custom.DB_NAME}
6      MasterUsername: ${self:custom.DB_USERNAME}
7      MasterUserPassword: ${self:custom.DB_PASSWORD}
8      Engine: postgres
9      EngineVersion: "11.9"
10      DBInstanceClass: db.t2.micro
11      DBSubnetGroupName: !Ref RDSSubnetGroup
12      VPCSecurityGroups:
13        - !GetAtt RDSSecurityGroup.GroupId
14
15  RDSSubnetGroup:
16    Type: AWS::RDS::DBSubnetGroup
17    Properties:
18      DBSubnetGroupDescription: "RDS subnet group"
19      SubnetIds:
20        - !Ref SubnetA
21        - !Ref SubnetB
22      Tags:
23        - Key: "Name"
24          Value: "RDSSubnetGroup"

As you can see, we’re creating a PostgreSQL database with the version number set to 11.9. This is the maximum PostgreSQL engine version that RDS Proxy currently supports. [1] This might change in the future, so I recommend looking it up.

This kind of setup makes our RDS instance only accessible to some other cases within the VPC. We’re going to look at how we can make our RDS instance publicly accessible later on in the article, as it requires a little tweaking.

Now all that’s left is to set up an RDS Proxy. We’ll see how to do that in the RdsProxyResources.yml file:

1Resources:
2  RDSProxy:
3    Type: AWS::RDS::DBProxy
4    Properties:
5      DBProxyName: ${self:custom.PROXY_NAME}
6      EngineFamily: POSTGRESQL
7      RoleArn: !GetAtt RDSProxyRole.Arn
8      Auth:
9        - AuthScheme: SECRETS
10          IAMAuth: DISABLED
11          SecretArn: !Ref RDSSecret
12      VpcSecurityGroupIds:
13        - !Ref RDSSecurityGroup
14      VpcSubnetIds:
15        - !Ref SubnetA
16        - !Ref SubnetB
17
18  RDSProxyTargetGroup:
19    Type: AWS::RDS::DBProxyTargetGroup
20    Properties:
21      TargetGroupName: default
22      DBProxyName: !Ref RDSProxy
23      DBInstanceIdentifiers:
24        - !Ref RDSInstance
25
26  RDSSecret:
27    Type: AWS::SecretsManager::Secret
28    Properties:
29      SecretString: '{"username":"${self:custom.DB_USERNAME}", "password":"${self:custom.DB_PASSWORD}"}'
30
31  RDSProxyRole:
32    Type: AWS::IAM::Role
33    Properties:
34      AssumeRolePolicyDocument:
35        Version: "2012-10-17"
36        Statement:
37          - Effect: Allow
38            Action: "sts:AssumeRole"
39            Principal:
40              Service: "rds.amazonaws.com"
41      Policies:
42        - PolicyName: RDSProxyPolicy
43          PolicyDocument:
44            Version: "2012-10-17"
45            Statement:
46              - Effect: Allow
47                Action: "secretsmanager:GetSecretValue"
48                Resource: !Ref RDSSecret
49              - Effect: Allow
50                Action: "kms:Decrypt"
51                Resource: "arn:aws:kms:${self:provider.region}:#{AWS::AccountId}:key/*"
52                Condition:
53                  StringEquals:
54                    kms:ViaService: "secretsmanager.${self:provider.region}.amazonaws.com"

The first thing to notice is that we use VpcSecurityGroupIds and VpcSubnetIds properties to put the Proxy in the same VPC as the RDS. It is mandatory if we want our Proxy to be able to connect to the RDS.

Several things are happening in the Auth property:

• AuthScheme property specifies the authentication type which the Proxy uses to connect to the RDS

• In SecretArn we specify the secret which we want the Proxy to use to connect to the RDS. We store this secret in Amazon Secrets Manager (line 26)

• IAMAuth property allows us to choose if we want to require or disallow IAM authentication for connections to the proxy

For the Proxy to be able to read secrets from the Amazon Secrets Manager, we need to create and specify a role in the RoleArn property that allows it to do that. The RDSProxyRole (line 31) allows the Proxy to read and decrypt values store in the secrets manager.

We use the RDSProxyTargetGroup resource to connect the RDS Proxy to the RDS. We can use the same Proxy instance for connecting to multiple RDS instances. That is why the DBInstanceIdentifiers property takes a list of arguments.

All that’s left is to add the code for the proxyHealthCheck Lambda function. We use this Lambda to test if our Proxy - RDS connection works properly. Here’s how the proxyHealthCheck.js file looks like:

1const { Client } = require('pg')
2
3const proxyHealthCheck = async (event, context) => {
4  console.log(JSON.stringify({ event, context }))
5
6  console.log('Creating database client')
7  const client = new Client({
8    database: process.env.DB_NAME,
9    user: process.env.DB_USER,
10    password: process.env.DB_PASS,
11    host: process.env.DB_HOST,
12    port: process.env.DB_PORT,
13  })
14
15  let response
16  try {
17    console.log('Connecting to database')
18    await client.connect()
19
20    console.log('Quering the database')
21    const { rowCount } = await client.query('SELECT $1::text AS message', ['Hello world!'])
22    
23    response = {
24      statusCode: 200,
25      body: JSON.stringify({
26        serverTimestamp: new Date().toISOString(),
27        db: rowCount === 1 ? 'Ok' : 'Fail'
28      })
29    }
30  } catch (error) {
31    console.error(error)
32    response = {
33      statusCode: 500,
34      body: error.message
35    }
36  } finally {
37    console.log('Closing database connection')
38    await client.end()
39  }
40
41  console.log(response)
42  return response
43}
44
45module.exports.handler = proxyHealthCheck

Now that everything is setup, we should test it. Run:

1$ serverless deploy --stage dev

and you’ll get an endpoint in your terminal, which you can use to test the Proxy by sending a GET request to it.

RDS Public Accessibility

Making your RDS publicly accessible allows you to connect to it from your local environment like you would to a standard database. This makes it easy for you to run things like migration scripts or seed scripts on your RDS. If you try to do this with the current setup, your function will throw a timeout error. We need to make a few adjustments to be able to do this.

First, we need to create an Internet Gateway and attach it to our VPC. To do this, add the following resources to the VpcResources.yml file:

1InternetGateway:
2  Type: AWS::EC2::InternetGateway
3  Properties:
4    Tags:
5      - Key: "Name"
6        Value: "InternetGateway"
7
8VPCGA:
9  Type: AWS::EC2::VPCGatewayAttachment
10  Properties:
11    VpcId: !Ref VPC
12    InternetGatewayId: !Ref InternetGateway

We also need to modify the VPC resource to look like this:

1VPC:
2  Type: AWS::EC2::VPC
3  Properties:
4    CidrBlock: ${self:custom.VPC_CIDR}.0.0.0/16
5    EnableDnsSupport: true
6    EnableDnsHostnames: true
7    Tags:
8      - Key: "Name"
9        Value: "VPC"

Details about what EnableDnsSupport and EnableDnsHostnames do. The important thing to know is that they are required for what we’re trying to do.

Next, we need to modify the RDS instance in the RdsResources.yml file to look like this:

1RDSInstance:
2  DependsOn: VPCGA
3  Type: AWS::RDS::DBInstance
4  Properties:
5    DBName: ${self:custom.DB_NAME}
6    MasterUsername: ${self:custom.DB_USERNAME}
7    MasterUserPassword: ${self:custom.DB_PASSWORD}
8    Engine: postgres
9    EngineVersion: "11.9"
10    DBInstanceClass: db.t2.micro
11    AllocatedStorage: "20"
12    PubliclyAccessible: true
13    DBSubnetGroupName: !Ref RDSSubnetGroup
14    VPCSecurityGroups:
15      - !GetAtt RDSSecurityGroup.GroupId

We added the following properties:

• DependsOn: VPCGA to make sure that the VPC Internet Gateway attachment gets created before the RDS instance

• PubliclyAccessible: true to make the RDS instance publicly accessible

Everything else in the RdsResources.yml file stays the same.

Next, we need to create a Route Table within our VPC and make a Route to the Internet Gateway. We also need to associate our Subnets to that Route Table. We do this in the RoutingResources.yml file:

1Resources:
2  RouteTablePublic:
3    DependsOn: VPCGA
4    Type: AWS::EC2::RouteTable
5    Properties:
6      VpcId: !Ref VPC
7      Tags:
8        - Key: "Name"
9          Value: "RouteTablePublic"
10
11  RoutePublic:
12    Type: AWS::EC2::Route
13    Properties:
14      DestinationCidrBlock: 0.0.0.0/0
15      GatewayId: !Ref InternetGateway
16      RouteTableId: !Ref RouteTablePublic
17
18  RouteTableAssociationSubnetA:
19    Type: AWS::EC2::SubnetRouteTableAssociation
20    Properties:
21      RouteTableId: !Ref RouteTablePublic
22      SubnetId: !Ref SubnetA
23
24  RouteTableAssociationSubnetB:
25    Type: AWS::EC2::SubnetRouteTableAssociation
26    Properties:
27      RouteTableId: !Ref RouteTablePublic
28      SubnetId: !Ref SubnetB

Make sure to uncomment the line that loads this resource in the serverless.yml file (line 44).

You can now deploy your stack again by running:

1$ serverless deploy --stage dev

Now your RDS is publicly accessible. You can see how this is tested in the example app for this tutorial here. If you want to clone the example app repo, don’t forget to change the DB_HOST variable in the .env.sample file to the endpoint of your RDS instance. You can find that your AWS RDS console.

Conclusion

This pretty much covers it. If you get errors during your deployments, try finding the mistake in the AWS CloudFormation console. Find your stack and go to Events tab. There you can see which resources failed, why, and when.

The following article will show you how to give your Lambdas internet access by making a VPC public.

Here’s a link to the example app for this tutorial containing all the used code:

https://github.com/montecha/examples/tree/main/serverless-framework-rds-proxy - Connect to preview