This article discusses how to use Terraform to integrate the Amazon API gateway with the Cequence Unified API Protection (UAP) platform. You can also integrate using CloudFormation.

Further details on CloudWatch are available in Amazon documentation.

Integration architecture

The Amazon API gateway integration for the Cequence UAP platform is made of the following components.

Amazon API Gateway

Receives incoming requests from clients and routes the requests to the appropriate API backend services or applications.

Generates detailed log entries for each request and response, including headers, body, query parameters, and other metadata relevant to Cequence UAP platform analysis.

Amazon CloudWatch

Receives log events from API Gateway and stores them in log groups.

Note: Amazon CloudWatch limitations limit batch sizes to 1MB and 5000 transactions per second, per region. Use the Service Quotas service to change the transaction-per-second limit. The batch size limit cannot be increased.

Amazon CloudWatch Events

A serverless event management service that enables the triggering of an AWS service at a scheduled interval. Amazon CloudWatch Events triggers an AWS Lambda function that captures API Gateway log events every minute.

AWS Lambda function

Triggered every minute by AWS CloudWatch Events. Pulls the API Gateway log entries from AWS CloudWatch Events, transforms the entries into the payload format used by the Cequence UAP platform, aggregates the entries over the last minute of activity, and posts the batch to the Cequence UAP platform for analysis.

The procedures in this article automate the Lambda function configuration by discovering all API gateways and their deployed stages, then configuring the dependencies above. A Terraform script that provides this automation can be applied to an individual AWS account.

What you'll need

• Cequence Bridge release 5.6.2 or later.
• Terraform release v1.9.5 or later.
• AWS CLI release 2.14.0 or later
• Current installations of node.js and npm.
• Access to the jq, ed, and zip UNIX tools.
• A client ID and client secret generated from the Cequence UAP platform.

Generating a client ID and client Secret

Several Cequence components must authenticate to the Cequence UAP platform in order to transmit and receive data. Create authentication credentials in the Cequence UAP platform to enable this authentication.

1. Log in to the UAP management portal UI.
   The URL for the management portal is typically of the form https://ui.<your-tenant-name>.<domain>. Replace <your-tenant-name> with the name of your Cequence tenant organization. Replace <domain> with your domain name.
2. Select General Settings > User Management.
   The User Management pane appears.
3. Click the Clients tab.
4. Click Add New Client.
   The new client dialog box appears.
5. Type the client name in the Client Name field.
   This name is the client ID. Note the client ID for later use.
6. Enable the Traffic Management toggle.
7. (Optional) To change the token lifespan from the default of 1800 seconds, type a whole number of seconds in Token Lifespan.
8. Click Save.
   A dialog box with the client secret appears.
9. Click the blue Copy icon to copy the secret to the clipboard, then click Close.
   The client is now set up. Note the client name for future use.
   The client list appears.
10. Note the value of the client secret for later use. This value will not be shown again later on the UI for security reasons.

Prerequisites

• An AWS account with sufficient permissions to manage API Gateway, Lambda, IAM, CloudWatch, EventBridge Scheduler, and EC2.
• Working REST and HTTP APIs deployed to various stages in API Gateway in an AWS account.
• Access to an AWS CLI user account with specific privileges and roles, which are described in a following section.

• An AWS policy named according to the following format.

  iam_apigw_lambda_cloudwatch_eventbridge_cq_integration

• The AWS CLI user account must have the privileges described in the AWS CLI user privileges section of this article.

AWS CLI user privileges

Setting up the Amazon API Gateway integration requires the use of an AWS CLI user account with an access key, secret, and the privileges listed in the following JSON code. These restricted permissions apply to resources that are created by Terraform.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:GetRolePolicy",
                "iam:PassRole",
                "iam:AttachRolePolicy",
                "iam:DetachRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::*:role/cequence_*",
                "arn:aws:iam::*:role/AmazonAPIGatewayPushToCloudWatchLogs2-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:GetFunction",
                "lambda:UpdateFunctionConfiguration",
                "lambda:UpdateFunctionCode",
                "lambda:ListVersionsByFunction",
                "lambda:GetFunctionCodeSigningConfig"
            ],
            "Resource": "arn:aws:lambda:*:*:function:cequence-api-gateway-cloudwatch-lambda1-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "scheduler:CreateSchedule",
                "scheduler:DeleteSchedule",
                "scheduler:GetSchedule"
            ],
            "Resource": "arn:aws:scheduler:*:*:schedule/default/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:DeleteLogGroup",
                "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/lambda/cequence-api-gateway-cloudwatch-lambda1-*",
                "arn:aws:logs:*:*:log-group:/aws/apigateway/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeRegions",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "apigateway:GET",
                "apigateway:POST",
                "apigateway:PUT",
                "apigateway:PATCH",
                "apigateway:DELETE"
            ],
            "Resource": [
                "arn:aws:apigateway:*::/restapis/*/stages/*",
                "arn:aws:apigateway:*::/restapis/*/stages",
                "arn:aws:apigateway:*::/restapis/*/deployments",
                "arn:aws:apigateway:*::/account"
            ]
        }
    ]
}

The Terraform script requires the following roles and permissions.

API Gateway Role (aws_iam_role.api_gateway_cloudwatch)

• Purpose: Allows API Gateway to push logs to CloudWatch
• Trust Policy: Allows only apigateway.amazonaws.com to assume this role
• Permissions: Uses AWS managed policy AmazonAPIGatewayPushToCloudWatchLogs
• Used by: API Gateway account settings for CloudWatch logging

Lambda Role (aws_iam_role.lambda_role)

• Purpose: Enables Lambda function to interact with CloudWatch Logs
• Trust Policy: Allows only lambda.amazonaws.com to assume this role
• Custom Policy Permissions:

• logs:CreateLogGroup
  logs:CreateLogStream
  logs:PutLogEvents
  logs:FilterLogEvents
  logs:Unmask

• Scope: Full access to CloudWatch logs resources (arn:aws:logs:*:*:*)

EventBridge Role (aws_iam_role.eventbridge_role)

• Purpose: Allows EventBridge to invoke the Lambda function
• Trust Policy: Allows only scheduler.amazonaws.com to assume this role

• Custom Policy Permissions:

  lambda:InvokeFunction

• Scope: Limited to only the specific Lambda function ARN

Installing the integration with Terraform

The compressed archive file that contains the integration bundle contains a file named main.tf. Terraform uses this file during installation.

The main.tf configuration file is structured into two sections, a section for global AWS resources and a section for regional resources. This approach provides comprehensive API scanning and monitoring across regions, maintaining resource isolation and regional compliance.

• The Global AWS resources section creates account-wide resources that are not region-specific. These resources are shared across all regions in the AWS account.
• The Regional Resources section creates and deploys region-specific components. These resources enable the monitoring of APIs deployed across multiple AWS regions within the same account. Each region has an independent set of monitoring resources. The region specific components include:
  • Cequence Lambda function
  • EventBridge scheduler
  • CloudWatch log groups
  • API Gateway configurations

1. Download the compressed archive file that contains the plugin bundle.

2. Extract the archive file. The extraction creates the cequence directory structure.

   tar -xvf cequence-aws-api-gateway-x.x.x.b7xxxxf34.tar.gz

3. Navigate to the scripts directory under cequence and grant execute permission to the scripts by running the following command.

   cd cequence/scripts
   chmod +x *.sh

4. Create a copy of the example environment properties file by running the following command.

   cp .env.example .env

   Note:- The .env file in the bundle contains variables to cater for both Terraform and CloudFormation. The following table defines the properties of the environment variables for Terraform only, leaving other variables with their default values.

   Variable	Description
   cequence_client_id	The client ID generated earlier in this article
   cequence_client_secret	The client secret generated earlier in this article
   cequence_auth_endpoint	The endpoint for Cequence Edge or  Bridge authentication.
   cequence_edge_endpoint	The endpoint for the Cequence Edge or  Bridge.
   cequence_aws_regions	all or a single region or list of comma separated regions
   cequence_is_aws_organization_deployment	false for Terraform Deployment Currently only used for CloudFormation to create the bucket policy to allow access from all the accounts in the organization.
   cequence_is_multi_account_deployment	false for Terraform Deployment. For Terraform Deployment only Single Region is supported
   cequence_deployment_type	terraform
   cequence_log_level	Legal values are info, debug, and trace. Used for diagnostics and troubleshooting. Note that trace is more verbose than debug. For production use cases, set to info.
   cequence_auto_install_dependencies	true by default. When set to true, installs dependencies automatically. Note: Installs JQ, AWS CLI, Terraform CLI, ED and ZIP.
   cequence_auto_uninstall_dependencies	false by default. When set to true, uninstalls dependencies automatically. Note: Uninstalls JQ, AWS CLI, Terraform CLI, ED and ZIP.
   cequence_rest_api_enabled	true When set to true, enables REST APIs.
   cequence_http_api_enabled	true When set to true, enables HTTP APIs.
   cequence_auto_discover_apis	true When set to false, use the api-config.json file to explicitly list the APIs to enable for a single region.
   is_existing_http_log_group	false by default. Set to true when access logs are already enabled to a log group.
   cequence_http_log_group_name	cequence-http-api-access-logs is the default value. When you don't specify a name for the log group, Cequence creates a new log grooup with the default name.
   cequence_tf_auto_approve	Default is false. When set to true, the artifacts created or changed by an integration script are not presented for review.
   cequence_res_content_types	Ignore responses that do not match any content type specified as the value of this variable. We recommend not changing the default value.
   cequence_static_file_extensions	Ignore responses with a URI that matches any of the below extensions. We recommend not changing the default value.

5. Edit the environment properties file to change the values of specific variables.
   Replace the placeholders for the client ID and client Secret with the actual values. Type the URIs for the Cequence Bridge Authentication endpoint and Cequence Bridge Edge endpoint.

6. Enable the integration by running the following script.

   ./enable_aws_apigw_integration.sh

   The script scans all APIs in the AWS account and enables the integrations for deployed REST and HTTP APIs in all regions. As a best practice, confirm that the terraform.tfstate files are kept current, then re-run this script on a regular interval to discover and catalogue new APIs.

Disabling the integration

You can take several approaches to disabling the integration.

Disabling REST API functionality only

1. Open the .env file in a text editor.
2. Change the value of the cequence_rest_api_enabled variable to false.
3. Save and exit the editor.

4. Run the following script.

   ./disable_aws_apigw_integration.sh

Disabling HTTP API functionality only

1. Open the .env file in a text editor.
2. Change the value of the cequence_http_api_enabled variable to false.
3. Save and exit the editor.

4. Run the following script.

   ./disable_aws_apigw_integration.sh

Disabling all functionality

1. Open the .env file in a text editor.
2. Change the value of the cequence_rest_api_enabled variable to false.
3. Change the value of the cequence_http_api_enabled variable to false.
4. Save and exit the editor.

5. Run the following script.

   ./disable_aws_apigw_integration.sh

Destroying artifacts created by the integration enabling script

To delete all resources and artifacts created by the enabling script, open the .env file in a text editor

and change the values of the following variables.

1. cequence_rest_api_enabled to false.
2. cequence_http_api_enabled to false.
3. cequence_cfn_auto_delete_from_s3 to false.

Run the disabling script with the --destroy flag.

./disable_aws_apigw_integration.sh --destroy

Notes

1. AWS API Gateway limits log events to 1024 bytes. Log events larger than 1024 bytes, such as request and response bodies, are truncated by API Gateway before submission to CloudWatch logs.

2. For REST APIs where AWS truncates the request and response body, Cequence sets the body as below

{ "cq_body_truncated": "true" }

For REST APIs where the Cequence plugin is unable to capture the request and response if they are truncated, below header is set from the plugin and sent to the Cequence UAP Platform

"cq-discovery-only" = "true"

3. For all HTTP APIs below header is set from the plugin and sent to the Cequence UAP Platform since the request and response bodies is unavailable

"cq-discovery-only" = "true"

4. Set the retention period for CloudWatch Log Groups associated with APIs deployed in AWS API Gateway to at least 1 day, or according to the your retention policy.

5. The Request and Response body for HTTP APIs are not logged by default in AWS. Only the following variables are captured and sent to the Cequence UAP platform.

• requestTimeEpoch
• requestId
• accountId
• stage
• instance_id
• ip
• host
• http-version
• http-method
• uri-query-fragment
• status-code