This article discusses how to use Terraform to integrate the Amazon API Gateway with the Cequence platform. You can also integrate using CloudFormation. This article is intended for AWS administrators who deploy and manage the integration on an individual AWS account.

Further details on CloudWatch are available in Amazon's documentation. This link goes to an external site that may change over time.

Integration architecture

The Amazon API Gateway integration for the Cequence platform is made of the following components.

Architecture diagram showing API Gateway, CloudWatch, EventBridge Scheduler, and Lambda

Amazon API Gateway

Receives incoming requests from clients and routes the requests to the appropriate API backend services or applications.

Generates detailed log entries for each request and response, including headers, body, query parameters, and other metadata relevant to Cequence platform analysis.

Amazon CloudWatch

Receives log events from API Gateway and stores them in log groups.

Amazon CloudWatch limitations limit batch sizes to 1MB and 5000 transactions per second, per region. Use the Service Quotas service to change the transaction-per-second limit. The batch size limit cannot be increased. This link goes to an external site that may change over time.

Amazon EventBridge Scheduler

A serverless event management service that enables the triggering of an AWS service at a scheduled interval. Amazon EventBridge Scheduler triggers an AWS Lambda function that fetches API Gateway log events every minute.

AWS Lambda function

Triggered every minute by Amazon EventBridge Scheduler. Pulls the API Gateway log entries from AWS CloudWatch, transforms the entries into the payload format used by the Cequence platform, aggregates the entries over the last minute of activity, and posts the batch to the Cequence platform for analysis.

The procedures in this article automate the Lambda function configuration by discovering all API gateways and their deployed stages, then configuring the dependencies needed. A Terraform script that provides this automation can be applied to an individual AWS account.

Cequence integration API flow

The integration moves data through the following stages.

The client sends an API request to AWS API endpoints through the AWS gateway.
API request and response data is logged to the corresponding CloudWatch log stream.
A Lambda function runs every 60 seconds, pulls all applicable transactions, and sends them to the Cequence platform using the Edge or Bridge endpoint.

Limitations

Before you begin, review the following limitations of the Cequence platform's integration with AWS API Gateway.

The Cequence plugin does not capture HTTP API headers and payload, due to AWS API Gateway limitations. The plugin adds the following placeholder values to the transaction instead.
```
Request Headers:
"cq-no-header: true"

Response Body:
{"cq-endpoints-only": "true"}
```
Transactions can take 60 to 120 seconds to reach the Cequence platform.
Request and response bodies over 32KB are truncated to 32KB for the Cequence platform. When truncation occurs, the plugin adds the header cq-body-truncated: true.

What you'll need

Cequence Bridge release 5.6.2 or later.
Terraform release v1.9.5 or later.
AWS CLI release 2.14.0 or later.
Current installations of node.js and npm.
Access to the jq, ed, and zip UNIX tools.
A client ID and client secret generated from the Cequence platform.

Generating a client ID and client secret

Several Cequence components must authenticate to the Cequence platform in order to transmit and receive data. The following procedure creates the authentication credentials those components require.

Log in to the Cequence platform management portal. The URL for the management portal typically has the form https://ui.<your tenant name>.<domain>. Replace <your tenant name> with the name of your Cequence tenant organization. Replace <domain> with your domain name.
Navigate to General Settings > User Management.
The User Management pane appears.
Select the Clients tab.
Select Add New Client.
The new client dialog box appears.
In the Client Name field, type a name for the client.
This name becomes the client ID. Note the client ID for later use.
Enable the Traffic Management toggle.
(Optional) To change the token lifespan from the default of 1800 seconds, type a whole number of seconds in Token Lifespan.
Select Save.
A dialog box displaying the client secret appears.
Select the copy icon to copy the secret to the clipboard, then select Close.
The client list appears.
Note the value of the client secret for later use. For security reasons, this value does not display again.

Prerequisites

An AWS account with sufficient permissions to manage API Gateway, Lambda, IAM, CloudWatch, EventBridge Scheduler, and EC2.
Working REST and HTTP APIs deployed to various stages in API Gateway in an AWS account.
Access to an AWS CLI user account with the privileges described in the following section.
An AWS policy named iam_apigw_lambda_cloudwatch_eventbridge_cq_integration, attached to the AWS CLI user account described above.

AWS CLI user privileges

Setting up the Amazon API Gateway integration requires an AWS CLI user account with an access key, secret, and the privileges listed in the following JSON. These restricted permissions apply to resources that Terraform creates.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:GetRole",
                "iam:DeleteRole",
                "iam:PutRolePolicy",
                "iam:DeleteRolePolicy",
                "iam:GetRolePolicy",
                "iam:PassRole",
                "iam:AttachRolePolicy",
                "iam:DetachRolePolicy"
            ],
            "Resource": [
                "arn:aws:iam::*:role/cequence_*",
                "arn:aws:iam::*:role/AmazonAPIGatewayPushToCloudWatchLogs2-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lambda:CreateFunction",
                "lambda:DeleteFunction",
                "lambda:GetFunction",
                "lambda:UpdateFunctionConfiguration",
                "lambda:UpdateFunctionCode",
                "lambda:ListVersionsByFunction",
                "lambda:GetFunctionCodeSigningConfig"
            ],
            "Resource": "arn:aws:lambda:*:*:function:cequence-api-gateway-cloudwatch-lambda1-*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "scheduler:CreateSchedule",
                "scheduler:DeleteSchedule",
                "scheduler:GetSchedule"
            ],
            "Resource": "arn:aws:scheduler:*:*:schedule/default/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:DeleteLogGroup",
                "logs:PutRetentionPolicy"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:/aws/lambda/cequence-api-gateway-cloudwatch-lambda1-*",
                "arn:aws:logs:*:*:log-group:/aws/apigateway/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeRegions",
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "apigateway:GET",
                "apigateway:POST",
                "apigateway:PUT",
                "apigateway:PATCH",
                "apigateway:DELETE"
            ],
            "Resource": [
                "arn:aws:apigateway:*::/restapis/*/stages/*",
                "arn:aws:apigateway:*::/restapis/*/stages",
                "arn:aws:apigateway:*::/restapis/*/deployments",
                "arn:aws:apigateway:*::/account"
            ]
        }
    ]
}

The Terraform script requires the following roles and permissions.

API Gateway role (aws_iam_role.api_gateway_cloudwatch)

Purpose: allows API Gateway to push logs to CloudWatch.
Trust policy: allows only apigateway.amazonaws.com to assume this role.
Permissions: uses the AWS managed policy AmazonAPIGatewayPushToCloudWatchLogs.
Used by: API Gateway account settings for CloudWatch logging.

Lambda role (aws_iam_role.lambda_role)

Purpose: enables the Lambda function to interact with CloudWatch logs.
Trust policy: allows only lambda.amazonaws.com to assume this role.
Custom policy permissions: logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, logs:FilterLogEvents, logs:Unmask.
Scope: full access to CloudWatch logs resources (arn:aws:logs:*:*:*).

EventBridge role (aws_iam_role.eventbridge_role)

Purpose: allows EventBridge to invoke the Lambda function.
Trust policy: allows only scheduler.amazonaws.com to assume this role.
Custom policy permissions: lambda:InvokeFunction.
Scope: limited to only the specific Lambda function ARN.

Installing the integration with Terraform

The compressed archive file that contains the integration bundle contains a file named main.tf. Terraform uses this file during installation.

The main.tf configuration file is structured into two sections, a section for global AWS resources and a section for regional resources. This approach provides comprehensive API scanning and monitoring across regions, maintaining resource isolation and regional compliance.

The Global AWS resources section creates account-wide resources that are not region-specific. These resources are shared across all regions in the AWS account.
The Regional resources section creates and deploys region-specific components. These resources enable the monitoring of APIs deployed across multiple AWS regions within the same account. Each region has an independent set of monitoring resources, including the Cequence Lambda function, the EventBridge scheduler, CloudWatch log groups, and API Gateway configurations.

Download the compressed archive file that contains the plugin bundle.
Extract the archive file. The extraction creates the cequence directory structure.
```
tar -xvf cequence-aws-api-gateway.tar.gz
cd cequence
```
Navigate to the scripts directory and grant execute permission to the scripts.
```
cd scripts
chmod +x *.sh
```

Create a copy of the example environment properties file.

cp .env.example .env

The .env file in the bundle contains variables for both Terraform and CloudFormation deployments. The following variables affect Terraform deployments. Leave all other variables at their default values.

# =============================================================================
# 1. MANDATORY
# =============================================================================
cequence_is_auth_enabled=true
cequence_client_id=your_client_id_here
cequence_client_secret=your_client_secret_here
cequence_auth_endpoint=https://example.cequence.ai/auth/realms/example/protocol/openid-connect/token
cequence_edge_endpoint=https://edge.example.cequence.ai/api-transactions
# cloudformation | terraform
cequence_deployment_type=terraform
cequence_rest_api_enabled=true
cequence_http_api_enabled=true

# =============================================================================
# 2. REGIONS
#     all  |  us-east-1,us-west-2  |  scripts/regions-config.csv
# =============================================================================
cequence_aws_regions=all

# =============================================================================
# 3. MONITORING (optional)
# =============================================================================
cequence_http_log_group_name=cequence-http-api-access-logs
# true = API Gateway already has a CloudWatch Logs role attached (skip attach)
cequence_apigw_cloudwatch_role_already_attached=false
# false = keep existing HTTP DestinationArns; true = rewrite all to the Cequence log group
cequence_http_overwrite_log_destinations=false
cequence_log_retention_days=1
cequence_sampling_enabled=false
cequence_default_sampling_rate=1.0
cequence_api_discovery_ttl_ms=300000
# CloudWatch fetch window in milliseconds, passed through to the Lambda function
cequence_log_fetch_interval_ms=60000
cequence_log_fetch_limit=10000
cequence_log_ingestion_buffer_ms=60000
# Edge or Bridge HTTP connection pool size (default 5)
cequence_connection_pool_size=5
# Maximum POST body size per Edge request in bytes. Default 1000000. Lower this value if you see HTTP 413 errors.
# cequence_max_message_size_bytes=1000000

# =============================================================================
# 4. MULTI-ACCOUNT AND ORGANIZATIONS
#     Not applicable to Terraform deployments. Leave both set to false.
#     Terraform deployments target a single AWS account, and can target one region or several.
# =============================================================================
cequence_is_multi_account_deployment=false
cequence_is_aws_organization_deployment=false

# =============================================================================
# 5. LAMBDA AND COLLECTOR RUNTIME
# =============================================================================
cequence_log_level=info
cequence_local_log_mode=false
cequence_auto_install_dependencies=true
cequence_auto_uninstall_dependencies=false
# When true, changes are applied without an interactive review prompt.
cequence_tf_auto_approve=true

# =============================================================================
# 6. CONTENT FILTERING (optional)
# =============================================================================
# Ignore responses whose content type does not match cequence_res_content_types. Keep the default value unless directed otherwise.
# cequence_res_content_types=
# Ignore responses whose URI matches cequence_static_file_extensions. Keep the default value unless directed otherwise.
# cequence_static_file_extensions=

# =============================================================================
# 7. OPTIONAL FILE CONFIGS (JSON)
#     API allow list: cp api-config.example.json scripts/api-config.json
# =============================================================================
cequence_auto_discover_apis=true
# When false, use api-config.json to explicitly list the APIs to enable for the single configured region.
# cequence_api_config_path=scripts/api-config.json

Edit the environment properties file to change the values of the mandatory variables. Replace the placeholders for the client ID and client secret with the values you generated earlier in this article, and provide the URIs for the Cequence Bridge authentication endpoint and Cequence Bridge Edge endpoint.

Deploy the integration.

./deploy.sh

The script scans all APIs in the AWS account and enables the integration for deployed REST and HTTP APIs in the configured regions. A successful run shows a deployment summary similar to the following.

HTTP API access logging enabled for all targets.
[SUCCESS] HTTP API access logging enablement completed.
[SUCCESS] Deployment finished successfully.
[SUCCESS] What was deployed:
[SUCCESS]   Method:              terraform
[SUCCESS]   Infrastructure:      Terraform modules applied in the regions below
[SUCCESS]   Regions:             us-west-2,ap-south-1

As a best practice, keep the terraform.tfstate file current, and re-run this script on a regular interval to discover and catalogue new APIs.

The transactions appear on the Cequence platform Detection and Mitigation dashboard.

Enabling sampling

Sampling controls what proportion of traffic the Lambda function forwards to the Cequence platform. Configure sampling before deployment by setting the following variables in the .env file.

cequence_sampling_enabled=true
cequence_default_sampling_rate=0.5

When cequence_default_sampling_rate is set to 1.0, all traffic reaches the Cequence platform. When set to a lower value, such as 0.5, that percentage of traffic reaches the Cequence platform and the Lambda function drops the remainder.

Updating and disabling the integration

The upgrade.sh script applies any changes you make to the .env file, including disabling REST or HTTP API capture. Edit the .env file, then run the script from the scripts directory.

./upgrade.sh

A successful run shows output similar to the following.

HTTP API access logging enabled for all targets.
[SUCCESS] HTTP API access logging enablement completed.
[SUCCESS] Upgrade done
[SUCCESS]   cequence_deployment_type: terraform
[SUCCESS]   cequence_auto_install_dependencies: true
[SUCCESS]   cequence_rest_api_enabled: true
[SUCCESS]   cequence_http_api_enabled: true
[SUCCESS]   cequence_enabled_regions: ap-south-1
[SUCCESS]   cequence_auto_discover_apis: true

Disabling REST API functionality only

Change the value of the cequence_rest_api_enabled variable to false in the .env file, then run ./upgrade.sh.

Disabling HTTP API functionality only

Change the value of the cequence_http_api_enabled variable to false in the .env file, then run ./upgrade.sh.

Disabling all functionality

Change the value of the cequence_rest_api_enabled variable and the cequence_http_api_enabled variable to false in the .env file, then run ./upgrade.sh.

Destroying artifacts created by the deployment script

To delete all resources and artifacts created by the deployment script, run the destroy script with the destroy flag.

./destroy.sh --destroy

A successful run shows output similar to the following.

Destroy complete! Resources: 19 destroyed.
[SUCCESS] Terraform destroy completed.
[INFO] Running force cleanup to catch any residual resources.
[INFO] Clearing local Terraform state.
[SUCCESS] Destroy finished successfully.
[SUCCESS] What was removed:
[SUCCESS]   Method:              terraform destroy (with force cleanup of residuals)
[SUCCESS]   Regions targeted:    us-west-2,ap-south-1
[SUCCESS]   REST API capture:    was true
[SUCCESS]   HTTP API capture:    was true
[SUCCESS] Cequence AWS API Gateway integration resources should no longer be present.

When resources remain after a destroy, check the AWS console for Lambda, EventBridge, IAM, and log groups in the target regions, or re-run the destroy script.

The deployment, upgrade, and destroy scripts are an interface to Terraform. They do not use AWS CloudFormation.

Troubleshooting

The following issue has a known resolution.

When the test phase produces the error "errorMessage": "Cannot find package 'undici' imported from /var/task/index.mjs", the Node module was not properly deployed to Lambda. Verify the contents of the deployment zip file and upload it again.

Notes

The following notes describe logging behavior and log retention.

AWS API Gateway limits log events to 1024 bytes. Log events larger than 1024 bytes, such as request and response bodies, are truncated by API Gateway before submission to CloudWatch logs. This link goes to an external site that may change over time.
For HTTP APIs, request and response bodies are unavailable. The plugin sets the following response body and sends it to the Cequence platform.
```
{"cq-endpoints-only": "true"}
```
Set the retention period for CloudWatch log groups associated with APIs deployed in AWS API Gateway to at least one day, or according to your retention policy.

Search

AWS API Gateway Integration using Terraform

AWS API Gateway Integration using Terraform

Integration architecture

Amazon API Gateway

Amazon CloudWatch

Amazon EventBridge Scheduler

AWS Lambda function

Cequence integration API flow

Limitations

What you'll need

Generating a client ID and client secret

Prerequisites

AWS CLI user privileges

API Gateway role (aws_iam_role.api_gateway_cloudwatch)

Lambda role (aws_iam_role.lambda_role)

EventBridge role (aws_iam_role.eventbridge_role)

Installing the integration with Terraform

Enabling sampling

Updating and disabling the integration

Disabling REST API functionality only

Disabling HTTP API functionality only

Disabling all functionality

Destroying artifacts created by the deployment script

Troubleshooting

Notes