Amazon API Gateway is a fully managed AWS service for creating, publishing, maintaining, monitoring, and securing APIs at any scale. API Gateway acts as the entry point for applications that access data, business logic, or functionality from backend services.

Amazon API Gateway

API Gateway receives incoming requests from clients and routes each request to the appropriate backend service. API Gateway also generates detailed log entries for each request and response, including headers, body, query parameters, and other metadata relevant to Cequence Unified API Protection (UAP) platform analysis.

Amazon CloudWatch

Amazon CloudWatch receives log events from API Gateway and stores the events in log groups.

Note: Amazon CloudWatch imposes limits of 1 MB per batch and 5,000 transactions per second per region. Use the Service Quotas service to increase the transactions-per-second limit. The batch size limit cannot be increased.

Amazon EventBridge Scheduler

Amazon EventBridge Scheduler is a serverless event management service that triggers an AWS Lambda function at a scheduled interval. The scheduler triggers the Lambda function once per minute to fetch API Gateway log events.

AWS Lambda function

The Lambda function runs once per minute on the EventBridge Scheduler trigger. The Lambda function pulls API Gateway log entries from AWS CloudWatch, transforms the entries into the payload format used by the Cequence UAP platform, aggregates the entries over the preceding minute of activity, and posts the batch to the Cequence UAP platform for analysis.

The procedures in this article automate the Lambda function configuration by discovering all API gateways and their deployed stages, then configuring the required dependencies. A Terraform script provides this automation and can be applied to an individual AWS account.

Generating a client ID and client secret

Several Cequence components must authenticate to the Cequence UAP platform to transmit and receive data. The following procedure creates the authentication credentials required for this authentication.

1. Log in to the UAP management portal. The URL takes the form https://ui.<your-tenant-name>.<domain>. Replace <your-tenant-name> with the name of your Cequence tenant organization and replace <domain> with your domain name.
2. Navigate to General Settings > User Management. The User Management pane appears.
3. Select the Clients tab.
4. Select Add New Client. The new client dialog box appears.
5. In the Client Name field, type the client name. This name becomes the client ID. Note the client ID for later use.
6. Enable the Traffic Management toggle.
7. (Optional) To change the token lifespan from the default of 1,800 seconds, type a whole number of seconds in the Token Lifespan field.
8. Select Save. A dialog box appears displaying the client secret.
9. Select the blue Copy icon to copy the secret to the clipboard, then select Close. The client list appears.
10. Note the client secret value for later use. The UAP management portal does not display this value again.

Prerequisites

• An AWS account with sufficient permissions to manage API Gateway, Lambda, IAM, CloudWatch, EventBridge Scheduler, and EC2.
• Working APIs deployed to one or more stages in API Gateway (REST, HTTP, or both).
• An AWS CLI user with an access key and secret.
• Terraform CLI release 1.9.5 or later.

• An AWS policy attached to the AWS CLI user (for example, named iam_apigw_lambda_cloudwatch_eventbridge_cq_intregration) that permits the following actions.

  • Reading available regions.
  • Creating, updating, and deleting IAM roles and policies used by API Gateway, Lambda, and EventBridge.
  • Managing Lambda functions, CloudWatch Logs, API Gateway resources, and required S3 buckets.
  • Managing CloudFormation stacks and reading AWS Organizations details when needed.

  See the example policy in docs/iam/example-user-iam-policy.json.

Setting up the Cequence and AWS API Gateway integration using Terraform

The following procedures set up the integration for REST and HTTP APIs using Terraform.

Step 1: Download and unzip the bundle

1. Create a folder named cequence_aws_apigw and place the downloaded .tar.gz file into the folder.

2. From the cequence_aws_apigw folder, run the following commands to extract the archive and make all scripts executable.

   tar -xvf cequence-aws-api-gateway-x.x.xxxxxxx.tar.gz
   cd cequence/scripts
   chmod +x *.sh

Step 2: Create the environment properties file

1. From the cequence/scripts directory, copy the example environment file.

   cp .env.example .env
   vi .env

2. Edit the variables in .env as needed. Variables not listed below can remain at their default values.

Remote S3 backend is supported for Terraform state (tfstate) and is enabled by default. The following table describes the key variables in the environment file.

For the full list of variables with inline comments, see the .env.example file in the distribution bundle.

Step 3: Run the enable integration script

1. From the cequence/scripts directory, run the following command.

   ./enable_aws_apigw_integration.sh

   The script scans all APIs in the AWS account and enables the integration for all deployed REST and HTTP APIs across all configured stages.

Verifying logs in the Cequence UAP platform

After the integration is enabled, Lambda statistics and batch processing behavior are visible in the Cequence UAP platform through the metrics the integration exposes. The following is an example of the statistics payload.

{
  "log_group_count": 25,
  "total_events_fetched": 3780000,
  "total_transactions_parsed_from_streams": 223500,
  "total_batches": 750,
  "successful_batches": 750,
  "failed_batches": 0,
  "average_batch_size": 298,
  ...
}

Disabling the integration

The following procedures describe the available options for disabling the integration.

Disable only the REST API integration

1. In the .env file, set cequence_rest_api_enabled=false.
2. From the cequence/scripts directory, run ./disable_aws_apigw_integration.sh.

Disable only the HTTP API integration

1. In the .env file, set cequence_http_api_enabled=false.
2. From the cequence/scripts directory, run ./disable_aws_apigw_integration.sh.

Disable both the REST API and HTTP API integrations

1. In the .env file, set both cequence_rest_api_enabled=false and cequence_http_api_enabled=false.
2. From the cequence/scripts directory, run ./disable_aws_apigw_integration.sh. The EventBridge cron scheduler is deleted, but other integration resources remain.

Destroy all integration artifacts

To remove all resources and artifacts created by the enable integration script, follow this procedure.

1. In the .env file, set the following values.

   cequence_rest_api_enabled=false
   cequence_http_api_enabled=false

2. From the cequence/scripts directory, run the following command.

   ./disable_aws_apigw_integration.sh --destroy

Note: The shell script is an interface to Terraform.

API limitations

HTTP APIs

Request and response bodies for HTTP APIs are not logged. The integration captures and sends only the following attributes to the Cequence UAP platform.

requestTimeEpoch, requestId, accountId, stage, instance_id, ip, host,
http-version, http-method, uri-query-fragment, status-code

For details, see the AWS HTTP API logging variables documentation.

REST APIs

When a REST API request or response body exceeds 1 KB, the integration sends a truncation placeholder in place of the body. The placeholder format depends on the content type.

When a REST API request and response are truncated and cannot be fully captured, the integration adds the header "cq-discovery-only": "true" to the payload. The integration also adds this header to all HTTP API payloads, since HTTP API bodies are not available.

For HTTP API payloads, the integration sets the body to {"cq-endpoints-only": "true"}.

Troubleshooting

The following table describes a known error and its resolution.