You can configure the Cequence Unified API Protection (UAP) platform to export data to Kafka.

Configuring Kafka

Configure Apache Kafka to receive data exported from the Cequence UAP platform. 

Deploy or access a Kafka cluster

Set up a Kafka cluster to receive exported data. You can use a managed service or deploy Kafka on your own infrastructure.

1. For a managed service, use Confluent Cloud, Amazon MSK, or another cloud provider's managed offering.
2. For self-hosted deployment, see the Apache Kafka Quickstart.
3. Note the bootstrap server address (hostname and port) for your Kafka cluster.

Create a Kafka topic

Create a topic to store exported data from the Cequence UAP platform.

1. Use the Kafka command-line tool to create a topic.

   bin/kafka-topics.sh --create --topic cequence-uap-exports --bootstrap-server broker-host:port --partitions 3 --replication-factor 1

2. Adjust the values of the --partitions and --replication-factor parameters based on your throughput requirements and infrastructure.

3. Verify the topic was created.

   bin/kafka-topics.sh --list --bootstrap-server broker-host:port

Optional: Create a user and configure permissions

If your Kafka cluster uses SASL/PLAIN authentication or access control lists, create credentials and assign permissions.

For self-hosted Kafka with ACLs:

1. Create a principal and assign write permissions to the topic.

   bin/kafka-acls.sh --authorizer-properties zookeeper.connect=zk-host:port --add --allow-principal User:cequence-uap --operation Write --topic cequence-uap-exports

2. Note the username for later use.

For a managed Kafka service:

1. Follow your provider's documentation to create a user and assign topic-level produce permissions.
2. Note the username and password.

Optional: Configure network access

Ensure your Kafka brokers are reachable from the Cequence UAP environment. Update firewall rules or security groups to allow inbound connections on the Kafka port (default 9092). Restrict access to trusted IP addresses or networks.

Test the connection

Verify that you can write to and read from your Kafka topic. Use the Kafka command-line tools to test connectivity and permissions.

Write a test message to your topic:

bin/kafka-console-producer.sh --broker-list broker-host:port --topic cequence-uap-exports

Type a test message and press Enter. Then read the message back from the topic:

bin/kafka-console-consumer.sh --bootstrap-server broker-host:port --topic cequence-uap-exports --from-beginning --max-messages 1

If the test message appears, your connection and permissions are configured correctly.

Configuring the Cequence UAP platform

After configuring Kafka, configure the Cequence UAP platform to send data.

1. Log in to the Cequence UAP platform.
2. From the left navigation bar, select Integrations > Data Export.
3. In the Add a new integration section, choose a type from the Select a type drop-down.
   The new data export dialog appears with the Overview tab selected.
4. In the Overview tab, type the following information.
   • Integration name
   • Bootstrap Servers: A comma-separated list of Kafka Broker URLs.
   • Sink Topic: Specify a single sink topic.
5. Determine which event types you want to export. The following event types are available.
   • Risk events
   • Bot detection events
   • Bot mitigation events
   • Audit log events
6. For each event type you're exporting, click the corresponding tab, then perform the "Defining event criteria and export fields" procedure following this procedure.
7. Verify that the Enable Integration toggle is on. Click Save.

Defining event criteria and export fields

This procedure applies when you are currently defining a new data export integration in the Cequence UAP platform and have already configured the Overview tab.

1. Locate the event type section you want to configure.
   The following event types are available.
   1. Risk Events
   2. Bot Detection Events
   3. Bot Mitigation Events
   4. Audit Log Events
2. Enable the event type toggle.
3. In Event Criteria, click the pencil icon.
   The criteria builder appears.
4. In the Criteria pane, choose OR (match any) or AND (match all).
5. Choose a field, a condition, and a value.
   For example, country (field) equals (condition) mx (value), or Method does not equal GET.
6. To define another criterion, repeat the previous step.
7. Click Add.
8. In the Export Fields section, click the pencil icon.
   The field selector appears.
9. Select individual fields, or click Select all.
   Type a string in the search field to filter fields.
10. Choose a transformation from the drop-down for each selected field.
    The following transformations are available.
    • hash
    • show only last 4 characters
    • format preserving encryption (FPE)
    • None
11. Click Set.
12. Enter the transformation script.