# IAB bot detection enrichment
> Identify bot and spider traffic using the IAB/ABC International Spiders and Bots List based on IP and user agent.
> Source: https://docs.snowplow.io/docs/pipeline/enrichments/available-enrichments/iab-enrichment/
> Documentation index: https://docs.snowplow.io/llms.txt

The IAB Spiders and Robots enrichment uses the [IAB/ABC International Spiders and Bots List](https://iabtechlab.com/software/iababc-international-spiders-and-bots-list/) to determine whether an event was produced by a user or a robot/spider based on its IP address and user agent.

The Interactive Advertising Bureau (IAB) is an advertising business organization that develops industry standards, conducts research, and provides legal support for the online advertising industry.

Their internationally recognized list of spiders and bots is regularly maintained to try and identify the IP addresses of known bots and spiders.

## How the enrichment works

This enrichment performs several checks using the IAB database files and your custom override lists.

Here is the logic it uses. The values in parentheses are for the `reason` field in the IAB [entity](/docs/fundamentals/entities/) attached to the event.

```mermaid
flowchart TD
  IncList{{User agent in the custom **include** list?}}:::nowrap
  IncList -->|Yes| NotBot1(["Not bot"])
  IncList -->|No| ExcList{{User agent in the custom **exclude** list?}}:::nowrap
  ExcList -->|Yes| Bot1("**Bot**<br/>(FAILED_UA_EXCLUDE)")
  ExcList -->|No| IpFile{{IP address in the IAB IP file?}}:::nowrap
  IpFile -->|Yes| Bot2("**Bot**<br/>(FAILED_IP_EXCLUDE)")
  IpFile -->|No| NullUA{{User agent is null?}}:::nowrap
  NullUA -->|Yes| NotBot2(["Not bot"])
  NullUA -->|No| IncFile{{User agent in the IAB **include** file?}}:::nowrap
  IncFile -->|Yes| ExcFile{{User agent in the IAB **exclude** file?}}:::nowrap
  IncFile -->|No| Bot3("**Bot**<br/>(FAILED_UA_INCLUDE)")
  ExcFile -->|Yes| Bot4("**Bot**<br/>(FAILED_UA_EXCLUDE)")
  ExcFile -->|No| NotBot3(["Not bot"])
  classDef nowrap white-space:nowrap
```

A user agent string will match one of the lists or files if it contains a string from that list or file. The matching is case-insensitive.

For example, the user agent string `Chrome Chrome MyBot Chrome` will match an entry named `mybot`.

## Configuration

The enrichment takes these parameters:

| Parameter              | Required | Description                                                                                                                       |
| ---------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `ipFile`               | ✅        | Path to IP address exclude file. Already provided for CDI customers.                                                              |
| `excludeUseragentFile` | ✅        | Path to user agent exclude file. Already provided for CDI customers.                                                              |
| `includeUseragentFile` | ✅        | Path to user agent include file. Already provided for CDI customers.                                                              |
| `includeUseragents`    | ❌        | Additional user agent patterns to classify as browsers, extending `includeUseragentFile`. Case-insensitive substring match.       |
| `excludeUseragents`    | ❌        | Additional user agent patterns to classify as spiders/robots, extending `excludeUseragentFile`. Case-insensitive substring match. |

**Console:**

Configure the parameters in the Console enrichment editor. Keep the Console defaults for the `uri` fields. For example:

```json
{
  "ipFile": {
    "database": "ip_exclude_current_cidr.txt",
    "uri": "<use default value from Console>"
  },
  "excludeUseragentFile": {
    "database": "exclude_current.txt",
    "uri": "<use default value from Console>"
  },
  "includeUseragentFile": {
    "database": "include_current.txt",
    "uri": "<use default value from Console>"
  },
  "excludeUseragents": [
    "BotNotCaughtByIAB"
  ],
  "includeUseragents": [
    "MyServerSideTrackerThatIsNotABot"
  ]
}
```

**Self-Hosted:**

For Self-Hosted, [provide a complete JSON](/docs/pipeline/enrichments/managing-enrichments/terraform/). For example:

```json
{
  "schema": "iglu:com.snowplowanalytics.snowplow.enrichments/iab_spiders_and_robots_enrichment/jsonschema/1-0-1",
  "data": {
    "name": "iab_spiders_and_robots_enrichment",
    "vendor": "com.snowplowanalytics.snowplow.enrichments",
    "enabled": false,
    "parameters": {
      "ipFile": {
        "database": "ip_exclude_current_cidr.txt",
        "uri": "<your file location>"
      },
      "excludeUseragentFile": {
        "database": "exclude_current.txt",
        "uri": "<your file location>"
      },
      "includeUseragentFile": {
        "database": "include_current.txt",
        "uri": "<your file location>"
      },
      "excludeUseragents": [
        "BotNotCaughtByIAB"
      ],
      "includeUseragents": [
        "MyServerSideTrackerThatIsNotABot"
      ]
    }
  }
}
```

***

> **Tip: Testing with Micro**
>
> Unsure if your enrichment configuration is correct or works as expected? You can easily test it using [Snowplow Micro](/docs/testing/snowplow-micro/), either [through Console](/docs/testing/snowplow-micro/console/) or [on your machine](/docs/testing/snowplow-micro/local/).

### IAB files

> **Tip: Snowplow CDI**
>
> If you're using Snowplow CDI, you don't need to configure these. Use the default values provided in Console.

There are three configuration fields that correspond to the IAB/ABC database files:

| Field name             | Description                                                       |
| ---------------------- | ----------------------------------------------------------------- |
| `ipFile`               | Denylist of IP addresses considered to be robots or spiders       |
| `excludeUseragentFile` | Denylist of user agent strings considered to be robots or spiders |
| `includeUseragentFile` | Allowlist of user agent strings considered to be browsers         |

All three are mandatory and must have two inner fields:

- The `database` field containing the name of the database file.
- The `uri` field containing the URI of the bucket in which the database file is found. This field supports `http`, `https`, `gs`, and `s3` schemes.

The database filenames must be as follows:

| Field name                      | Filename                        |
| ------------------------------- | ------------------------------- |
| `ipFile.database`               | `"ip_exclude_current_cidr.txt"` |
| `excludeUseragentFile.database` | `"exclude_current.txt"`         |
| `includeUseragentFile.database` | `"include_current.txt"`         |

### Custom user agent lists

Applies to **Enrich 6.8.0+**[Latest: 6.14.1](/docs/api-reference/versions/#feature-availability-by-version "The latest Enrich version is 6.14.1")

In addition to the IAB database files, you can provide custom lists of user agent strings to supplement or override the detection. Two optional fields can be added to the `parameters` section:

| Field name          | Description                                                     |
| ------------------- | --------------------------------------------------------------- |
| `excludeUseragents` | A list of user agent strings to be treated as robots or spiders |
| `includeUseragents` | A list of user agent strings to be treated as browsers          |

Both fields accept a JSON array of strings. They are optional and default to empty lists if omitted.

A user agent matching `excludeUseragents` produces the following output values:

| Field           | Value               |
| --------------- | ------------------- |
| `spiderOrRobot` | `true`              |
| `category`      | `SPIDER_OR_ROBOT`   |
| `reason`        | `FAILED_UA_EXCLUDE` |
| `primaryImpact` | `UNKNOWN`           |

A user agent matching `includeUseragents` produces the following output values:

| Field           | Value        |
| --------------- | ------------ |
| `spiderOrRobot` | `false`      |
| `category`      | `BROWSER`    |
| `reason`        | `PASSED_ALL` |
| `primaryImpact` | `NONE`       |

Example:

```json
"excludeUseragents": ["my-custom-bot/1.0", "internal-crawler"],
"includeUseragents": ["my-legitimate-app/2.0"]
```

This is useful when you need to flag or allowlist specific user agents without modifying the IAB database files themselves.

For example, you can use `includeUseragents` to stop the enrichment from classifying events from your own server-side trackers as bots. See [Exclude your own server-side tracking](/docs/events/filtering-bot-events/#exclude-your-own-server-side-tracking) for details.

> **Tip: Use a custom user agent**
>
> Configure your server-side trackers to send a custom user agent string that identifies your application, such as `my-server-side-tracker/1.0`. Then add this string to `includeUseragents`. Avoid adding a default HTTP client user agent such as `okhttp/4.2.2`: genuine bots often use the same libraries, so the enrichment would also classify them as browsers.
>
> To set a custom user agent, see the documentation for the [Node.js](/docs/sources/node-js-tracker/configuration/#set-the-useragent-withsetuseragent), [Java](/docs/sources/java-tracker/tracking-specific-client-side-properties/#overriding-autogenerated-event-properties), [Python](/docs/sources/python-tracker/subject/#set_useragent), [Go](/docs/sources/golang-tracker/adding-extra-data-the-subject-class/#set-the-useragent-with-setuseragent), [.NET](/docs/sources/net-tracker/subject/#setuseragent), [PHP](/docs/sources/php-tracker/subjects/#setuseragent), [Ruby](/docs/sources/ruby-tracker/adding-data-events/#adding-user-and-platform-data-with-subject), [Scala](/docs/sources/scala-tracker/subject-methods/#setting-the-useragent-withsetuseragent), [Rust](/docs/sources/rust-tracker/initialization-and-configuration/#configuration-of-subject-informationsubject), or [C++](/docs/sources/c-tracker/adding-extra-data-the-subject-class/#set-custom-user-agent-withset_useragent) tracker.

## Input

This enrichment uses the following [fields](/docs/fundamentals/canonical-event/) of a Snowplow event:

- `useragent` to determine an event's user agent, which will be validated against the databases described in `excludeUseragentFile` and `includeUseragentFile`, as well as the custom lists in `excludeUseragents` and `includeUseragents`
- `user_ipaddress` to determine an event's IP address, which will be validated against the database described in `ipFile`
- `derived_tstamp` to determine an event's datetime. Some entries in the Spiders and Robots List can be considered "stale", and will be given a `category` of `INACTIVE_SPIDER_OR_ROBOT` rather than `ACTIVE_SPIDER_OR_ROBOT` based on their age

## Output

This enrichment adds a `spiders_and_robots` entity to the enriched event.

### `spiders_and_robots`

**Type:** Entity

Schema for an entity generated by the IAB Spiders and Robots enrichment

**Properties:**

| Property                  | Description                                                                                                                                                                                                                  |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spiderOrRobot` _boolean_ | _Required._ true if the IP address or user agent checked against the list is a spider or robot, false otherwise                                                                                                              |
| `category` __             | _Required._ Category based on activity if the IP/UA is a spider or robot, BROWSER otherwise Must be one of: `SPIDER_OR_ROBOT`, `ACTIVE_SPIDER_OR_ROBOT`, `INACTIVE_SPIDER_OR_ROBOT`, `BROWSER`                               |
| `reason` __               | _Required._ Type of failed check if the IP/UA is a spider or robot, PASSED\_ALL otherwise Must be one of: `FAILED_IP_EXCLUDE`, `FAILED_UA_INCLUDE`, `FAILED_UA_EXCLUDE`, `PASSED_ALL`                                        |
| `primaryImpact` __        | _Required._ Whether the spider or robot would affect page impression measurement, ad impression measurement, both or none Must be one of: `PAGE_IMPRESSIONS`, `AD_IMPRESSIONS`, `PAGE_AND_AD_IMPRESSIONS`, `UNKNOWN`, `NONE` |