Track AI crawlers

AI crawlers such as GPTBot and ClaudeBot don't run JavaScript, so the script never sees them. A small snippet on your server checks each request's user agent and reports AI crawler hits to Calmstats, so the AI Traffic report shows which crawlers read which pages.

You'll need an API key from Settings → API keys. Only the site owner's key can report hits.

Next.js

1. Set the API key as CALMSTATS_API_KEY in your app's environment.
2. Save this as proxy.ts next to your app folder, or merge it into the one you have.
3. Deploy.

proxy.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from 'next/server';

const AI_CRAWLERS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|Bytespider|Amazonbot|CCBot|cohere-training-data-crawler|cohere-ai|MistralAI-User|DuckAssistBot|YouBot/i;

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const userAgent = request.headers.get('user-agent') ?? '';
  if (AI_CRAWLERS.test(userAgent) && process.env.CALMSTATS_API_KEY) {
    event.waitUntil(
      fetch('https://calmstats.com/api/ai-bots', {
        method: 'POST',
        headers: {
          'content-type': 'application/json',
          authorization: `Bearer ${process.env.CALMSTATS_API_KEY}`,
        },
        body: JSON.stringify({
          website_id: 'YOUR-SITE-ID',
          hits: [{
            user_agent: userAgent,
            path: request.nextUrl.pathname,
            ip: request.headers.get('x-forwarded-for')?.split(',')[0]?.trim(),
            timestamp: new Date().toISOString(),
          }],
        }),
      }).catch(() => {}),
    );
  }
  return NextResponse.next();
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

WordPress

1. Add define('CALMSTATS_API_KEY', 'your key'); to wp-config.php.
2. Save this as wp-content/mu-plugins/calmstats-ai-crawlers.php (create the mu-plugins folder if it is missing). It loads automatically.
3. If a page cache such as Cloudflare, WP Rocket or your host's cache serves crawlers before WordPress runs, use the Cloudflare Worker instead.

calmstats-ai-crawlers.php
<?php
/**
 * Plugin Name: Calmstats AI crawlers
 * Description: Reports AI crawler visits to Calmstats. Crawlers don't run JavaScript.
 */

add_action('shutdown', function () {
    if (!defined('CALMSTATS_API_KEY') || empty($_SERVER['HTTP_USER_AGENT'])) {
        return;
    }
    $user_agent = (string) $_SERVER['HTTP_USER_AGENT'];
    if (!preg_match('/GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|Bytespider|Amazonbot|CCBot|cohere-training-data-crawler|cohere-ai|MistralAI-User|DuckAssistBot|YouBot/i', $user_agent)) {
        return;
    }
    wp_remote_post('https://calmstats.com/api/ai-bots', array(
        'blocking' => false,
        'timeout'  => 2,
        'headers'  => array(
            'Content-Type'  => 'application/json',
            'Authorization' => 'Bearer ' . CALMSTATS_API_KEY,
        ),
        'body' => wp_json_encode(array(
            'website_id' => 'YOUR-SITE-ID',
            'hits' => array(array(
                'user_agent'  => substr($user_agent, 0, 512),
                'path'        => substr(strtok($_SERVER['REQUEST_URI'] ?? '/', '?'), 0, 2048),
                'status_code' => http_response_code() ?: 200,
                'ip'          => $_SERVER['REMOTE_ADDR'] ?? null,
                'timestamp'   => gmdate('c'),
            )),
        )),
    ));
});

Cloudflare

1. For any site whose domain is proxied by your own Cloudflare account, whatever it is built with.
2. Create a Worker with this code and add CALMSTATS_API_KEY as a secret (Settings → Variables and Secrets).
3. Add a route for your domain, such as example.com/*, so the Worker runs in front of the site.

worker.js
const AI_CRAWLERS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|Bytespider|Amazonbot|CCBot|cohere-training-data-crawler|cohere-ai|MistralAI-User|DuckAssistBot|YouBot/i;

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    const userAgent = request.headers.get('user-agent') || '';
    if (AI_CRAWLERS.test(userAgent) && env.CALMSTATS_API_KEY) {
      ctx.waitUntil(
        fetch('https://calmstats.com/api/ai-bots', {
          method: 'POST',
          headers: {
            'content-type': 'application/json',
            authorization: `Bearer ${env.CALMSTATS_API_KEY}`,
          },
          body: JSON.stringify({
            website_id: 'YOUR-SITE-ID',
            hits: [{
              user_agent: userAgent,
              path: new URL(request.url).pathname,
              status_code: response.status,
              ip: request.headers.get('cf-connecting-ip') || undefined,
              timestamp: new Date().toISOString(),
            }],
          }),
        }).catch(() => {}),
      );
    }
    return response;
  },
};

Node.js

1. Needs Node 18 or newer (built-in fetch). Set CALMSTATS_API_KEY in the server environment.
2. Add the middleware with app.use(calmstatsAiCrawlers) before your routes.
3. Deploy.

calmstats-ai-crawlers.js
const AI_CRAWLERS = /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|anthropic-ai|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|Bytespider|Amazonbot|CCBot|cohere-training-data-crawler|cohere-ai|MistralAI-User|DuckAssistBot|YouBot/i;

// app.use(calmstatsAiCrawlers) before your routes.
function calmstatsAiCrawlers(req, res, next) {
  const userAgent = req.headers['user-agent'] || '';
  if (AI_CRAWLERS.test(userAgent) && process.env.CALMSTATS_API_KEY) {
    res.on('finish', () => {
      fetch('https://calmstats.com/api/ai-bots', {
        method: 'POST',
        headers: {
          'content-type': 'application/json',
          authorization: `Bearer ${process.env.CALMSTATS_API_KEY}`,
        },
        body: JSON.stringify({
          website_id: 'YOUR-SITE-ID',
          hits: [{
            user_agent: userAgent,
            path: req.path || req.url.split('?')[0],
            status_code: res.statusCode,
            ip: (req.headers['x-forwarded-for'] || '').split(',')[0].trim() || req.socket.remoteAddress,
            timestamp: new Date().toISOString(),
          }],
        }),
      }).catch(() => {});
    });
  }
  next();
}

module.exports = { calmstatsAiCrawlers };

Recognised crawlers

Shopify, Wix and Squarespace don't run your own server code, so they can't report crawlers unless your domain runs through your own Cloudflare account.