Akamai

Overview

This documentation describes the implementation of an Akamai EdgeWorker designed to detect traffic from AI search engines and agentic web crawlers using User-Agent analysis. Instead of modifying content inline, the EdgeWorker routes AI systems to machine-optimized endpoints, improving how your website is consumed, interpreted, and referenced across the agentic web.

This approach enables:

• Clearer machine-readable signals for AI models

• Better control over how AI agents access and process your content

• Separation of human vs. AI requester behavior in your analytics

• Improved presence across AI-driven discovery platforms (ChatGPT, Perplexity, Gemini, Claude, Grok, etc.)

Architecture

High-Level Flow

Incoming Request → User-Agent Analysis → AI Bot Detection → Routing Decision → URL Rewrite

Key Components

• AI Bot Detection Engine Identifies modern agentic web crawlers via User-Agent rules.

• Routing Logic Applies URL rewrites for AI-optimized content.

• Configuration Management Centralized bot patterns and routing rules.

• Logging & Observability Tracks how AI agents browse, fetch, and interpret your content.

AI Search Engine Bot Detection & Routing — Akamai

Introduction

This EdgeWorker implementation identifies traffic from AI search engines and agentic web crawlers such as ChatGPT, Perplexity, Gemini, Claude, Grok, DeepSeek, and more.

As AI models increasingly pull, structure, and reuse web content to answer user prompts, this routing layer allows you to:

• Serve AI-specific variants of content

• Improve clarity of machine-consumable signals

• Understand how AI systems traverse and interpret your site

• Strengthen your visibility across the agentic web

Supported AI Search Engines

Introduction

This Netlify Edge Function implementation provides intelligent detection and routing of AI search engine traffic. The system identifies requests from major AI-powered search services and routes them to specialized content endpoints optimized for AI consumption.

Supported AI Search Engines

ChatGPT / OpenAI, Gemini / Google AI, AI Overviews, Perplexity, Claude, Grok, DeepSeek, and generic AI modes.

Configuration

AI Bot Pattern Configuration

{
  "aiSearchBots": {
    "chatgpt": ["chatgpt-user", "gptbot", "openai-searchbot", "chatgpt-search", "openai-imagesbot"],
    "gemini": ["gemini-bot", "google-extended", "geminibot", "bard", "googlebot-ai"],
    "aiOverviews": ["google-inspectiontool", "google-ai-overview", "google-sge"],
    "perplexity": ["perplexitybot", "perplexity-search", "perplexityai"],
    "claude": ["claude-web", "claudebot", "claude-search", "anthropic-ai"],
    "grok": ["grok-bot", "grokbot", "xai-bot", "x-ai-bot"],
    "deepseek": ["deepseekbot", "deepseek-search", "deepseek-ai"],
    "aiMode": ["ai-mode-bot", "aimode", "intelligent-agent"]
  }
}

Routing Rules Configuration

{
  "routingRules": {
    "chatgpt": { "method": "path_prefix", "target": "/chatgpt", "preserveQuery": true },
    "gemini": { "method": "path_prefix", "target": "/gemini", "preserveQuery": true },
    "aiOverviews": { "method": "path_prefix", "target": "/ai-overview", "preserveQuery": true },
    "perplexity": { "method": "path_prefix", "target": "/perplexity", "preserveQuery": true },
    "claude": { "method": "path_prefix", "target": "/claude", "preserveQuery": true },
    "grok": { "method": "path_prefix", "target": "/grok", "preserveQuery": true },
    "deepseek": { "method": "path_prefix", "target": "/deepseek", "preserveQuery": true },
    "aiMode": { "method": "path_prefix", "target": "/ai-mode", "preserveQuery": true }
  }
}

Implementation

Core Detection Function

function detectAISearchBot(userAgent) {
    const ua = userAgent.toLowerCase();

    if (ua.includes('chatgpt') || ua.includes('gptbot') || ua.includes('openai'))
        return 'chatgpt';

    if (ua.includes('gemini') || ua.includes('bard') || ua.includes('google-extended'))
        return 'gemini';

    if (ua.includes('google-sge') || ua.includes('ai-overview'))
        return 'aiOverviews';

    if (ua.includes('perplexity'))
        return 'perplexity';

    if (ua.includes('claude') || ua.includes('anthropic'))
        return 'claude';

    if (ua.includes('grok') || ua.includes('xai-bot'))
        return 'grok';

    if (ua.includes('deepseek'))
        return 'deepseek';

    if (ua.includes('ai-mode') || ua.includes('intelligent-agent'))
        return 'aiMode';

    return null;
}

URL Routing Function

function generateAIBotUrl(originalUrl, aiBot, config) {
    const url = new URL(originalUrl);
    const rule = config.routingRules[aiBot];

    if (!rule) return originalUrl;

    switch (rule.method) {
        case 'path_prefix':
            url.pathname = rule.target + url.pathname;
            break;
        case 'subdomain':
            url.hostname = rule.target;
            break;
        case 'parameter':
            url.searchParams.set(rule.param || 'ai', rule.value || aiBot);
            break;
    }

    if (!rule.preserveQuery) {
        url.search = '';
    }

    return url.toString();
}

Complete EdgeWorker Implementation

import { aiSearchBots, routingRules } from './config.js';

export async function onClientRequest(request) {
    const userAgent = request.getHeader('User-Agent') || '';
    const originalUrl = request.url;

    // Detect AI search engine or agentic web crawler
    const aiBot = detectAISearchBot(userAgent);

    if (!aiBot) {
        // Human traffic — no special handling needed
        return;
    }

    // Generate AI-optimized URL
    const targetUrl = generateAIBotUrl(originalUrl, aiBot, { routingRules });

    // Add request metadata for analytics
    request.setHeader('X-AI-Bot', aiBot);
    request.setHeader('X-Original-URL', originalUrl);
    request.setHeader('X-AI-Optimized', 'true');

    // Route the AI agent to specialized content
    request.url = targetUrl;

    // Observability
    console.log(`Agentic Web Routing: ${aiBot} -> ${targetUrl}`);
}

Routing Methods (Agentic Web–Ready)

Path Prefix Routing

Best for agentic clarity:

• /article/ai-trends → /chatgpt/article/ai-trends

Subdomain Routing

Useful for large multi-property ecosystems:

• www.example.com/page → perplexity.example.com/page

Parameter Routing

Lightweight AI classification:

• /documentation → /documentation?ai=claude

Environment Configuration

Production

Focus on stability, clarity for AI models, and observability.

Staging

Used for expanding bot patterns, experimenting with new AI indexing behaviors, etc.

(all JSON examples updated to reflect agentic terminology; no SEO terms.)

Monitoring & Analytics

AI-Specific Headers

• X-AI-Bot - Identifies which AI system accessed the content

• X-AI-Optimized - Indicates that agentic routing occurred

• X-Original-URL - For reconstructing navigation flow

Example Log

{
  "timestamp": "2025-01-01T12:00:00Z",
  "aiService": "chatgpt",
  "originalUrl": "/article/example",
  "routedUrl": "/chatgpt/article/example",
  "userAgent": "ChatGPT-User/1.0",
  "routingMethod": "path_prefix"
}

Deployment

• Upload EdgeWorker bundle

• Apply routing rule via Property Manager

• Enable monitoring

• Deploy to staging, validate AI routing

• Roll out progressively to production

Performance & Security (Agentic Web Context)

• AI-specific rate limits

• Pattern validation

• Known bot IP verification

• Agent behavior anomaly detection

That’s it! You have now successfully configured Limy router for your website. Data should begin to populate on your dashboard within an hour.