# Fastly

### Overview

This documentation describes a Fastly Compute\@Edge implementation that detects traffic from AI search engines and agentic web crawlers using User-Agent analysis. The function routes AI systems to machine-optimized endpoints, improving how your content is consumed, interpreted, and referenced across the agentic web.

### Architecture

**Incoming Request → User-Agent Analysis → AI Bot Detection → Routing Decision → URL Rewrite**

#### Key Components

* AI Bot Detection Engine
* Routing Logic
* Configuration Management
* Logging & Observability

## AI Search Engine Bot Detection  

### Introduction

This implementation identifies traffic from AI search engines and agentic web crawlers such as ChatGPT, Perplexity, Gemini, Claude, Grok, and DeepSeek. The system routes these agents to specialized endpoints optimized for AI consumption, improving machine readability, model context accuracy, and your overall presence across the agentic web.

## Configuration

#### AI Bot Pattern Configuration (config.json)

```json
{
  "aiSearchBots": {
    "chatgpt": [
      "chatgpt-user",
      "gptbot",
      "openai-searchbot",
      "chatgpt-search",
      "openai-imagesbot"
    ],
    "gemini": [
      "gemini-bot",
      "google-extended",
      "geminibot",
      "bard",
      "googlebot-ai"
    ],
    "aiOverviews": [
      "google-inspectiontool",
      "google-ai-overview",
      "google-sge"
    ],
    "perplexity": [
      "perplexitybot",
      "perplexity-search",
      "perplexityai"
    ],
    "claude": [
      "claude-web",
      "claudebot",
      "claude-search",
      "anthropic-ai"
    ],
    "grok": [
      "grok-bot",
      "grokbot",
      "xai-bot",
      "x-ai-bot"
    ],
    "deepseek": [
      "deepseekbot",
      "deepseek-search",
      "deepseek-ai"
    ],
    "aiMode": [
      "ai-mode-bot",
      "aimode",
      "intelligent-agent"
    ]
  }
}
```

#### Routing Rules Configuration (routing.json)

```json
{
  "routingRules": {
    "chatgpt": {
      "method": "path_prefix",
      "target": "/chatgpt",
      "preserveQuery": true
    },
    "gemini": {
      "method": "path_prefix",
      "target": "/gemini",
      "preserveQuery": true
    },
    "aiOverviews": {
      "method": "path_prefix",
      "target": "/ai-overview",
      "preserveQuery": true
    },
    "perplexity": {
      "method": "path_prefix",
      "target": "/perplexity",
      "preserveQuery": true
    },
    "claude": {
      "method": "path_prefix",
      "target": "/claude",
      "preserveQuery": true
    },
    "grok": {
      "method": "path_prefix",
      "target": "/grok",
      "preserveQuery": true
    },
    "deepseek": {
      "method": "path_prefix",
      "target": "/deepseek",
      "preserveQuery": true
    },
    "aiMode": {
      "method": "path_prefix",
      "target": "/ai-mode",
      "preserveQuery": true
    }
  }
}
```

## Implementation

#### aiDetection.ts

```ts
export type AIBotType =
  | "chatgpt"
  | "gemini"
  | "aiOverviews"
  | "perplexity"
  | "claude"
  | "grok"
  | "deepseek"
  | "aiMode"
  | null;

export function detectAISearchBot(userAgent: string): AIBotType {
  const ua = userAgent.toLowerCase();

  if (ua.includes("chatgpt") || ua.includes("gptbot") || ua.includes("openai"))
    return "chatgpt";

  if (ua.includes("gemini") || ua.includes("bard") || ua.includes("google-extended"))
    return "gemini";

  if (ua.includes("google-sge") || ua.includes("ai-overview"))
    return "aiOverviews";

  if (ua.includes("perplexity"))
    return "perplexity";

  if (ua.includes("claude") || ua.includes("anthropic"))
    return "claude";

  if (ua.includes("grok") || ua.includes("xai-bot"))
    return "grok";

  if (ua.includes("deepseek"))
    return "deepseek";

  if (ua.includes("ai-mode") || ua.includes("intelligent-agent"))
    return "aiMode";

  return null;
}
```

#### routing.ts

```ts
import type { AIBotType } from "./aiDetection";

interface RoutingRule {
  method: "path_prefix" | "subdomain" | "parameter";
  target: string;
  preserveQuery: boolean;
  param?: string;
  value?: string;
}

interface RoutingConfig {
  routingRules: Record<string, RoutingRule>;
}

export function generateAIBotUrl(
  originalUrl: string,
  aiBot: AIBotType,
  config: RoutingConfig
): string {
  if (!aiBot) return originalUrl;

  const url = new URL(originalUrl);
  const rule = config.routingRules[aiBot];

  if (!rule) return originalUrl;

  switch (rule.method) {
    case "path_prefix":
      url.pathname = rule.target + url.pathname;
      break;
    case "subdomain":
      url.hostname = rule.target;
      break;
    case "parameter":
      url.searchParams.set(rule.param || "ai", rule.value || aiBot);
      break;
  }

  if (!rule.preserveQuery) {
    url.search = "";
  }

  return url.toString();
}
```

#### main.ts (Fastly Compute\@Edge)

```ts
import { detectAISearchBot } from "./aiDetection";
import { generateAIBotUrl } from "./routing";
import routingRules from "./routing.json";

addEventListener("fetch", (event) => event.respondWith(handleRequest(event)));

async function handleRequest(event: FetchEvent): Promise<Response> {
  const request = event.request;
  const userAgent = request.headers.get("user-agent") || "";
  const originalUrl = request.url;

  const aiBot = detectAISearchBot(userAgent);
  if (!aiBot) {
    return fetch(request);
  }

  const targetUrl = generateAIBotUrl(originalUrl, aiBot, { routingRules });

  const newRequest = new Request(targetUrl, request);
  newRequest.headers.set("X-AI-Bot", aiBot);
  newRequest.headers.set("X-Original-URL", originalUrl);
  newRequest.headers.set("X-AI-Optimized", "true");

  return fetch(newRequest);
}
```

## Routing Methods

#### Path Prefix Routing

`/docs/intro` → `/chatgpt/docs/intro`

#### Subdomain Routing

`example.com/page` → `claude.example.com/page`

#### Parameter Routing

`/guide` → `/guide?ai=perplexity`

## Monitoring & Analytics

### Headers

* `X-AI-Bot`
* `X-Original-URL`
* `X-AI-Optimized`

### Log Format

```json
{
  "timestamp": "2025-01-01T12:00:00Z",
  "aiService": "chatgpt",
  "originalUrl": "/article/example",
  "routedUrl": "/chatgpt/article/example",
  "userAgent": "ChatGPT-User/1.0",
  "routingMethod": "path_prefix"
}
```

## Deployment

1. Add files to your Compute\@Edge project
2. Build the package
3. Upload & activate your service
4. Verify routing in staging
5. Deploy to production

That’s it! You have now successfully configured Limy router for your website. Data should begin to populate on your dashboard within an hour.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.getlimy.ai/maze-redirect/a2a/fastly.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.