Cloudflare

Overview

This Cloudflare Worker automatically detects and routes AI search engine bot traffic to different URLs based on User-Agent analysis. It differentiates between AI-powered search engines and human users, allowing you to serve optimized content to AI crawlers while providing the full experience to human visitors.

Integrations;

• AI Bot Detection: Identifies AI search engine bots using User-Agent pattern matching

• Traffic Segmentation: Separates AI search engine traffic from human users

• Flexible Routing: Supports subdomain, path-based, and domain-based routing

• Performance Optimized: Lightweight detection with minimal latency impact

Supported AI Search Engine Bots

Installation

Prerequisites

• Cloudflare account

• Wrangler CLI installed (npm install -g wrangler)

Deployment Steps

1. Initialize Worker:

   wrangler init ai-bot-router
   cd ai-bot-router

2. Replace Code: Copy the provided code into src/index.js

3. Deploy:

   wrangler deploy

4. Configure Routes: In Cloudflare Dashboard → Workers → Routes, add:

   yourdomain.com/*

Usage Examples

Example 1: Subdomain Routing

AI search engines are routed to a dedicated subdomain:

Result: yourdomain.com/content → ai.yourdomain.com/content

Example 2: Path-Based Routing

AI bots get a specific path prefix:

Result: yourdomain.com/article → yourdomain.com/ai/article

Example 3: Separate Domain

AI bots are routed to a completely different domain:

Result: AI bot sees yourdomain.com but gets content from ai-content.yourdomain.com

Advanced Customization

Adding New AI Bot Patterns

Extend the botPatterns array in detectBot():

const botPatterns = [
  // ... existing AI bot patterns
  /newaibot/i,
  /custom-ai-crawler/i,
  /your-ai-tool/i
];

Custom AI Bot Handling

Modify the routing logic to handle specific AI bots differently:

async function handleBotTraffic(request, url, userAgent) {
  if (/GPTBot/i.test(userAgent)) {
    // Special handling for ChatGPT
    return handleChatGPTBot(request, url);
  }
  
  // Default AI bot handling
  return handleGenericAIBot(request, url);
}

Support

For issues and questions:

• Check Cloudflare Worker logs in the dashboard

• Monitor response codes and AI bot behavior

• Test with various AI search engine User-Agents

• Review detection patterns in logs for accuracy

Version History

• v1.0: Initial release with AI search engine bot detection and routing

• Current implementation focuses specifically on AI-powered search engines and crawlers

Select HTTP requests dataset, and choose the following fields to send:

• Request

  • ClientIP - IP Address of the client.

  • ClientRequestHost - Host requested by the client.

  • ClientRequestMethod - HTTP method of client request.

  • ClientRequestReferer - HTTP request referrer.

  • ClientRequestURI - URI requested by the client.

  • ClientRequestUSerAgent - User agent reported by the client

• Performance

  • EdgeStartTimestamp - Timestamp at which the edge received request from the client.

  • EdgeEndTimestamp - Timestamp at which the edge finished sending response to the client.

• Response

  • EdgeResponseBytes - Number of bytes returned by the edge to the client.

  • EdgeReponseStatus - HTTP status code returned by Cloudflare to the client.

That’s it! You have now successfully configured Limy router for your website. Data should begin to populate on your dashboard within an hour.