# Nginx

This guide explains how to configure your existing Nginx server to stream access logs in JSON format to Limy's log ingestion endpoint.

### Overview

You will:

1. Update your Nginx configuration to output access logs in JSON format
2. Install and configure Fluent Bit to stream logs to the Limy endpoint

Choose the setup method that matches your environment:

* Bare Metal / VM Setup
* Kubernetes Setup

***

## Bare Metal / VM Setup

### Step 1: Configure Nginx JSON Access Logs

Add the following `log_format` directive to your Nginx configuration's `http` block (typically in `/etc/nginx/nginx.conf`):

```nginx
http {
    # JSON log format with all available fields
    log_format json_combined escape=json '{'
        '"log_type":"nginx_access",'
        '"timestamp":"$time_iso8601",'
        '"time_local":"$time_local",'
        '"msec":"$msec",'

        '"client_ip":"$remote_addr",'
        '"client_port":"$remote_port",'
        '"remote_user":"$remote_user",'

        '"request":"$request",'
        '"request_method":"$request_method",'
        '"request_uri":"$request_uri",'
        '"uri":"$uri",'
        '"args":"$args",'
        '"scheme":"$scheme",'
        '"server_protocol":"$server_protocol",'
        '"request_length":$request_length,'
        '"request_time":$request_time,'

        '"status":$status,'
        '"body_bytes_sent":$body_bytes_sent,'
        '"bytes_sent":$bytes_sent,'

        '"host":"$host",'
        '"server_addr":"$server_addr",'
        '"server_port":"$server_port",'
        '"server_name":"$server_name",'
        '"hostname":"$hostname",'
        '"nginx_version":"$nginx_version",'
        '"pid":"$pid",'

        '"connection":"$connection",'
        '"connection_requests":"$connection_requests",'
        '"pipe":"$pipe",'

        '"http_host":"$http_host",'
        '"http_user_agent":"$http_user_agent",'
        '"http_referer":"$http_referer",'
        '"http_accept":"$http_accept",'
        '"http_accept_encoding":"$http_accept_encoding",'
        '"http_accept_language":"$http_accept_language",'
        '"http_content_type":"$http_content_type",'
        '"http_content_length":"$http_content_length",'
        '"http_x_forwarded_for":"$http_x_forwarded_for",'
        '"http_x_forwarded_proto":"$http_x_forwarded_proto",'
        '"http_x_real_ip":"$http_x_real_ip",'
        '"http_x_request_id":"$http_x_request_id",'

        '"sent_http_content_type":"$sent_http_content_type",'
        '"sent_http_content_length":"$sent_http_content_length",'

        '"upstream_addr":"$upstream_addr",'
        '"upstream_status":"$upstream_status",'
        '"upstream_response_time":"$upstream_response_time",'
        '"upstream_response_length":"$upstream_response_length",'
        '"upstream_connect_time":"$upstream_connect_time",'
        '"upstream_header_time":"$upstream_header_time",'
        '"upstream_cache_status":"$upstream_cache_status",'

        '"ssl_protocol":"$ssl_protocol",'
        '"ssl_cipher":"$ssl_cipher",'
        '"ssl_session_reused":"$ssl_session_reused",'
        '"ssl_server_name":"$ssl_server_name",'

        '"gzip_ratio":"$gzip_ratio"'
    '}';

    # Use the JSON format for access logs
    access_log /var/log/nginx/access.log json_combined;

    # ... rest of your existing configuration
}
```

#### Apply the Nginx Configuration

```bash
# Test the configuration
sudo nginx -t

# Reload Nginx to apply changes
sudo systemctl reload nginx
```

***

### Step 2: Install Fluent Bit

#### Option A: Debian/Ubuntu

```bash
# Add the Fluent Bit repository
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# Start and enable the service
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
```

#### Option B: RHEL/CentOS/Amazon Linux

```bash
# Add the Fluent Bit repository
curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

# Start and enable the service
sudo systemctl start fluent-bit
sudo systemctl enable fluent-bit
```

#### Option C: Docker

```bash
docker run -d \
  --name fluent-bit \
  -v /var/log/nginx:/var/log/nginx:ro \
  -v /path/to/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro \
  -v /path/to/parsers.conf:/fluent-bit/etc/parsers.conf:ro \
  fluent/fluent-bit:latest
```

***

### Step 3: Configure Fluent Bit

Create the Fluent Bit configuration file at `/etc/fluent-bit/fluent-bit.conf`:

```ini
[SERVICE]
    Flush         5
    Log_Level     info
    Daemon        off
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   0.0.0.0
    HTTP_Port     2020

[INPUT]
    Name              tail
    Path              /var/log/nginx/access.log
    Parser            nginx_json
    Tag               nginx.access
    Refresh_Interval  5
    Mem_Buf_Limit     10MB
    Skip_Long_Lines   On
    DB                /var/lib/fluent-bit/nginx.db

[OUTPUT]
    Name              http
    Match             nginx.*
    Host              stream.getlimy.ai
    Port              443
    URI               /
    Format            json
    TLS               On
    Header            X-API-Key YOUR_API_KEY_HERE
```

Create the parsers file at `/etc/fluent-bit/parsers.conf`:

```ini
[PARSER]
    Name        nginx_json
    Format      json
    Time_Key    timestamp
    Time_Format %Y-%m-%dT%H:%M:%S%z
    Time_Keep   On
```

#### Replace the API Key

Replace `YOUR_API_KEY_HERE` in the configuration with the API key provided to you by Limy.

***

### Step 4: Start Fluent Bit

```bash
# Restart Fluent Bit to apply the new configuration
sudo systemctl restart fluent-bit

# Verify it's running
sudo systemctl status fluent-bit

# Check logs for any errors
sudo journalctl -u fluent-bit -f
```

***

## Kubernetes Setup

### Step 1: Update Nginx ConfigMap

If you're using Nginx in Kubernetes, update your Nginx ConfigMap to include the JSON log format. The key change is logging to `/dev/stdout` so Kubernetes can capture the logs.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: <your-namespace>
data:
  nginx.conf: |
    user nginx;
    worker_processes auto;
    error_log /dev/stderr warn;
    pid /var/run/nginx.pid;

    events {
        worker_connections 1024;
    }

    http {
        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        # JSON log format with all available fields
        log_format json_combined escape=json '{'
            '"log_type":"nginx_access",'
            '"timestamp":"$time_iso8601",'
            '"time_local":"$time_local",'
            '"msec":"$msec",'

            '"client_ip":"$remote_addr",'
            '"client_port":"$remote_port",'
            '"remote_user":"$remote_user",'

            '"request":"$request",'
            '"request_method":"$request_method",'
            '"request_uri":"$request_uri",'
            '"uri":"$uri",'
            '"args":"$args",'
            '"scheme":"$scheme",'
            '"server_protocol":"$server_protocol",'
            '"request_length":$request_length,'
            '"request_time":$request_time,'

            '"status":$status,'
            '"body_bytes_sent":$body_bytes_sent,'
            '"bytes_sent":$bytes_sent,'

            '"host":"$host",'
            '"server_addr":"$server_addr",'
            '"server_port":"$server_port",'
            '"server_name":"$server_name",'
            '"hostname":"$hostname",'
            '"nginx_version":"$nginx_version",'
            '"pid":"$pid",'

            '"connection":"$connection",'
            '"connection_requests":"$connection_requests",'
            '"pipe":"$pipe",'

            '"http_host":"$http_host",'
            '"http_user_agent":"$http_user_agent",'
            '"http_referer":"$http_referer",'
            '"http_accept":"$http_accept",'
            '"http_accept_encoding":"$http_accept_encoding",'
            '"http_accept_language":"$http_accept_language",'
            '"http_content_type":"$http_content_type",'
            '"http_content_length":"$http_content_length",'
            '"http_x_forwarded_for":"$http_x_forwarded_for",'
            '"http_x_forwarded_proto":"$http_x_forwarded_proto",'
            '"http_x_real_ip":"$http_x_real_ip",'
            '"http_x_request_id":"$http_x_request_id",'

            '"sent_http_content_type":"$sent_http_content_type",'
            '"sent_http_content_length":"$sent_http_content_length",'

            '"upstream_addr":"$upstream_addr",'
            '"upstream_status":"$upstream_status",'
            '"upstream_response_time":"$upstream_response_time",'
            '"upstream_response_length":"$upstream_response_length",'
            '"upstream_connect_time":"$upstream_connect_time",'
            '"upstream_header_time":"$upstream_header_time",'
            '"upstream_cache_status":"$upstream_cache_status",'

            '"ssl_protocol":"$ssl_protocol",'
            '"ssl_cipher":"$ssl_cipher",'
            '"ssl_session_reused":"$ssl_session_reused",'
            '"ssl_server_name":"$ssl_server_name",'

            '"gzip_ratio":"$gzip_ratio"'
        '}';

        # Log to stdout for Kubernetes log collection
        access_log /dev/stdout json_combined;

        sendfile on;
        keepalive_timeout 65;

        # Include your server blocks or other config files
        include /etc/nginx/conf.d/*.conf;
    }
```

Mount this ConfigMap in your Nginx deployment:

```yaml
spec:
  containers:
    - name: nginx
      image: nginx:latest
      volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
  volumes:
    - name: nginx-config
      configMap:
        name: nginx-config
```

***

### Step 2: Create Fluent Bit Secret for API Key

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: limy-api-key
  namespace: <your-namespace>
type: Opaque
stringData:
  api-key: "YOUR_API_KEY_HERE"
```

Apply the secret:

```bash
kubectl apply -f limy-secret.yaml
```

***

### Step 3: Deploy Fluent Bit ConfigMap

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: <your-namespace>
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Path              /var/log/containers/*<your-nginx-app-label>*.log
        Parser            docker
        Tag               nginx.access
        Refresh_Interval  5
        Mem_Buf_Limit     10MB
        Skip_Long_Lines   On
        DB                /tmp/flb_nginx.db

    # Filter to only include nginx access logs (not startup messages)
    [FILTER]
        Name          grep
        Match         nginx.*
        Regex         log log_type

    # Parse the JSON from the log field
    [FILTER]
        Name          parser
        Match         nginx.*
        Key_Name      log
        Parser        nginx_json
        Reserve_Data  Off

    [OUTPUT]
        Name              http
        Match             nginx.*
        Host              stream.getlimy.ai
        Port              443
        URI               /
        Format            json
        TLS               On
        Header            X-API-Key ${LIMY_API_KEY}

  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        nginx_json
        Format      json
        Time_Key    timestamp
        Time_Format %Y-%m-%dT%H:%M:%S%z
        Time_Keep   On
```

> **Note:** Replace `<your-nginx-app-label>` in the `Path` with your Nginx pod name or label pattern (e.g., `*nginx*` or `*my-app-nginx*`).

***

### Step 4: Deploy Fluent Bit DaemonSet

```yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: <your-namespace>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
  - kind: ServiceAccount
    name: fluent-bit
    namespace: <your-namespace>
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: <your-namespace>
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
        - key: node-role.kubernetes.io/control-plane
          effect: NoSchedule
      containers:
        - name: fluent-bit
          image: fluent/fluent-bit:latest
          ports:
            - containerPort: 2020
          env:
            - name: LIMY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: limy-api-key
                  key: api-key
          volumeMounts:
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc/
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config
```

***

### Step 5: Apply Kubernetes Resources

```bash
# Apply all resources
kubectl apply -f nginx-configmap.yaml
kubectl apply -f limy-secret.yaml
kubectl apply -f fluent-bit-configmap.yaml
kubectl apply -f fluent-bit-daemonset.yaml

# Restart Nginx pods to pick up the new config
kubectl rollout restart deployment/<your-nginx-deployment> -n <your-namespace>

# Verify Fluent Bit is running
kubectl get pods -l app=fluent-bit -n <your-namespace>

# Check Fluent Bit logs
kubectl logs -l app=fluent-bit -n <your-namespace> -f
```

***

### Alternative: Using Helm

If you prefer Helm, you can install Fluent Bit using the official chart:

```bash
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

helm install fluent-bit fluent/fluent-bit \
  --namespace <your-namespace> \
  --set config.inputs="[INPUT]\n    Name tail\n    Path /var/log/containers/*nginx*.log\n    Parser docker\n    Tag nginx.access" \
  --set config.outputs="[OUTPUT]\n    Name http\n    Match nginx.*\n    Host stream.getlimy.ai\n    Port 443\n    URI /\n    Format json\n    TLS On\n    Header X-API-Key YOUR_API_KEY_HERE"
```

For more complex configurations, create a `values.yaml` file and use:

```bash
helm install fluent-bit fluent/fluent-bit -f values.yaml -n <your-namespace>
```

***

## Verification

### 1. Verify Nginx is Logging in JSON Format

**Bare Metal / VM:**

```bash
tail -5 /var/log/nginx/access.log
```

**Kubernetes:**

```bash
kubectl logs <nginx-pod-name> -n <your-namespace> | head -5
```

You should see JSON-formatted log entries like:

```json
{"log_type":"nginx_access","timestamp":"2025-12-16T15:30:45+00:00","client_ip":"192.168.1.100","request_method":"GET","request_uri":"/api/health","status":200,...}
```

### 2. Verify Fluent Bit is Running

**Bare Metal / VM:**

```bash
curl http://localhost:2020/api/v1/metrics
```

**Kubernetes:**

```bash
kubectl port-forward <fluent-bit-pod> 2020:2020 -n <your-namespace>
# In another terminal:
curl http://localhost:2020/api/v1/metrics
```

### 3. Generate Test Traffic

```bash
# Make a test request to your Nginx server
curl -I http://<your-nginx-host>/
```

***

## Troubleshooting

### Nginx Configuration Errors

```bash
# Test configuration syntax
sudo nginx -t  # Bare metal
kubectl exec <nginx-pod> -- nginx -t  # Kubernetes

# Check error log
sudo tail -f /var/log/nginx/error.log  # Bare metal
kubectl logs <nginx-pod> -n <your-namespace>  # Kubernetes
```

### Fluent Bit Issues

**Bare Metal / VM:**

```bash
sudo journalctl -u fluent-bit -f
fluent-bit -c /etc/fluent-bit/fluent-bit.conf --dry-run
```

**Kubernetes:**

```bash
kubectl logs -l app=fluent-bit -n <your-namespace> -f
kubectl describe pod -l app=fluent-bit -n <your-namespace>
```

### Common Issues

| Issue                    | Solution                                                                                       |
| ------------------------ | ---------------------------------------------------------------------------------------------- |
| Logs not appearing       | Ensure `access_log` path matches Fluent Bit `Path`. In K8s, ensure Nginx logs to `/dev/stdout` |
| Permission denied        | Run Fluent Bit with appropriate permissions. In K8s, check RBAC and volume mounts              |
| Connection refused       | Verify network connectivity to `stream.getlimy.ai`. Check network policies in K8s              |
| 401 Unauthorized         | Check that `X-API-Key` header value is correct                                                 |
| No logs in K8s           | Verify the log path pattern matches your pod names. Check `kubectl logs` output directly       |
| Startup messages in logs | Ensure the `grep` filter is configured to filter by `log_type`                                 |

***

## Log Fields Reference

| Field                  | Description                         |
| ---------------------- | ----------------------------------- |
| `timestamp`            | ISO 8601 timestamp                  |
| `client_ip`            | Client IP address                   |
| `client_port`          | Client port                         |
| `remote_user`          | Authenticated username (Basic Auth) |
| `request`              | Full request line                   |
| `request_method`       | HTTP method (GET, POST, etc.)       |
| `request_uri`          | Request URI with query string       |
| `uri`                  | Request URI without query string    |
| `args`                 | Query string parameters             |
| `scheme`               | http or https                       |
| `server_protocol`      | HTTP version                        |
| `request_length`       | Request size in bytes               |
| `request_time`         | Request processing time (seconds)   |
| `status`               | HTTP response status code           |
| `body_bytes_sent`      | Response body size                  |
| `bytes_sent`           | Total response size                 |
| `host`                 | Request host                        |
| `server_addr`          | Server IP address                   |
| `server_port`          | Server port                         |
| `server_name`          | Server name from config             |
| `hostname`             | Machine hostname                    |
| `http_user_agent`      | User-Agent header                   |
| `http_referer`         | Referer header                      |
| `http_x_forwarded_for` | X-Forwarded-For header              |
| `upstream_*`           | Upstream/proxy metrics              |
| `ssl_*`                | TLS/SSL connection details          |
| `gzip_ratio`           | Compression ratio                   |

***

## Support

If you encounter any issues, please contact Limy support with:

* Fluent Bit version (`fluent-bit --version`)
* Nginx version (`nginx -v`)
* Your deployment type (Bare Metal, Docker, or Kubernetes)
* Relevant error logs
* Your configuration files (with API key redacted)