CloudFront Logs Analysis with Athena and self-hosted Grafana

Grafana Dashboard

Introduction

This post presents an alternative solution for CloudFront log analysis, building on the static website architecture described in my previous post related to QuickSight. Instead of using Amazon QuickSight for visualisation, this post is about using a self-hosted Grafana instance connected to Athena. This approach targets users needing more control, flexibility, and open-source capabilities.

Cost Estimation

While using Athena and Grafana provides powerful insights, costs can accumulate based on data volume and infrastructure usage:

• Amazon S3 Storage: Costs depend on the size of stored logs (Standard tier: ~$0.023 per GB per month).
• AWS Athena: Charges per query ($5 per TB scanned).
• Amazon EC2 (for Grafana server): ~$15 per month (t3.small instance).
• Application Load Balancer: ~$20–30 per month, depending on traffic and usage.
• Route 53 (optional, for custom domain): ~$1–2 per month.

Project Structure

This solution enhances the static website infrastructure introduced in my earlier post by introducing a Grafana-based monitoring system for CloudFront logs. Grafana runs on an EC2 instance behind an Application Load Balancer, which is securely accessible over HTTPS.

Core Components

• Amazon S3 Bucket: Stores raw CloudFront logs, Athena query results, and Grafana provisioning files.
• Athena Table and Athena View: Provides a structured, queryable interface for the raw log files.
• Grafana Server: Self-hosted on an EC2 instance, automatically configured via user data.
• Application Load Balancer (ALB): Exposes Grafana securely via HTTPS, optionally tied to a custom domain with Route 53.
• Networking Stack — optionally (VPC, Subnet, Internet Gateway, Route Table, Security Groups): Provides the necessary networking infrastructure for the Grafana server and Load Balancer, ensuring secure and scalable connectivity.

Infrastructure schema

Configuration files:

1. Athena Data Source Configuration:

apiVersion: 1
datasources:
  - name: Athena
    type: grafana-athena-datasource
    uid: athena
    access: proxy
    isDefault: true
    jsonData:
      defaultRegion: ${REGION}
      catalog: AwsDataCatalog
      database: ${DATABASE}
      workgroup: primary
      outputLocation: s3://${BUCKET}/athena-query-results

Dashboard Provider Configuration:

apiVersion: 1
providers:
  - name: 'CloudFront Logs'
    orgId: 1
    folder: ''
    type: file
    disableDeletion: false
    editable: true
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

Grafana Dashboard Definition:

{
    "annotations": {
      "list": [
        {
          "builtIn": 1,
          "datasource": {
            "type": "grafana",
            "uid": "-- Grafana --"
          },
          "enable": true,
          "hide": true,
          "iconColor": "rgba(0, 211, 255, 1)",
          "name": "Annotations & Alerts",
          "type": "dashboard"
        }
      ]
    },
    "editable": true,
    "fiscalYearStartMonth": 0,
    "graphTooltip": 0,
    "id": 4,
    "links": [],
    "panels": [
      {
        "datasource": {
          "type": "grafana-athena-datasource",
          "uid": "athena"
        },
        "fieldConfig": {
          "defaults": {
            "color": {
              "mode": "palette-classic"
            },
            "custom": {
              "axisBorderShow": false,
              "axisCenteredZero": false,
              "axisColorMode": "text",
              "axisLabel": "",
              "axisPlacement": "auto",
              "barAlignment": 0,
              "barWidthFactor": 0.6,
              "drawStyle": "line",
              "fillOpacity": 0,
              "gradientMode": "none",
              "hideFrom": {
                "legend": false,
                "tooltip": false,
                "viz": false
              },
              "insertNulls": false,
              "lineInterpolation": "linear",
              "lineWidth": 1,
              "pointSize": 5,
              "scaleDistribution": {
                "type": "linear"
              },
              "showPoints": "auto",
              "spanNulls": false,
              "stacking": {
                "group": "A",
                "mode": "none"
              },
              "thresholdsStyle": {
                "mode": "off"
              }
            },
            "mappings": [],
            "thresholds": {
              "mode": "absolute",
              "steps": [
                {
                  "color": "green"
                },
                {
                  "color": "red",
                  "value": 80
                }
              ]
            }
          },
          "overrides": []
        },
        "gridPos": {
          "h": 8,
          "w": 12,
          "x": 0,
          "y": 0
        },
        "id": 1,
        "options": {
          "legend": {
            "calcs": [],
            "displayMode": "list",
            "placement": "bottom",
            "showLegend": true
          },
          "tooltip": {
            "hideZeros": false,
            "mode": "single",
            "sort": "none"
          }
        },
        "pluginVersion": "11.6.0",
        "targets": [
          {
            "connectionArgs": {
              "catalog": "__default",
              "database": "__default",
              "region": "__default",
              "resultReuseEnabled": false,
              "resultReuseMaxAgeInMinutes": 60
            },
            "datasource": {
              "type": "grafana-athena-datasource",
              "uid": "athena"
            },
            "format": 1,
            "rawSQL": "SELECT * FROM cloudfront_aggregated_logs;",
            "refId": "A",
            "table": "cloudfront_aggregated_logs"
          }
        ],
        "title": "CloudFront Logs Dashboard",
        "type": "timeseries"
      }
    ],
    "preload": false,
    "schemaVersion": 41,
    "tags": [],
    "templating": {
      "list": []
    },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
    "timepicker": {},
    "timezone": "browser",
    "title": "Requests Overview",
    "uid": "grafana-athena-datasource",
    "version": 1
  }

Prerequisites

Ensure the following prerequisites are in place:

• An AWS account with sufficient permissions to create and manage resources.
  - The AWS CLI installed on your local machine.
  - Infrastructure deployed with a CloudFormation templates from my previous posts or from my GitLab repo (if applicable).

Deployment

1. Fill in all necessary parameters in the CloudFormation template and launch the CloudFormation stack.

aws cloudformation create-stack \
    --stack-name grafana-server \
    --template-body file://infrastructure/grafana_dashboard.yaml \
    --capabilities CAPABILITY_NAMED_IAM

2. Access Grafana server.

Navigate to your custom domain (configured in Route 53) in your browser. Login with default Grafana credentials (unless customized during the boot process). Preconfigured dashboards and the Athena data source will be automatically available.

3. Cleanup Resources.

After testing, stop CloudFront logging with simple script, delete all data from the S3 bucket, delete the CloudFormation stacks.

./infrastructure/disable_cloudfront_distribution_logging.sh <cloudfront-id>

aws s3 rm s3://$BUCKET_NAME --recursive

aws cloudformation delete-stack --stack-name grafana-server

Conclusion

Switching from Amazon QuickSight to a self-hosted Grafana instance offers greater flexibility, extensibility, and full control over CloudFront log visualisation. However, this benefit comes with a slight increase in operational complexity and infrastructure cost.

QuickSight remains a better option for users needing a fast, managed solution, while Grafana is ideal for teams requiring deep customisation and broader observability integrations.

If you found this post helpful and interesting, please click the clap button below to show your support. Feel free to use and share this post. 🙂