TDCTF Academy Logo TDCTF ACADEMY
@farishhz

9.5.2 Cloud Monitoring

Cloud Monitoring adalah praktik observasi dan pengukuran health, performa, dan keamanan infrastruktur cloud secara real-time. Berbeda dengan logging yang bersifat event-driven (merekam kejadian), monitoring bersifat metrics-driven - mengumpulkan data time-series tentang CPU, memory, network, latency, error rate, dan custom aplikasi. Monitoring yang efektif memungkinkan deteksi dini anomali, otomatisasi incident response, dan kepatuhan terhadap compliance.

Arsitektur Cloud Monitoring

Monitoring cloud mengikuti pipeline metrics yang terdiri dari:

  1. Collection - Agent atau API mengumpulkan metrics dari resource
  2. Storage - Time-series database (TSDB) menyimpan data
  3. Visualization - Dashboard menampilkan tren dan real-time data
  4. Alerting - Threshold dan anomaly detection memicu notifikasi
  5. Automation - Incident response dan auto-remediation
flowchart LR
    S[Sumber Metrics<br/>VM, Container, DB, API] --> A[Agen Monitoring<br/>CloudWatch Agent / Azure Monitor Agent / Ops Agent]
    A --> TSDB[Time-Series DB<br/>CloudWatch / Azure Monitor / GCP Monitoring]
    TSDB --> D[Dashboard<br/>CloudWatch Dashboard / Azure Workbooks / Grafana]
    TSDB --> AL[Alerting<br/>CloudWatch Alarms / Azure Alerts / Alertmanager]
    AL --> IR[Incident Response<br/>Lambda / Logic Apps / Cloud Functions]
    IR --> S

AWS CloudWatch - Metrics, Logs & Alarms

Amazon CloudWatch adalah layanan monitoring default untuk AWS. Mencakup metrics dari 70+ layanan AWS, custom metrics, log monitoring, dan alarms.

CloudWatch Metrics

Setiap resource AWS mengeluarkan metrics secara otomatis:

Layanan Metrics Utama
EC2 CPUUtilization, NetworkIn/Out, DiskRead/Write, StatusCheckFailed
RDS DatabaseConnections, ReadLatency, WriteLatency, FreeStorageSpace
ALB/NLB RequestCount, TargetResponseTime, HTTPCode_ELB_5xx, HealthyHostCount
Lambda Invocations, Duration, Throttles, Errors, ConcurrentExecutions
API Gateway Count, Latency, 4xxError, 5xxError

Custom Metrics - Memantau Aplikasi Kustom

Kirim metrics kustom dari aplikasi ke CloudWatch:

# CLI: mengirim custom metric
aws cloudwatch put-metric-data \
    --namespace "MyApp" \
    --metric-data '[
        {"MetricName": "OrderProcessingTime", "Unit": "Milliseconds", "Value": 245},
        {"MetricName": "ActiveUsers", "Unit": "Count", "Value": 1823},
        {"MetricName": "ErrorRate", "Unit": "Percent", "Value": 0.02, "Dimensions": [{"Name": "Service", "Value": "checkout-api"}]}
    ]'

Contoh implementasi di Python menggunakan boto3:

import boto3
import time

cloudwatch = boto3.client('cloudwatch', region_name='ap-southeast-1')

def send_business_metrics(order_count, revenue, error_rate):
    cloudwatch.put_metric_data(
        Namespace='ECommerceApp',
        MetricData=[
            {
                'MetricName': 'OrderCount',
                'Value': order_count,
                'Unit': 'Count',
                'Timestamp': time.time()
            },
            {
                'MetricName': 'Revenue',
                'Value': revenue,
                'Unit': 'USD',
                'Dimensions': [{'Name': 'Currency', 'Value': 'USD'}]
            },
            {
                'MetricName': 'ErrorRate',
                'Value': error_rate,
                'Unit': 'Percent',
                'Dimensions': [{'Name': 'Service', 'Value': 'Checkout'}]
            }
        ]
    )

CloudWatch Dashboard

Buat dashboard visual untuk operasional tim:

# Membuat dashboard dengan widget
aws cloudwatch put-dashboard \
    --dashboard-name "Production-Overview" \
    --dashboard-body '{
        "widgets": [
            {
                "type": "metric",
                "x": 0, "y": 0, "width": 12, "height": 6,
                "properties": {
                    "metrics": [
                        ["AWS/EC2", "CPUUtilization", {"stat": "Average"}]
                    ],
                    "period": 300,
                    "stat": "Average",
                    "region": "ap-southeast-1",
                    "title": "EC2 CPU Utilization"
                }
            },
            {
                "type": "log",
                "x": 12, "y": 0, "width": 12, "height": 6,
                "properties": {
                    "query": "SOURCE '/aws/lambda/api-gateway' | fields @timestamp, @message\n| filter @message like /ERROR|Exception/\n| limit 50",
                    "region": "ap-southeast-1",
                    "title": "Lambda Error Logs"
                }
            }
        ]
    }'

CloudWatch Anomaly Detection

Anomaly detection menggunakan machine learning untuk mempelajari pola metrics historis dan mendeteksi penyimpangan:

# Buat anomaly detection band untuk metric CPU
aws cloudwatch put-anomaly-detector \
    --namespace "AWS/EC2" \
    --metric-name "CPUUtilization" \
    --stat "Average" \
    --configuration '{
        "ExcludedTimeRanges": [],
        "MetricTimezone": "Asia/Jakarta"
    }'

# Buat alarm dengan anomaly detection band
aws cloudwatch put-metric-alarm \
    --alarm-name "CPU-Anomaly-Alert" \
    --alarm-description "CPU utilization di luar band normal" \
    --metrics '[
        {
            "Id": "m1",
            "ReturnData": true,
            "MetricStat": {
                "Metric": {
                    "Namespace": "AWS/EC2",
                    "MetricName": "CPUUtilization"
                },
                "Period": 300,
                "Stat": "Average"
            }
        },
        {
            "Id": "ad1",
            "Expression": "ANOMALY_DETECTION_BAND(m1, 2)",
            "Label": "CPUUtilization (expected)",
            "ReturnData": true
        }
    ]' \
    --evaluation-periods 2 \
    --alarm-actions arn:aws:sns:ap-southeast-1:123456789012:monitoring-team

Alarm Thresholds - Best Practice Thresholds

Metric Threshold Action
CPUUtilization > 90% selama 10 menit Scale out / send alert
StatusCheckFailed > 0 Replace instance
5xxError > 1% dari total request Notify engineering team
Latency p99 > 2000ms Investigasi bottleneck
FreeStorageSpace < 10% Extend volume

Azure Monitor - Metrics, Workbooks & Alerts

Azure Monitor menyediakan platform monitoring terpadu untuk seluruh Azure resources, aplikasi, dan on-premise infrastructure.

Azure Monitor Metrics

Azure Monitor secara otomatis mengumpulkan platform metrics dari setiap resource:

# Melihat metrics VM dengan Azure CLI
az monitor metrics list \
    --resource /subscriptions/xxx/resourceGroups/prod-rg/providers/Microsoft.Compute/virtualMachines/web-vm-01 \
    --metric "Percentage CPU" \
    --interval PT5M \
    --aggregation average,maximum

# Membuat metric alert
az monitor metrics alert create \
    --name "High-CPU-Alert" \
    --resource-group prod-rg \
    --scopes /subscriptions/xxx/resourceGroups/prod-rg/providers/Microsoft.Compute/virtualMachines/web-vm-01 \
    --condition "avg 'Percentage CPU' > 85" \
    --window-size 10m \
    --evaluation-frequency 5m \
    --action-groups /subscriptions/xxx/resourceGroups/security-rg/providers/microsoft.insights/actionGroups/oncall-team

Azure Application Insights - APM untuk Aplikasi

Application Insights adalah Application Performance Monitoring (APM) untuk aplikasi .NET, Java, Node.js, dan Python:

// Konfigurasi Application Insights di ASP.NET Core - appsettings.json
{
  "ApplicationInsights": {
    "ConnectionString": "InstrumentationKey=xxx;IngestionEndpoint=https://southeastasia-0.in.applicationinsights.azure.com/"
  }
}

// Pengukuran custom event
public class OrderController : ControllerBase
{
    private readonly TelemetryClient _telemetry;

    public OrderController(TelemetryClient telemetry)
    {
        _telemetry = telemetry;
    }

    [HttpPost]
    public async Task<IActionResult> CreateOrder(OrderRequest request)
    {
        var stopwatch = Stopwatch.StartNew();

        // Proses order
        await _orderService.ProcessOrder(request);

        stopwatch.Stop();

        // Kirim custom metric
        _telemetry.TrackMetric("OrderProcessingTime", stopwatch.ElapsedMilliseconds);
        _telemetry.TrackEvent("OrderPlaced", new Dictionary<string, string>
        {
            {"CustomerTier", request.CustomerTier},
            {"TotalItems", request.Items.Count.ToString()}
        });

        return Ok();
    }
}

Azure Workbooks - Dashboard Interaktif

Azure Workbooks memungkinkan pembuatan dashboard interaktif dengan KQL queries:

// Workbook query: Health status semua VM
{
  "type": 3,
  "content": {
    "version": "KqlItem/1.0",
    "query": "InsightsMetrics\n| where Namespace == 'Processor' and Name == 'UtilizationPercentage'\n| summarize AvgCPU = avg(Val) by Computer, bin(TimeGenerated, 5m)\n| project Computer, AvgCPU, HealthStatus = iff(AvgCPU > 90, 'CRITICAL', iff(AvgCPU > 70, 'WARNING', 'HEALTHY'))",
    "size": 0,
    "timeContext": {
      "durationMs": 3600000
    }
  },
  "name": "VM Health Status"
}

GCP Cloud Monitoring - Metrics Explorer & Alerting

GCP Cloud Monitoring (dulu Stackdriver) menyediakan observability untuk GCP dan AWS resources.

Metrics Explorer - Query Metrics

# Query CPU utilization untuk semua VM
gcloud monitoring metrics list \
    --filter="metric.type = compute.googleapis.com/instance/cpu/utilization" \
    --limit=10

# Membuat alert policy untuk CPU > 80%
gcloud alpha monitoring policies create \
    --display-name="High CPU Utilization Prod" \
    --condition-display-name="CPU > 80% for 5 minutes" \
    --condition-filter='resource.type="gce_instance" AND metric.type="compute.googleapis.com/instance/cpu/utilization"' \
    --condition-threshold-value=0.80 \
    --condition-threshold-duration=300s \
    --notification-channels="projects/my-project/notificationChannels/123456"

Uptime Check - Memantau Endpoint Eksternal

# Buat uptime check untuk aplikasi web
gcloud monitoring uptime-check-configs create \
    display-name="Production-Web-Check" \
    --resource-type="https://api.myapp.com/health" \
    --period=5 \
    --timeout=10 \
    --check-interval=300s \
    --selected-regions=asia-southeast1,us-west1,europe-west1

Ops Agent - Unified Agent untuk GCP

Ops Agent menggabungkan Logging Agent dan Monitoring Agent sebelumnya dalam satu paket:

# /etc/google-cloud-ops-agent/config.yaml
metrics:
  receivers:
    nginx_status:
      type: nginx
      stub_status_url: http://localhost:80/nginx_status
  service:
    pipelines:
      nginx_pipeline:
        receivers:
          - nginx_status
logging:
  receivers:
    app_logs:
      type: files
      include_paths:
        - /var/log/myapp/*.log
      record_log_file_path: true
  processors:
    parse_json:
      type: parse_json
  service:
    pipelines:
      app_pipeline:
        receivers:
          - app_logs
        processors:
          - parse_json

Incident Response Automation

Monitoring tanpa otomatisasi hanya akan membanjiri tim dengan notifikasi. Automation memungkinkan self-healing infrastructure.

AWS - Lambda + CloudWatch untuk Auto-Remediation

# Lambda function untuk auto-remediate security group changes
import boto3

ec2 = boto3.client('ec2')

def lambda_handler(event, context):
    """
    Dipicu oleh CloudWatch Alarm ketika security group tidak patuh.
    Kembalikan aturan security group ke baseline yang disetujui.
    """
    alarm_data = event['detail']
    sg_id = alarm_data.get('sourceResource', '')

    if not sg_id:
        return {'status': 'error', 'message': 'No security group ID found'}

    # Baseline: hanya allow HTTPS dari internet
    baseline_rules = [
        {
            'IpProtocol': 'tcp',
            'FromPort': 443,
            'ToPort': 443,
            'IpRanges': [{'CidrIp': '0.0.0.0/0'}]
        }
    ]

    # Revoke semua aturan ingress yang tidak sesuai
    try:
        current_rules = ec2.describe_security_group_rules(
            Filters=[{'Name': 'group-id', 'Values': [sg_id]}]
        )

        for rule in current_rules['SecurityGroupRules']:
            if not rule['IsEgress'] and not rule['CidrIpv4'] == '0.0.0.0/0':
                ec2.revoke_security_group_ingress(
                    GroupId=sg_id,
                    SecurityGroupRuleIds=[rule['SecurityGroupRuleId']]
                )

        # Apply baseline
        ec2.authorize_security_group_ingress(
            GroupId=sg_id,
            IpPermissions=baseline_rules
        )

        return {
            'status': 'remediated',
            'security_group': sg_id,
            'action': 'Baseline rules applied'
        }
    except Exception as e:
        return {'status': 'error', 'message': str(e)}

Azure - Logic Apps untuk Auto-Remediation

{
  "definition": {
    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
    "triggers": {
      "When_a_metric_alert_is_fired": {
        "type": "ApiConnection",
        "inputs": {
          "host": {
            "connection": {
              "name": "@parameters('$connections')['azuremonitor']['connectionId']"
            }
          },
          "method": "post",
          "body": {
            "alertContext": "@triggerBody()"
          }
        }
      }
    },
    "actions": {
      "Restart_VM": {
        "type": "ApiConnection",
        "inputs": {
          "host": {
            "connection": {
              "name": "@parameters('$connections')['azurerm']['connectionId']"
            }
          },
          "method": "post",
          "path": "/subscriptions/@{encodeURIComponent('xxx')}/resourceGroups/@{encodeURIComponent('prod-rg')}/providers/Microsoft.Compute/virtualMachines/@{encodeURIComponent(triggerBody()['data']['resourceName'])}/restart"
        }
      },
      "Send_Notification": {
        "type": "ApiConnection",
        "inputs": {
          "host": {
            "connection": {
              "name": "@parameters('$connections')['teams']['connectionId']"
            }
          },
          "method": "post",
          "body": {
            "message": "⚠️ VM @{triggerBody()['data']['resourceName']} di-restart karena CPU > 95% selama 10 menit"
          }
        }
      }
    }
  }
}

Compliance Monitoring

Compliance monitoring memastikan infrastruktur cloud selalu mematuhi kebijakan keamanan dan regulasi.

AWS Config - Compliance Rules

# AWS Config rule: S3 bucket harus encrypted
aws configservice put-config-rule \
    --config-rule '{
        "ConfigRuleName": "s3-bucket-ssl-requests-only",
        "Description": "Checks whether S3 buckets have policy requiring SSL",
        "Source": {
            "Owner": "AWS",
            "SourceIdentifier": "S3_BUCKET_SSL_REQUESTS_ONLY"
        },
        "Scope": {
            "ComplianceResourceTypes": ["AWS::S3::Bucket"]
        }
    }'

# Evaluasi compliance
aws configservice get-compliance-details-by-config-rule \
    --config-rule-name s3-bucket-ssl-requests-only

Azure Policy - Built-in Compliance Initiatives

# Assign Azure Policy untuk encryption requirements
az policy assignment create \
    --name "encryption-policy" \
    --display-name "Require Encryption at Rest" \
    --policy-set-definition "EnableEncryptionAtRest" \
    --resource-group prod-rg \
    --params '{
        "effect": {"value": "AuditIfNotExists"},
        "encryptionOnVm": {"value": "Enabled"}
    }'

# List all non-compliant resources
az policy state list \
    --resource-group prod-rg \
    --filter "complianceState eq 'NonCompliant'"

GCP Organization Policies - Guardrails Global

# Terapkan constraint: tidak boleh ada Service Account key yang dibuat
gcloud resource-manager org-policies set-policy \
    --organization=123456789012 \
    --policy-file='{
        "constraint": "constraints/iam.disableServiceAccountKeyCreation",
        "listPolicy": {
            "allValues": "DENY"
        }
    }'

# Constraint: hanya allow approved regions
gcloud resource-manager org-policies set-policy \
    --project=my-project \
    --policy-file='{
        "constraint": "constraints/gcp.resourceLocations",
        "listPolicy": {
            "allowedValues": [
                "asia-southeast1",
                "asia-southeast2",
                "us-west1"
            ]
        }
    }'

Third-Party Monitoring Tools

Meskipun cloud provider tools sangat mumpuni, banyak organisasi menggunakan third-party tools untuk observability multi-cloud.

1. Datadog - Unified Monitoring Multi-Cloud

Integrasi Datadog dengan AWS:

# datadog-agent-values.yaml (Helm)
datadog:
  apiKey: ${DD_API_KEY}
  site: ap1.datadoghq.com
  logs:
    enabled: true
    containerCollectAll: true
  processAgent:
    enabled: true
  systemProbe:
    enabled: true
  networkMonitoring:
    enabled: true

# Integrasi AWS di Datadog
# 1. Buat IAM role dengan policy DatadogAWSIntegrationRole
# 2. Datadog akan mengumpulkan CloudWatch metrics, CloudTrail logs, dan real-time data

2. New Relic - Full-Stack Observability

# Install New Relic Infrastructure Agent di EC2
echo "license_key: ${NEW_RELIC_LICENSE_KEY}" | sudo tee -a /etc/newrelic-infra.yml
sudo systemctl enable --now newrelic-infra

# New Relic Browser - monitoring frontend
# Tambahkan script injection di HTML

3. Grafana - Open Source Dashboard

Grafana dapat mengkonsolidasi metrics dari berbagai sumber:

# docker-compose untuk Grafana + Prometheus
version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.53.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD}
    volumes:
      - grafana-storage:/var/lib/grafana
      - ./datasources:/etc/grafana/provisioning/datasources

volumes:
  grafana-storage:

Konfigurasi datasource untuk AWS CloudWatch:

# /etc/grafana/provisioning/datasources/cloudwatch.yaml
apiVersion: 1
datasources:
  - name: AWS CloudWatch
    type: cloudwatch
    access: proxy
    jsonData:
      authType: default
      defaultRegion: ap-southeast-1
    secureJsonData:
      accessKey: ${AWS_ACCESS_KEY_ID}
      secretKey: ${AWS_SECRET_ACCESS_KEY}

Perbandingan Tools Monitoring

Fitur CloudWatch Azure Monitor GCP Monitoring Datadog Grafana
Native integration AWS only Azure only GCP only Multi-cloud Multi-source
Anomaly detection ML bands Smart Detection ML-based Watchdog + Alertmanager
Custom metrics Ya Ya Ya Ya Prometheus
SIEM integration + Splunk/ELK + Sentinel + Chronicle Log Management Loki
Dashboard CloudWatch Dashboard Workbooks Metrics Explorer Screenboards Grafana Dashboard
Pricing Per metric Per GB ingested Per volume Per host Open source (free)

Verifikasi

# AWS - cek alarm dan metrics
aws cloudwatch describe-alarms --state-value ALARM
aws cloudwatch get-metric-statistics \
    --namespace AWS/EC2 --metric-name CPUUtilization \
    --start-time "2026-07-22T08:00:00Z" \
    --end-time "2026-07-22T09:00:00Z" \
    --period 300 --statistics Average

# Azure - cek metric alert status
az monitor metrics alert list --resource-group prod-rg

# GCP - list alert policies
gcloud alpha monitoring policies list

# Grafana - cek datasource health
curl -s http://admin:${GRAFANA_PASSWORD}@localhost:3000/api/datasources

Referensi

← Sebelumnya 9.5.1 Cloud Logging
PADA HALAMAN INI