Beats Integration mit OpenSearch

Beats sind leichtgewichtige Datenversender, die auf verschiedenen Systemen installiert werden können, um spezifische Arten von Daten zu sammeln und an OpenSearch zu übermitteln. Durch die Kompatibilität mit der Elasticsearch-API können Beats nahtlos mit OpenSearch integriert werden.

62.1 Übersicht der verfügbaren Beats

62.2 Filebeat Konfiguration und Einsatz

62.2.1 Basis-Installation und Konfiguration

# filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/messages
    - /var/log/syslog
  fields:
    environment: production
    datacenter: eu-central

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "admin"
  password: "admin"
  ssl.verification_mode: "none"  # Für Produktivumgebungen entsprechend anpassen
  
  # Wichtig: Index-Template-Kompatibilität
  index: "logs-%{[agent.version]}-%{+yyyy.MM.dd}"
  pipeline: "filebeat-default"

setup.template.name: "filebeat"
setup.template.pattern: "filebeat-*"
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 1

62.2.2 Modulare Konfiguration

Filebeat unterstützt Module für verschiedene Anwendungen. Hier ein Beispiel für Nginx-Logs:

# filebeat.yml
filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]

# OpenSearch Output wie oben

62.2.3 Multiline-Log-Handling

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/java-app.log
  multiline:
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after

62.3 Metricbeat für System- und Service-Überwachung

62.3.1 Basis-Konfiguration

# metricbeat.yml
metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - load
    - memory
    - network
    - process
  enabled: true
  period: 10s
  processes: ['.*']

- module: docker
  metricsets:
    - container
    - cpu
    - image
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "admin"
  password: "admin"
  ssl.verification_mode: "none"
  index: "metrics-%{[agent.version]}-%{+yyyy.MM.dd}"

62.3.2 Kubernetes Monitoring

# metricbeat.yml für Kubernetes
metricbeat.modules:
- module: kubernetes
  metricsets:
    - container
    - node
    - pod
    - system
    - volume
  period: 10s
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none"
  
# Kube-state-metrics
- module: kubernetes
  enabled: true
  metricsets:
    - state_node
    - state_deployment
    - state_replicaset
    - state_pod
  period: 10s
  hosts: ["kube-state-metrics:8080"]

62.4 Packetbeat für Netzwerk-Monitoring

62.4.1 Basis-Konfiguration

# packetbeat.yml
packetbeat.interfaces.device: any

packetbeat.flows:
  timeout: 30s
  period: 10s

packetbeat.protocols:
- type: icmp
  enabled: true

- type: http
  ports: [80, 8080, 443]
  send_headers: true
  send_all_headers: true
  real_ip_header: "X-Forwarded-For"

- type: mysql
  ports: [3306]

- type: pgsql
  ports: [5432]

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "admin"
  password: "admin"
  ssl.verification_mode: "none"
  index: "packetbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

62.5 Auditbeat für Security Monitoring

62.5.1 Basis-Konfiguration

# auditbeat.yml
auditbeat.modules:
- module: auditd
  audit_rules: |
    -w /etc/passwd -p wa -k identity
    -w /etc/group -p wa -k identity
    -w /etc/sudoers -p wa -k sudoers
    -a always,exit -F arch=b64 -S execve -k exec

- module: file_integrity
  paths:
    - /bin
    - /usr/bin
    - /sbin
    - /usr/sbin
    - /etc

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "admin"
  password: "admin"
  ssl.verification_mode: "none"
  index: "auditbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

62.6 Heartbeat für Uptime Monitoring

62.6.1 Basis-Konfiguration

# heartbeat.yml
heartbeat.monitors:
- type: http
  schedule: '@every 10s'
  urls:
    - https://example.com
  check.response.status: [200]
  ssl.verification_mode: "none"

- type: tcp
  schedule: '@every 30s'
  hosts:
    - database:5432

- type: icmp
  schedule: '@every 5s'
  hosts:
    - 192.168.1.1
    - 192.168.1.2

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "admin"
  password: "admin"
  ssl.verification_mode: "none"
  index: "heartbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

62.7 Performance-Optimierung und Best Practices

62.7.1 Memory Management

# Allgemeine Performance-Einstellungen
queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: "5s"

output.elasticsearch:
  bulk_max_size: 50
  worker: 4
  
processors:
  - drop_fields:
      fields: ["agent.ephemeral_id", "ecs.version"]
  - drop_event:
      when:
        regexp:
          message: "^DEBUG"

62.7.2 Load Balancing und Failover

output.elasticsearch:
  hosts: ["https://opensearch1:9200", "https://opensearch2:9200"]
  loadbalance: true
  max_retries: 3
  backoff.init: "1s"
  backoff.max: "60s"

62.8 Monitoring der Beats

62.8.1 Internal Monitoring

monitoring:
  enabled: true
  elasticsearch:
    hosts: ["https://monitoring-opensearch:9200"]
    username: "beats_system"
    password: "yourpassword"

62.8.2 Metrics Exposition

http.enabled: true
http.port: 5066

monitoring.enabled: true
monitoring.elasticsearch:
  metrics.enabled: true

62.9 Troubleshooting und Debugging

62.9.1 Logging Konfiguration

logging.level: debug
logging.to_files: true
logging.files:
  path: /var/log/beats
  name: beat
  keepfiles: 7
  permissions: 0644

62.9.2 Diagnose-Modus

# Aktivieren Sie den Diagnose-Modus temporär
logging.selectors: ["*"]

62.10 Security Best Practices

62.10.1 TLS/SSL Konfiguration

output.elasticsearch:
  hosts: ["https://opensearch:9200"]
  protocol: "https"
  username: "beats_admin"
  password: "${BEATS_PASSWORD}"
  
  ssl:
    enabled: true
    certificate_authorities: ["/etc/pki/root/ca.pem"]
    certificate: "/etc/pki/client/cert.pem"
    key: "/etc/pki/client/cert.key"
    verification_mode: full

62.10.2 Credentials Management

62.11 Praktische Übungen

Diese Übungen helfen Ihnen, die verschiedenen Beats in der Praxis kennenzulernen und effektiv einzusetzen.

62 Beats Integration mit OpenSearch