Capturing the network traffic with Packetbeat

Greetings again everybody! Up to this point, we have covered almost the whole arquitecture we want to cover. For example, we learn about Elasticsearch for storing the data that we collect and how to deploy it, Kibana as a Web UI for visualizing the collected data, Filebeat for collecting data from our cluster, we saw what Logstash can do, collected metrics from the system and services running on the server with the help of Metricbeat, and know if your service is up thanks to Heartbeat. Now it is time to analyze your network traffic with Packetbeat.

Overview

Monitoring network traffic at the host level is no longer enough. This was useful long time ago,  when individual services ran on stand-alone machines. Nowadays, if you want to get a meaningful analysis today, you have to check network traffic at the process level.

Packetbeat is a real-time network packet analyzer, parses the application-level protocols on the fly, and correlates the messages into transactions, thus providing visibility between the servers of your network.

Just as any other Elastic Beat, Packetbeat is based on the libbeat framework.

Deploying Packetbeat in Docker

Let’s begin by adding a folder which will have Pcketbeat’s files. The changes in the project should be highlighted.

elastic-stack-demo
  +- elasticsearch-single-node-cluster
       +- elasticsearch
       |    +- Dockerfile-elasticsearch-single-node
       |    +- elasticsearch-single-node.yml
       +-filebeat
       |    +- Dockerfile
       |    +- filebeat-to-elasticsearch.yml
       |    +- filebeat-to-logstash.yml
       +-heartbeat
       |    +- Dockerfile
       |    +- heartbeat.yml
       |    +- http_dashboard.ndjson
       +-kibana
       |    +- Dockerfile-kibana-single-node
       |    +- kibana-single-node.yml
       +-logstash
       |    +- config
       |    |    +- logstash.yml
       |    |    +- pipelines.yml
       |    +- pipeline
       |    |    +- beats-example.conf
       |    |    +- data-stream-example.conf
       |    |    +- output.conf
       |    +- Dockerfile
       +-metricbeat
       |    +- Dockerfile
       |    +- metricbeat.yml
       +-packetbeat
       |    +- Dockerfile
       |    +- packetbeat.yml
       +- .env
       +- docker-compose-es-single-node.yml
       +- docker-compose-filebeat-to-elasticseach.yml
       +- docker-compose-filebeat-to-logstash.yml
       +- docker-compose-heartbeat.yml
       +- docker-compose-logstash.yml
       +- docker-compose-metricbeat.yml
       +- docker-compose-packetbeat.yml

As we have been doing so far, the first file we will be creating is the Dockerfile. Create it under elastic-stack-single-node-cluster/packetbeat/, and paste the following code:

ARG ELK_VERSION
FROM docker.elastic.co/beats/packetbeat:${ELK_VERSION}

# add custom configuration
COPY --chown=root:packetbeat packetbeat.yml /usr/share/packetbeat/packetbeat.yml

The file has nothing extraordinary. It is just specifying the base image and copying the configuration YAML file for Packetbeat. This configuration file looks like this:

###################### Packetbeat Configuration Example #######################
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/packetbeat/index.html

# =============================== Network device ===============================
# Select the network interface to sniff the data. You can use the "any"
# keyword to sniff on all connected interfaces.
packetbeat.interfaces.device: any

# The network CIDR blocks that are considered "internal" networks for
# the purpose of network perimeter boundary classification. The valid
# values for internal_networks are the same as those that can be used
# with processor network conditions.
#
# For a list of available values see:
# https://www.elastic.co/guide/en/beats/packetbeat/current/defining-processors.html#condition-network
packetbeat.interfaces.internal_networks:
  - private

# =================================== Flows ====================================
packetbeat.flows:
  # Enable Network flows. Default: true
  enabled: true
  # Set network flow timeout. Flow is killed if no packet is received before being
  # timed out.
  timeout: 30s
  # Configure reporting period. If set to -1, only killed flows will be reported
  period: 10s

# =========================== Transaction protocols ============================
packetbeat.protocols:
  - type: icmp
    # Enable ICMPv4 and ICMPv6 monitoring. The default is true.
    enabled: true
    # Overrides where this protocol's events are indexed.
    index: demo-icmp-index

  - type: dns
    # Enable DNS monitoring. Default: true
    enabled: true
    # Configure the ports where to listen for DNS traffic. You can disable
    # the DNS protocol by commenting out the list of ports.
    ports: [53]
    # include_authorities controls whether or not the dns.authorities field
    # (authority resource records) is added to messages.
    # Default: false
    include_authorities: true
    # include_additionals controls whether or not the dns.additionals field
    # (additional resource records) is added to messages.
    # Default: false
    include_additionals: true

  - type: http
    # Enable HTTP monitoring. Default: true
    enabled: true
    # Configure the ports where to listen for HTTP traffic. You can disable
    # the HTTP protocol by commenting out the list of ports.
    ports: [80, 5601, 9200, 8080, 8081, 5000, 8002]

  - type: memcache
    # Enable memcache monitoring. Default: true
    enabled: true
    # Configure the ports where to listen for memcache traffic. You can disable
    # the Memcache protocol by commenting out the list of ports.
    ports: [11211]

  - type: nfs
    # Enable NFS monitoring. Default: true
    enabled: true
    # Configure the ports where to listen for NFS traffic. You can disable
    # the NFS protocol by commenting out the list of ports.
    ports: [2049]

  - type: tls
    # Enable TLS monitoring. Default: true
    enabled: true
    # Configure the ports where to listen for TLS traffic. You can disable
    # the TLS protocol by commenting out the list of ports.
    ports:
      - 443   # HTTPS
      - 993   # IMAPS
      - 995   # POP3S
      - 5223  # XMPP over SSL
      - 8443
      - 8883  # Secure MQTT
      - 9243  # Elasticsearch

# ============================ Monitored processes =============================
# Packetbeat can enrich events with information about the process associated
# the socket that sent or received the packet if Packetbeat is monitoring
# traffic from the host machine. By default process enrichment is disabled.
# This feature works on Linux and Windows.
packetbeat.procs.enabled: true

# If you want to ignore transactions created by the server on which the shipper
# is installed you can enable this option. This option is useful to remove
# duplicates if shippers are installed on multiple servers. Default value is
# false.
packetbeat.ignore_outgoing: true

# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
# If this options is not defined, the hostname is used.
name: packetbeat-demo

# ================================= Processors =================================
# Processors are used to reduce the number of fields in the exported event or to
# enhance the event with external metadata. This section defines a list of
# processors that are applied one by one and the first one receives the initial
# event:
#
#   event -> filter1 -> event1 -> filter2 ->event2 ...
#
# The supported processors are drop_fields, drop_event, include_fields,
# decode_json_fields, and add_cloud_metadata.
processors:
  # The following example enriches each event with docker metadata, it matches
  # container id from log path available in `source` field (by default it expects
  # it to be /var/lib/docker/containers/*/*.log).
  - add_docker_metadata: ~
  # The following example enriches each event with host metadata.
  - add_host_metadata: ~

# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ['localhost:9200']

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards are disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

# ================================== Logging ===================================
# There are four options for the log output: file, stderr, syslog, eventlog
# The file output is the default.
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info

# If enabled, Packetbeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
logging.metrics.period: 30s

# Logging to rotating files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true

# ============================= X-Pack Monitoring ==============================
# Packetbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
monitoring.enabled: true

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
monitoring.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  #hosts: ['elasticsearch-demo:9200']

# =============================== HTTP Endpoint ================================
# Each beat can expose internal metrics through a HTTP endpoint. For security
# reasons the endpoint is disabled by default. This feature is currently experimental.
# Stats can be access through http://localhost:5066/stats . For pretty JSON output
# append ?pretty to the URL.
# Defines if the HTTP endpoint is enabled.
http.enabled: true

# The HTTP endpoint will bind to this hostname, IP address, unix socket or named pipe.
# When using IP addresses, it is recommended to only use localhost.
http.host: localhost

# Port on which the HTTP endpoint will bind. Default is 5066.
http.port: 5066

As you can see, we have included the description of each configuration option. Hopefully, it will be easier to understand it. However, the main idea behind it, is:

• Enable icmp, dns, http and tls transanction protocols.
• Enable providers, which work by watching for events on the system and translating those events into internal autodiscover events with a common format.
• Send the collected data to Elasticsearch for indexing.
• Export internal metrics to a central Elasticsearch monitoring cluster, by enabling x-pack monitoring. In our case, we will be using the same cluster.
• Enable experimental HTTP endpoint, which exposes internal metrics.

You might be curious why in this beat, when we want to reference it inside the configuration YAML file, we use localhost. We will explain the main reason later in this post, when we analyze its docker-compose file.

Now, we create a separate docker-compose file under elastic-stack-single-node-cluster/ and name it docker-compose-packetbeat.yml.

version: '3.9'
services:
  packetbeat-demo:
    hostname: packetbeat-demo
    container_name: packetbeat-demo
    build:
      context: ./packetbeat
      dockerfile: Dockerfile
      args:
        - ELK_VERSION=${ELK_VERSION}
    ports:
      - 5566:5066
    user: packetbeat
    cap_add:
      - NET_ADMIN
    # disable strict permission checks
    command: [ '-e', '-v', '--strict.perms=false' ]
    network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,...

By default, Docker networking will connect the Packetbeat container to an isolated virtual network, with a limited view of network traffic. We use docker-compose‘s option network_mode: host, because we wish to connect the container directly to the host network in order to see traffic destined for, and originating from, the host system.

So, for our use case, we have the Packetbeat container running with host networking and not attach it to the docker networks. Because of that, we are no longer able to connect it to the elasticsearch instance via http://elasticsearch-demo:9200, so replaced this config value to http://localhost:9200 instead. All this is done in packetbeat.yml.

Moreover, under Docker, Packetbeat runs as a non-root user, but requires some privileged network capabilities to operate correctly. Option cap_add: NET_ADMIN needs to be available to the container.

Great. We are ready to start Packetbeat, by executing the following command:

$ docker-compose -f docker-compose-packetbeat.yml up -d --build

If you go to Analytics > Dashboards and look for a dashboard called [Packetbeat] Overview ECS. Click it and you will see an overview of the network’s metrics:

Clean Up

To do a complete clean up, execute this command:

$ docker-compose -f docker-compose-es-single-node.yml -f docker-compose-filebeat-to-elasticseach.yml -f docker-compose-filebeat-to-logstash.yml -f docker-compose-logstash.yml -f docker-compose-metricbeat.yml -f docker-compose-heartbeat.yml -f docker-compose-packetbeat.yml down -v

Summary

In this post, we learn about Packetbeat and how it can be deployed within a dockerized enviroment. Once deployed, Packetbeat sniffs the traffic between your servers, parses the application-level protocols on the fly, and correlates the messages into transactions.

Please feel free to contact us. We will gladly response to any doubt or question you might have. In the mean time, you can download the source code from our official GitHub repository.