{"id":564,"date":"2021-12-26T15:20:52","date_gmt":"2021-12-26T15:20:52","guid":{"rendered":"https:\/\/www.canchito-dev.com\/public\/blog\/?p=564"},"modified":"2022-02-05T13:44:02","modified_gmt":"2022-02-05T13:44:02","slug":"introduction-to-elastic-stack","status":"publish","type":"post","link":"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/introduction-to-elastic-stack\/","title":{"rendered":"Introduction to Elastic Stack"},"content":{"rendered":"<h1>Introduction to Elastic Stack<\/h1>\n<div class=\"perfect-pullquote vcard pullquote-align-full pullquote-border-placement-left\"><blockquote><p><\/p>\n<p>Hello friends! In this post, we will give you a small introduction to Elastic Stack including all the products that build it.<\/p>\n<p><\/p><\/blockquote><\/div>\n<p><a class=\"donate-with-crypto\" href=\"https:\/\/commerce.coinbase.com\/checkout\/faf64f90-2e80-46ee-aeba-0fde14cbeb46\">Buy Me a Coffee<br \/>\n<\/a><br \/>\n<script src=\"https:\/\/commerce.coinbase.com\/v1\/checkout.js?version=201807\"><\/script><\/p>\n<h2 style=\"text-align: justify;\">Introduction<\/h2>\n<p style=\"text-align: justify;\">First things first. What is Elastic Stack? Well, Elastic Stack is a set of open-source products, originally designed to help users gather data from different sources and in any format, and allow them to perform some actions on these data, such as real-time searches, analysis or visualization.\u00a0You might know these group of products as ELK, which stands for <strong>E<\/strong>lasticsearch, <strong>K<\/strong>ibana and <strong>L<\/strong>ogstash. Later on, Beats product was added to the stack.<\/p>\n<p style=\"text-align: justify;\">Nowadays, Elastic Stack is also know for being a complete end-to-end log analysis solution. Thanks to it&#8217;s Observability feature, users can perform centralized logging helping them in identifying the problems with the web servers or applications fast and easy.<\/p>\n<h2>Elastic Stack Components<\/h2>\n<p style=\"text-align: justify;\">For now, we will just give a brief description of all the products that form the Elastic Stack.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" data-attachment-id=\"613\" data-permalink=\"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/introduction-to-elastic-stack\/elastic_stack\/\" data-orig-file=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/elastic_stack.png\" data-orig-size=\"623,366\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CANCHITO-DEV: Elastic Stack\" data-image-description=\"\" data-image-caption=\"&lt;p&gt;CANCHITO-DEV: Elastic Stack&lt;\/p&gt;\n\" data-large-file=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/elastic_stack.png\" class=\"aligncenter wp-image-613 size-full\" src=\"http:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/elastic_stack.png\" alt=\"CANCHITO-DEV: Elastic Stack\" width=\"623\" height=\"366\" srcset=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/elastic_stack.png 623w, https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/elastic_stack-300x176.png 300w\" sizes=\"auto, (max-width: 623px) 100vw, 623px\" \/><\/p>\n<ul class=\"default-list\">\n<li style=\"text-align: justify;\">At the heart of Elastic Stack, we find Elasticsearch. Consider it a distributed search and analytics engine, which provides near real-time search and analytics for all types of data. It does not matter the nature of your data, if it is structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches.<\/li>\n<li style=\"text-align: justify;\">Logstash is a data collection engine with real-time pipelining capabilities. It allow the unification and normalization of data coming from different sources before sending it into destinations of your choice.<\/li>\n<li style=\"text-align: justify;\">Beats are data shippers that you install as agents on your servers to send operational data to Elasticsearch.<\/li>\n<li style=\"text-align: justify;\">Kibana is data visualization\u00a0and exploration tool from that is specialized for large volumes of streaming and real-time data. The software makes huge and complex data streams more easily and quickly understandable through graphic representation.<\/li>\n<\/ul>\n<p style=\"text-align: justify;\">In our following posts, we will dive deeper into each of the components. But first, allow us to show you the architecture of what we will be building with the help of the following posts.<\/p>\n<h2>Elastic Stack Architecture in Docker<\/h2>\n<p style=\"text-align: justify;\">Because we will be deploying a single node cluster, we decided that using \u00a0Docker Compose is the easiest way of formulating a full stack example. The below image, is a visual representation of the architecture used in our example.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" data-attachment-id=\"616\" data-permalink=\"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/introduction-to-elastic-stack\/docker_elastic_stack_architecture\/\" data-orig-file=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture.png\" data-orig-size=\"1326,820\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"CANCHITO-DEV: Elastic Stack architecture in Docker\" data-image-description=\"\" data-image-caption=\"&lt;p&gt;CANCHITO-DEV: Elastic Stack architecture in Docker&lt;\/p&gt;\n\" data-large-file=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture-1024x633.png\" class=\"aligncenter wp-image-616 size-full\" src=\"http:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture.png\" alt=\"CANCHITO-DEV: Elastic Stack architecture in Docker\" width=\"1326\" height=\"820\" srcset=\"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture.png 1326w, https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture-300x186.png 300w, https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture-1024x633.png 1024w, https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture-768x475.png 768w, https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/docker_elastic_stack_architecture-624x386.png 624w\" sizes=\"auto, (max-width: 1326px) 100vw, 1326px\" \/><\/p>\n<p style=\"text-align: justify;\">In different steps, we will deploy each component. The first thing will be to deploy <a href=\"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/deploying-elastic-stack-cluster-single-node-in-docker\/\">Elasticsearch and Kibana<\/a>. That way, we have a cluster ready for indexing and storaging data, including a visual tool.<\/p>\n<p style=\"text-align: justify;\">Next, to test this cluster, we will deploy <a href=\"http:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/31\/deploying-filebeat-in-docker\/\">Filebeat<\/a>. This first filebeat will read docker and system logs, and send them directly to Elasticsearch.<\/p>\n<p style=\"text-align: justify;\">As a third step, we will deploy <a href=\"http:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/16\/getting-started-with-logstash\/\">Logstash<\/a>. We will not complecate things, so we will send docker and system logs to it, with the help of a second Filebeat. The purpose is to see how they communicate with each other, and how Logstash can parse and enricht the logs.<\/p>\n<p>As a fourth container, we will launch <a href=\"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/25\/collect-metrics-with-metricbet\/\">Metricbeat<\/a>. With it, we will be monitoring both docker and system stats before sending them to Elasticsearch.<\/p>\n<p style=\"text-align: justify;\">To see the network traffic in our interfaces, we will deploy <a href=\"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/31\/capturing-the-network-traffic-with-packetbeat\/\">Packetbeat<\/a> in a fifth step.<\/p>\n<p style=\"text-align: justify;\">Continuing, to audit the activities of users and processes on our systems, detect changes to critical files, and identify potential security policy violations, we will deploy <a href=\"http:\/\/www.canchito-dev.com\/public\/blog\/2022\/02\/05\/audit-the-activities-of-users-and-processes-on-your-systems-with-auditbeat\">Auditbeat<\/a>.<\/p>\n<p style=\"text-align: justify;\">And finally, we will deploy <a href=\"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/30\/know-if-your-service-is-available-with-heartbeat\/\">Heartbeat<\/a> in a sixth and last container. We have represented this Beat as if it was attached to Elasticsearch, Kibana and Logstash containers. But in reality, it is an independent container monitoring other containers via <em>http<\/em>, <em>tcp<\/em> and <em>icmp<\/em>.<\/p>\n<p>We really hope you will enjoy deploying this architecture as much as we did. So, without more, let&#8217;s move on and will learn how to <a href=\"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/deploying-elastic-stack-cluster-single-node-in-docker\">deploy Elastic Stack cluster (single-node) using Docker<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hello friends! In this post, we will give you a small introduction to Elastic Stack including all the products that build it.<\/p>\n","protected":false},"author":1,"featured_media":565,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[94,88,85,95,86,87,96],"tags":[97,90,91,92,93],"class_list":["post-564","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-beat","category-elastic-stack","category-elasticsearch","category-filebeat","category-kibana","category-logstash","category-metricbeat","tag-beats","tag-elastic-stack","tag-elasticsearch","tag-kibana","tag-logstash"],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/stack-logo-color.png","jetpack_shortlink":"https:\/\/wp.me\/p8EwXo-96","jetpack-related-posts":[{"id":590,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/02\/elastic-stack-beats\/","url_meta":{"origin":564,"position":0},"title":"Introduction to Elastic Stack Beats","author":"canchitodev","date":"January 2, 2022","format":false,"excerpt":"How is information sent to Elasticsearch? The answer is simple, using Beats or Logstash. In this post, we will give a brief introduction to Beats.","rel":"","context":"In &quot;Auditbeat&quot;","block_context":{"text":"Auditbeat","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/elk\/auditbeat\/"},"img":{"alt_text":"CANCHITO-DEV: Beats basic diagram","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/beats_diagram.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/beats_diagram.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/beats_diagram.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/beats_diagram.png?resize=700%2C400 2x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/beats_diagram.png?resize=1050%2C600 3x"},"classes":[]},{"id":560,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/26\/deploying-elastic-stack-cluster-single-node-in-docker\/","url_meta":{"origin":564,"position":1},"title":"Deploying Elastic Stack Cluster (single node) in docker","author":"canchitodev","date":"December 26, 2021","format":false,"excerpt":"Let us show you how to deploy a single node Elastic Stack cluster using docker. Hope you find in it useful!","rel":"","context":"In &quot;Elastic Stack&quot;","block_context":{"text":"Elastic Stack","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/elk\/elastic-stack\/"},"img":{"alt_text":"CANCHITO-DEV: Kibana first time up","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/kibana-first-time-up.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/kibana-first-time-up.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/kibana-first-time-up.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/kibana-first-time-up.png?resize=700%2C400 2x"},"classes":[]},{"id":622,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/16\/getting-started-with-logstash\/","url_meta":{"origin":564,"position":2},"title":"Getting Started with Logstash","author":"canchitodev","date":"January 16, 2022","format":false,"excerpt":"In this post, we give a brief introduction to Logstash.","rel":"","context":"In &quot;Docker&quot;","block_context":{"text":"Docker","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/docker\/"},"img":{"alt_text":"CANCHITO-DEV: Kibana's Management > Stack Monitoring with Logstash","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/management_stack_monitoring_with_logstash-1024x673.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/management_stack_monitoring_with_logstash-1024x673.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/management_stack_monitoring_with_logstash-1024x673.png?resize=525%2C300 1.5x"},"classes":[]},{"id":647,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/31\/know-if-your-service-is-available-with-heartbeat\/","url_meta":{"origin":564,"position":3},"title":"Know if your Service is Available with Heartbeat","author":"canchitodev","date":"January 31, 2022","format":false,"excerpt":"Learn how Heartbeat periodically checks the status of your services and determine whether they are available. All within a dockerized enviroment.","rel":"","context":"In &quot;Beat&quot;","block_context":{"text":"Beat","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/elk\/beat\/"},"img":{"alt_text":"CANCHITO-DEV: [Heartbeat] HTTP Monitoring","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=700%2C400 2x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=1050%2C600 3x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/heartbeat_dashboard_http_monitoring.png?resize=1400%2C800 4x"},"classes":[]},{"id":588,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2021\/12\/31\/deploying-filebeat-in-docker\/","url_meta":{"origin":564,"position":4},"title":"Deploying Filebeat in docker","author":"canchitodev","date":"December 31, 2021","format":false,"excerpt":"Learn about Filebeat and how it interact with the rest of the Elastic Stack components while you deploy it using docker.","rel":"","context":"In &quot;Beat&quot;","block_context":{"text":"Beat","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/elk\/beat\/"},"img":{"alt_text":"CANCHITO-DEV: Filebeat Overview","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/filebeat_overview.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/filebeat_overview.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/filebeat_overview.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2021\/12\/filebeat_overview.png?resize=700%2C400 2x"},"classes":[]},{"id":636,"url":"https:\/\/www.canchito-dev.com\/public\/blog\/2022\/01\/27\/collect-metrics-with-metricbet\/","url_meta":{"origin":564,"position":5},"title":"Collect Metrics with Metricbet","author":"canchitodev","date":"January 27, 2022","format":false,"excerpt":"Learn how you could use Metricbeat to monitor your servers by collecting metrics from the system and services running on the server.","rel":"","context":"In &quot;Beat&quot;","block_context":{"text":"Beat","link":"https:\/\/www.canchito-dev.com\/public\/blog\/category\/elk\/beat\/"},"img":{"alt_text":"CANCHITO-DEV: [Metricbeat System] Host overview ECS","src":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=350%2C200 1x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=700%2C400 2x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=1050%2C600 3x, https:\/\/i0.wp.com\/www.canchito-dev.com\/public\/blog\/wp-content\/uploads\/2022\/01\/metricbeat_system_dashboard__host_overview_ecs.png?resize=1400%2C800 4x"},"classes":[]}],"jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/posts\/564","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/comments?post=564"}],"version-history":[{"count":8,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/posts\/564\/revisions"}],"predecessor-version":[{"id":673,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/posts\/564\/revisions\/673"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/media\/565"}],"wp:attachment":[{"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/media?parent=564"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/categories?post=564"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.canchito-dev.com\/public\/blog\/wp-json\/wp\/v2\/tags?post=564"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}