8.1 Setting Up an Index for Docker Logs
For demonstration purposes, I created a Docker image with ELK and Filebeat, which includes a test program to generate logs. In this example, we’ll focus on setting up an index for Docker logs in Kibana.
Step 1: Clone the Repo and Start the ELK Stack
Download the repository with Docker Compose configurations for the ELK stack and launch it:
git clone https://github.com/et-soft/habr-elk
cd habr-elk
docker compose up
For convenience, the -d key is omitted so you can see the ELK stack starting process. It might take a few minutes. After a successful launch, you should see the following log entry:
{"type":"log","@timestamp":"2020-09-20T05:55:14Z","tags":["info","http","server","Kibana"],
"pid":6,"message":"http server running at http://0:5601"}
Step 2: Accessing Kibana
Open a web browser and go to http://localhost:5601.
Step 3: Setting Up the Log Index
In the left menu of Kibana, choose the Discover section to navigate to the index creation page.
Step 4: Creating a New Index
Click the Create index pattern button to open the setup form. In the Index pattern name field, enter logstash-*. If the setup is correct, Kibana will show indices that match the rule.
Step 5: Choosing the Time Field
On the next page, select the primary field with a timestamp — @timestamp.
Step 6: Verifying the Index
After completing the setup, you’ll see the index settings page. No further actions are required at this moment.
Step 7: Viewing the Logs
Go back to the Discover section, where you’ll see log entries from Docker containers.
8.2 Creating Dashboards for Log Visualization
Step 1. Create dashboards for log visualization
In the left menu, click on the "Dashboard" section — this will take you to the page where dashboards can be created.
Step 2. Tap on "Create new dashboard," and you'll be taken to a screen where you can add objects to this very Dashboard.
Step 3. Click on the "Create new" button, and Kibana will ask you to choose how to display the data. There are tons of options, but for the sake of example, let’s dig into two — graphs in the "Vertical Bar" style and tabular representation "Data Table." Everything else is set up similarly.
8.3 Logs Histogram
Let’s make a histogram with the "Vertical Bar" type as an example—it’ll show the ratio of successful and failed service requests. After setting it up, it’ll look something like this:
Requests with response codes < 400 are considered successful, and everything >= 400 is marked as problematic.
First, we need to select a data source for the chart. Let’s use the Index Pattern we created earlier.
Once the source is selected, a single general chart appears by default. Let’s tweak it.
In the "Buckets" section, click "Add," select "X-axis," and configure the X-axis. Time stamps will be plotted along it. In the "Aggregation" field, choose "Date Histogram," and in the "Field" field, specify "@timestamp." The "Minimum interval" field can stay at "Auto."
Click "Update," and the chart will display the number of requests every 30 seconds.
Now let’s configure the Y-axis columns. At the moment, it shows the total number of requests for a time interval. Let’s change the aggregation to "Sum Bucket" to add both successful and problematic requests. In the Bucket → Aggregation section, select "Filters" and set the filter to "statusCode >= 400." In the "Custom label" field, write a clear label for the metric.
Click "Update," and the chart will display only problematic requests.
Want to change the column colors? Click on the circle in the chart legend, and a window will appear to pick a color.
Now let’s add data for successful requests. In the "Metrics" section, click "Add" and select "Y-axis." Configure it in the same way, but for the filter, specify "statusCode < 400."
Change the color for the new column, and now we have a visualization of the ratio between successful and problematic requests.
Don’t forget to save the chart by clicking "Save" at the top of the screen. Give it a name, and it’ll appear on the Dashboard.
8.4 Logs Table
Let's create a "Data Table," to see the list of all URLs that had requests and their count. We'll start, just like with Vertical Bar, by selecting the data source.
After choosing the source, a table with one column will appear on the screen, showing the total number of requests for the selected time interval.
Now let's configure the table. Go to the "Buckets" block, click "Add," and select "Split rows."
In the "Aggregation" field that appears, select "Terms," and in "Field," specify "url.keyword."
In the "Custom label" field, enter a readable name, like "Url." Then click "Update" — and our table is ready. It will show the count of requests to each URL for the selected time interval.
To save the table, click "Save" at the top of the screen, enter a name (like, "Urls"), and return to the Dashboard. Now we see both of our created visualizations.
GO TO FULL VERSION