7.1 ELK Stack Components
ELK Stack, made up of Elasticsearch, Logstash, and Kibana, is a powerful tool for centralized logging and data analysis. It lets you collect, store, analyze, and visualize logs from different sources, including Docker containers. In this lecture, we'll take a detailed look at how to set up ELK Stack for centralized logging of Docker containers.
ELK Stack Components
- Elasticsearch — a distributed search and analytics system used for storing and searching logs.
- Logstash — a tool for collecting, processing, and forwarding logs to Elasticsearch.
- Kibana — a web interface for visualizing and analyzing data stored in Elasticsearch.
7.2 Installing Elasticsearch
On Linux
1. Adding the Elasticsearch repository
Download and install the public repository key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Add the Elasticsearch repository:
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > \
/etc/apt/sources.list.d/elastic-7.x.list'
sudo apt-get update
2. Installing Elasticsearch
sudo apt-get install elasticsearch
3. Starting and enabling auto-start for Elasticsearch
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
On MacOS
1. Installing via Homebrew
brew update
brew install elasticsearch
2. Starting Elasticsearch
elasticsearch
On Windows
1. Downloading the installation file
Download the latest version of Elasticsearch from the official website: https://www.elastic.co/downloads/elasticsearch.
2. Installing Elasticsearch
Run the downloaded installer and follow the on-screen instructions.
3. Starting Elasticsearch
After installation, navigate to the Elasticsearch folder and start the service via PowerShell:
.\bin\elasticsearch.bat
To enable auto-start for Elasticsearch, add it to Windows services via services.msc.
7.3 Installing Logstash
On Linux
1. Installing Logstash
Install Logstash from the Elastic repository:
sudo apt-get install logstash
2. Configuring Logstash
Create the configuration file logstash.conf:
sudo nano /etc/logstash/conf.d/logstash.conf
Example configuration for collecting Docker logs:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "docker-logs-%{+YYYY.MM.dd}"
}
}
3. Starting and enabling auto-start for Logstash
Start and enable auto-start for Logstash:
sudo systemctl start logstash
sudo systemctl enable logstash
On MacOS
1. Installing Logstash via Homebrew
brew update
brew install logstash
2. Configuring Logstash
Create the configuration file logstash.conf in the Logstash configuration folder:
nano /usr/local/etc/logstash/logstash.conf
Example configuration:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "docker-logs-%{+YYYY.MM.dd}"
}
}
3. Starting Logstash
Start Logstash via Homebrew:
logstash -f /usr/local/etc/logstash/logstash.conf
On Windows
1. Downloading and Installing Logstash
Download the latest version of Logstash from the official website: https://www.elastic.co/downloads/logstash.
Extract the Logstash archive to a convenient location (e.g., C:\logstash).
2. Configuring Logstash
Create the configuration file logstash.conf in the Logstash configuration folder:
notepad C:\logstash\config\logstash.conf
Example configuration:
input {
beats {
port => 5044
}
}
filter {
json {
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "docker-logs-%{+YYYY.MM.dd}"
}
}
3. Starting Logstash
Start Logstash with the configuration:
cd C:\logstash
bin\logstash.bat -f config\logstash.conf
7.4 Installing Kibana
On Linux
1. Installing Kibana
Install Kibana:
sudo apt-get install kibana
2. Starting and Enabling Kibana to Auto-start
Start and enable auto-start for Kibana:
sudo systemctl start kibana
sudo systemctl enable kibana
Kibana will be available at http://localhost:5601.
On MacOS
1. Installing Kibana via Homebrew
brew update
brew install kibana
2. Starting Kibana
Start Kibana via Homebrew:
brew services start kibana
Kibana will be available at http://localhost:5601.
On Windows
1. Downloading and Installing Kibana
Download the latest version of Kibana from the official website: https://www.elastic.co/downloads/kibana.
Unpack the Kibana archive to a convenient location (for example, C:\kibana).
2. Starting Kibana
Start Kibana:
cd C:\kibana\bin
kibana.bat
Kibana will be available at http://localhost:5601.
8.5 Configuring Filebeat to Collect Docker Logs
On Linux
1. Installing Filebeat
Install Filebeat from the Elastic repository:
sudo apt-get install filebeat
2. Configuring Filebeat
Configure Filebeat to collect Docker logs:
sudo nano /etc/filebeat/filebeat.yml
Configuration example:
filebeat.inputs:
- type: docker
containers.ids:
- '*'
processors:
- add_docker_metadata: ~
output.logstash:
hosts: ["localhost:5044"]
3. Starting and Enabling Auto-Start for Filebeat
Start and enable auto-start for Filebeat:
sudo systemctl start filebeat
sudo systemctl enable filebeat
On MacOS
1. Installing Filebeat via Homebrew
brew update
brew install filebeat
2. Configuring Filebeat
Configure Filebeat by editing the filebeat.yml file:
nano /usr/local/etc/filebeat/filebeat.yml
Configuration example:
filebeat.inputs:
- type: docker
containers.ids:
- '*'
processors:
- add_docker_metadata: ~
output.logstash:
hosts: ["localhost:5044"]
3. Starting Filebeat
Start Filebeat using Homebrew:
brew services start filebeat
On Windows
1. Downloading and Installing Filebeat
Download the latest version of Filebeat from the official website: https://www.elastic.co/downloads/beats/filebeat.
Unpack the Filebeat archive into a convenient location (e.g., C:\filebeat).
2. Configuring Filebeat
Edit the filebeat.yml file:
notepad C:\filebeat\filebeat.yml
Configuration example:
filebeat.inputs:
- type: docker
containers.ids:
- '*'
processors:
- add_docker_metadata: ~
output.logstash:
hosts: ["localhost:5044"]
3. Starting Filebeat
Start Filebeat:
cd C:\filebeat
filebeat.exe -e -c filebeat.yml
GO TO FULL VERSION