How to Install Elasticsearch on Ubuntu 20.04
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.
Elasticsearch is built on Apache Lucene and was first released in 2010 by Elasticsearch N.V. (now known as Elastic). Known for its simple REST APIs, distributed nature, speed, and scalability, Elasticsearch is the central component of the Elastic Stack, a set of free and open tools for data ingestion, enrichment, storage, analysis, and visualization.
Commonly referred to as the ELK Stack (after Elasticsearch, Logstash, and Kibana), the Elastic Stack now includes a rich collection of lightweight shipping agents known as Beats for sending data to Elasticsearch.
What is Elasticsearch used for? The speed and scalability of Elasticsearch and its ability to index many types of content mean that it can be used for a number of use cases:
- Application search
- Logging and log analytics
- Infrastructure metrics and container monitoring
- Website search
- Enterprise search
- Application performance monitoring
- Geospatial data analysis and visualization
- Security analytics
- Business analytics
We'll start by describing the environment, then we’ll walk through how each component is installed, and finish by configuring our sandbox server to send its system logs to Logstash and view them via Kibana.
Installing Elasticsearch
Elasticsearch is a widely used database and a search server, and it’s the main component of the ELK setup.
Elasticsearch’s benefits include:
Easy installation and use
A powerful internal search technology (Lucene)
A RESTful web interface
The ability to work with data in schema-free JSON documents (NoSQL)
Open-source
There are various ways to install Elasticsearch but we will be using DEB packages.
First, we need java on an ubuntu machine
Installing Java
Elasticsearch is a Java application, so the first step is to install Java.
Run the following as root or user with sudo privileges command to install the OpenJDK package:
sudo apt install default-jdk Verify that Java is installed:
java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-8u191-b12-2ubuntu0.16.04.1-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)Install the apt-transport-https package
sudo apt-get install apt-transport-https Then use wget command to download the deb package for elasticsearch-7.15
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-amd64.deb once downloaded, use the following command to install it
sudo dpkg -i elasticsearch-7.15.0-amd64.deb This results in Elasticsearch being installed in /usr/share/elasticsearch/ with its configuration files placed in /etc/elasticsearch and its init script added in /etc/init.d/elasticsearch.
To make sure Elasticsearch starts and stops automatically with the server, add its init script to the default run level's.
sudo systemctl enable elasticsearch.service Configuring Elasticsearch
Now that Elasticsearch and its Java dependencies have been installed, it is time to configure Elasticsearch. The Elasticsearch configuration files are in the /etc/elasticsearch directory. There are two files:
elasticsearch.yml configures the Elasticsearch server settings. This is where all options, except those for logging, are stored, which is why we are mostly interested in this file.
logging.yml provides configuration for logging. In the beginning, you don’t have to edit this file. You can leave all default logging options. You can find the resulting logs in /var/log/elasticsearch by default. The first variables to customize on any Elasticsearch server are node. name and cluster.name in elasticsearch.yml. As their names suggest, node.name specifies the name of the server (node) and the cluster to which the latter is associated.
Open the Elasticsearch configuration file at: /etc/elasticsearch/elasticsearch.yml, and apply the following configurations:
http.port: 9200
If you don't customize these variables, a node.name will be assigned automatically in respect to the Droplet hostname. The cluster.name will be automatically set to the name of the default cluster.
The cluster.name value is used by the auto-discovery feature of Elasticsearch to automatically discover and associate Elasticsearch nodes to a cluster. Thus, if you don’t change the default value, you might have unwanted nodes, found on the same network, in your cluster.
Configure Remote Access
In some cases when we hit URL in the browser, the http://localhost:9200 is not opened, for that, we need to configure the following in That's it. Elasticsearch has been installed on your Ubuntu 20.04 server.
To do so, open the elasticsearch.yml configuration file:
sudo nano /etc/elasticsearch/elasticsearch.yml Search for the line that contains network.host, uncomment it, and change the value to 0.0.0.0:
network.host: 0.0.0.0 other property we can configure as,
cluster.name: elasticsearch
node.name: "My First Node"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch Restart the Elasticsearch service for the changes to take effect:
sudo systemctl restart elasticsearch Testing Elasticsearch
To verify that Elasticsearch is running on port 9200, use curl to send an HTTP request to port 9200 on localhost:
curl -X GET "localhost:9200/"
{
"name" : "ip-172-31-39-56",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "2mAj2hr1Te-Vf9Sqcr_a_A",
"version" : {
"number" : "7.15.0",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "79d65f6e357953a5b3cbcc5e2c7c21073d89aa29",
"build_date" : "2021-09-16T03:05:29.143308416Z",
"build_snapshot" : false,
"lucene_version" : "8.9.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
It may take 5-10 seconds for the service to start. If you get the error "Failed to connect to localhost port 9200: Connection refused", wait for a few seconds and try again.
To view the messages logged by the Elasticsearch service, use the following command:
sudo journalctl -u elasticsearch Using Elasticsearch
To start using Elasticsearch, let’s add some data first. As already mentioned, Elasticsearch uses a RESTful API, which response to the usual CRUD commands: create, read, update, and delete. For working with it, we’ll use again curl.
You can add your first entry with the command:
curl -X POST 'http://localhost:9200/tutorial/helloworld/1' -d '{ "message": "Hello World!" }' You should see the following response:
Output
{"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"_shards":{"total":2,"successful":1,"failed":0},"created":true}
With curl, we have sent an HTTP POST request to the Elasticsearch server. The URI of the request was /tutorial/helloworld/1 with several parameters: the tutorial is the index of the data in Elasticsearch. helloworld is the type.
1 is the id of our entry under the above index and type. You can retrieve this first entry with an HTTP GET request.
curl -X GET 'http://localhost:9200/tutorial/helloworld/1' The result should look like this:
Output
{"_index":"tutorial","_type":"helloworld","_id":"1","_version":1,"found":true,"_source":{ "message": "Hello World!" }}
To modify an existing entry, you can use an HTTP PUT request.
curl -X PUT 'localhost:9200/tutorial/helloworld/1?pretty' -d '
{
"message": "Hello People!"
}'
Elasticsearch should acknowledge successful modification like this:
Output
{
"_index" : "tutorial",
"_type" : "helloworld",
"_id" : "1",
"_version" : 2,
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"created" : false
}
Installing Logstash
Logstash is an open-source tool that collects, parses, and stores logs for future use and makes rapid log analysis possible. Logstash is useful for aggregating logs from multiple sources, like a cluster of Docker instances, and parsing them from text lines into a structured format such as JSON. In the ELK Stack, Logstash uses Elasticsearch to store and index logs.
Install Logstash with:
sudo apt-get install logstash Create a Logstash configuration file:
sudo vim /etc/logstash/conf.d/apache-01.conf Enter the following configuration:
input {
file {
path => "/home/ubuntu/apache-daily-access.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
} This file is telling Logstash to collect the local /home/ubuntu/apache-daily-access.log file and send it to Elasticsearch for indexing.
Finally, start Logstash to read the configuration:
sudo service logstash start To make sure the data is being indexed, use:
sudo curl -XGET 'localhost:9200/_cat/indices?v&pretty'You should see your new Logstash index created:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .geoip_databases hX01XsLKS9-C1dNAzDC4CQ 1 0 43 0 40.7mb 40.7mb
green open .apm-custom-link ZpgWxjIzRHWLuUcEbvaW4g 1 0 0 0 208b 208b
yellow open logstash-2021.09.23-000001 yoEJww8ARKurj9u0t5gINw 1 1 14 0 44kb 44kb
yellow open %services-2021.09.23 _R8Z4kcmTo-mVmuZoirfuA 1 1 1 0 7.4kb 7.4kb
green open .apm-agent-configuration Qniu8b7RRimw5ia4euO0kg 1 0 0 0 208b 208b
green open .kibana_task_manager_7.14.2_001 u_G0TOVWQ2CLycHJNVW4eA 1 0 15 27401 2.8mb 2.8mb
green open .kibana_7.14.2_001 TPIIkn4HSbS5NfV9Gpeisw 1 0 2562 48 3.6mb 3.6mb
yellow open metricbeat-7.14.2-2021.09.23-000001 Q4HCzslsTFm-ylleCLPWFw 1 1 41741 0 34.2mb 34.2mb
yellow open mservices-2021.09.23 f-7lFA3LRhOTUgNsdrN4GQ 1 1 1 0 7.4kb 7.4kb
green open .tasks MHg_3GIWQ9WKg_cYa0GFKw 1 0 6 0 23kb 23kb
green open .kibana-event-log-7.14.2-000001 yrnTYvaERcGFhPxOAsYtXA 1 0 7 0 17.5kb 17.5kb
Installing Kibana Ubuntu 20.04
Kibana is an open-source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster.
Users can create bar, line, and scatter plots; pie charts; and maps on top of large volumes of data.
Among other uses, Kibana makes working with logs super easy and even fun, and its graphical web interface lets beginners execute powerful log searches.
To install Kibana, use this command:
sudo apt-get install kibana Open the Kibana configuration file and enter the following configurations:
sudo nano /etc/kibana/kibana.yml
put the following statements in the file.
server.port: 5601
server.host: "localhost"
elasticsearch.hosts: ["http://localhost:9200"]
Start Kibana:
sudo service kibana start
To check in your browser, open http://localhost:5601 after Kibana is started, this may take a few minutes.
Post/Questions related to How To Install and Configure Elasticsearch on Ubuntu 20.04
How to set username and password for Elastic Search and Kibana on Ubuntu
In this article, we have seen How To Install and Configure Elasticsearch on Ubuntu 20.04.
0 Comments
Post a Comment