[ElasticSearch] Installation and Main Usage of Elasticsearch

William

Elasticsearch is a free to use open source search engine, created on Apache Lucene library. It can be installed on various platforms while using docker, deb, rpm, msi and tar.gz, .zip archives. In this guide we will perform installation with the help of RPM package manager. According to the official ElasticSearch documentation, recommendation for resources are only sensitive for RAM usage if looking for a production virtual server. 64 GB of RAM would be ideal environment for Elasticsearch to run, however 32 or 16 GB can also be used. 8 GB of RAM is also possible, however, with it multiple machines would be required. Talking about CPU, the system is not very sensitive, since it can run on 2-8 cores.

Requirements:

Linux VPS service with at least 8 GB of RAM. Recommendation at least 16 GB of RAM.

Installed Java 8.

Preparing for installation:

If you have not installed Java on the virtual server prior to this setup, you can perform the installation by running:

sudo yum install java-1.8.0-openjdk-devel 

Before starting the installation of Elasticsearch, it is required to download and install public signing key:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Installation

As we have introduced, the installation will be done while using RPM package manager on Linux 32 package with CentOS 7 installed. For the installation, firstly create a file in /etc/yum.repos.d/ directory named "elasticsearch.repo":

nano /etc/yum.repos.d/elasticsearch.repo

Add the following contents into the file:

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

When the file is saved, you can proceed with the installation by running:

sudo yum install elasticsearch

Configuration of Elasticsearch

By default Elasticsearch loads configuration from /etc/elasticsearch/elasticsearch.yml file. Here you can set path to where store data and logs, manage startup memory usage, set custom port and adjust network settings. By default, Elasticsearch uses 9200 port. For security reasons, it is recommended to uncomment "network.host" line and set its value as "localhost" and line "http.port:9200". You can open the file by running:

nano /etc/elasticsearch/elasticsearch.yml

network.host: localhost
http.port: 9200

Make sure you save the changes you have done to the configuration file.

Starting and stopping Elasticsearch

For Elasticsearch to be started automatically when the virtual server starts, run the following:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service

 

For starting and stopping the service manually, use the following:

sudo systemctl start elasticsearch.service

sudo systemctl stop elasticsearch.service

Kibana

Kibana is an analytics and visualization platform for Elasticsearch usage. It simplifies the management and understanding of your data. 

Installation of Kibana is done in similar way as Elasticsearch itself. Firstly, in /etc/yum.repos.d/ create "kibana.repo" file:

nano /etc/yum.repos.d/kibana.repo

Add the following contents into the file:

[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

After saving the file, Kibana can be installed by running:

sudo yum install kibana

When installation is completed, you can set Kibana to start automatically when the VPS starts:

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable kibana.service

Configuration of Kibana

Kibana has a separate configuration file that is located at "/etc/kibana/kibana.yml". Open this file and uncomment the following lines to allow connections over localhost:

server.port: 5601

server.host: "localhost"

elasticsearch.hosts: ["http://localhost:9200"]

After adjusting the configuration file, enable Kibana service:

sudo systemctl start kibana.service

sudo systemctl stop kibana.service

Since Kibana can be access only through localhost, further configuration will be done on Nginx to allow proxy access. For this we will have to install Nginx.

Nginx Installation

If Nginx is not installed, perform the installation by running:

sudo yum install epel-release

sudo yum install nginx httpd-tools

Create a new virtual host configuration file in Nginx conf.d directory, with adjusting "your-domain.com.conf" to your used domain:

nano /etc/nginx/conf.d/your-domain.com.conf

Enter the following contents into the file (again, adjust "your-domain.com" to your used domain:

server {
    listen 80;
 
    server_name your-domain.com;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.kibana;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

For Nginx to start after the virtual server boots and start it at the moment use:

sudo systemctl enable nginx
sudo systemctl start nginx

Make sure you restart your nginx so that all previous configurations would take effect. This can be done by running:

sudo systemctl restart nginx

After setting up Nginx, you will have to create admin user with a password. Adjust the following command and set your own wanted user instead of "kibanat4vps". Recommendation would be not to use only "kibana" since the access should be secure:

echo "kibanat4vps:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.kibana

After entering this command, you will have to enter wanted password for the user and verify it.

Logstash Installation

For processing data before it is uploaded to Elasticsearch, recommendation is to use Logstash. Installation is done by running:

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.rpm
rpm -ivh logstash-6.2.4.rpm

After the installation is completed, the service can be started and set to start automatically on virtual server boot by running:

systemctl restart logstash
systemctl enable logstash

Connecting to Kibana

If all configuration was done properly, the access to Kibana should be available over the domain you have used during Nginx configuration. After accessing your domain, you would be requested to enter username and password that was generated at the very end of Nginx configuration section.

When logged in, if you would like to upload your own data, you can choose from the following options:

In our example we will use already prepared data template of Flights that is offered during first login after the installation is completed.

After accessing to the Kibana dashboard, you will find main sections:

• Search field at the top of the page with filtering options;
• Time selector on the right of the Search field, that allows setting your wanted time strips;
• Visualization field at the bottom of the Search field that shows data related to the set Search and Time details;
• Application column located on the justify, provides you with general options on Kibana management.
  

Each of the visualization elements have control options at the top right and bottom justifycorners. For your convenience, each of the elements can be re-located by using "Edit" option located on top of "Search" field. After adjusting the elements into wanted position, you can save the made changes by using "Save" option that will appear at the top justify corner or remove the changes by using "Cancel" option. If you want to save current dashboard before making any adjustments, good way to do so is by using "Clone" option to set yourself a backup of current dashboard.

Application column:

• Discover - gives access for exploring the data. Through here you can reach each document of your index that matches requested pattern and details of it.
• Visualize - simplifies the creation of visual data interpretation, by choosing wanted type of visualization and assigning specific index for it.
  
• Dashboard - displays the collection of your visualizations.
  
• Canvas - tool for generating visual adds to the selected data with additional colors, shapes and text.
  
• Maps - allows data analysis based on the geographical details of the collected data.
  
• Machine Learning - used when the data scales and manual data maintenance is no longer possible, that analyzes the data for you.
  
• Infrastructure - provides UI to monitor the infrastructure you have created.
  
• Logs - logs that provides detailed information on servers, containers and services.
  
• APM - (Application Performance Monitoring) tool that collects performance metrics and errors of your applications.
  
• Uptime - a monitoring tool for network. Requires configuration of Heartbeat.
  
• Dev Tools - contains console, Search profiler and Grok Debugger tools for your interaction with Kibana.
  
• Stack Monitoring - Visualizes monitoring data of whole Elastic Stack and monitors Kibana itself.
  
• Management - General settings section, for regular adjustments of Kibana.
  

For a more detailed description of usage of the Kibana tool on your virtual server, due to its wide application possibilities, our recommendation would be to check official Kibana documentation.