July Release Confetti
150+ New Courses, Hands‑On Labs, And
Interactive Learning Activities
Learn More

ELK Stack 5.0 Installation and configuration. Part 1 - VirtualBox, Network, JDK8, Elasticsearch, Configuration check, Logs

Introduction

This guide covers ELK Stack 5.0 Installation and configuration inside virtual lab, based on Oracle Virtual Box with Internal NAT Network between hosts.


Looking for part 2 on how to install Kibana and Filebeat? Click here.

Getting Started

We will start from Initial Virtual Box hosts configuration, needed for this guide, configure network interfaces and continue with ELK components installation and configuration (Oracle JDK 8, Elasticsearch, Kibana, Filebeat)


Sources / Resources


https://www.virtualbox.org

https://www.centos.org/

http://www.oracle.com/technetwork/java/javase/documentation/index.html

https://www.elastic.co/



Initial Virtual Box configuration

Our Lab Servers:

1. ELK Master 1 CentOS – Elasticsearch, Kibana, Filebeat

user_5256_582393c521605.png


Please take into account that the current setup was tested with 6GB of RAM for ELK Master 1 virtual host. In case you will use less – elasticsearch errors and crashes are possible. Also, you should be aware of the official hardware requirements for elasticsearch – 64 GB of RAM (on hardware node) with minimal recommended amount of RAM being 16 GB. Hardware hosts with less than 8 GB tends to be counterproductive, because of the high probability of usage of swap, and sub-optimal operation with low amount of memory. Still, for our test cluster, 6 GB will be fine.



2. ELK Slave 1 CentOS – Filebeat

user_5256_582395ef1b9e0.png


Note: In case you will need to use elasticsearch with a small amount of memory (for example, in virtual lab) – you should look closer at Heap Size ($ES_HEAP_SIZE) tuning, Fielddata Size and Fielddata Circuit Braker settings.


For ELK Slave 1 host we will use 2 GB of RAM.

user_5256_58239647169ec.png


user_5256_582396845efb5.png


Note that for both servers (ELK Master 1 and ELK Slave 1) we used 8 GB virtual HDD VDI. This is the recommended setting for current lab, but it’s up to you, to give more space if you plan on playing with this lab more and (hopefully) expand it with our next guides.


Virtual Box has many possibilities for network interconnection layer emulation, and, though this guide is not a networking subject, I strongly recommend you to read more about this subject:


https://www.virtualbox.org/manual/ch06.html



In short, with current network configuration (NAT Network) all hosts will achieve a few important points at the same time:

  • All hosts will have access to Internet via NAT (via your host OS)
  • All hosts will be in a separate, independent network, that does not interfere with your current network connections
  • We will be able to use direct connections via port forwarding. In this way we will have a comfortable way to work with virtual servers (ssh, web browser, etc)

Open Virtual Box menu - File – Preferences - Network


user_5256_5823974fe0223.png


user_5256_58239767af4c1.png


user_5256_5823978c3b001.png


user_5256_582397ad123bc.png


Now, you can install Centos 7 Minimal system on ELK Master 1 and ELK Slave 1 lab servers.


You can download Centos 7 from official site:


https://www.centos.org/download/


We will also create port forwarding rules in File – Preferences – Network – Port Forwarding. This will give us the possibility to use our favorite terminal emulator with virtual servers. You can use any program with which you are comfortable: putty, mremoteng, ZOC in case you work from windows, or any terminal emulator in Linux (gnome-terminal, Konsole, etc).


Tip:

In case of Linux I strongly suggest to use tmux terminal emulator (or screen).


https://tmux.github.io/



user_5256_5823981fbffcd.png


ELK Master 1:


Name: SSH (any arbitrary unique name)
Protocol: TCP
Host IP: 127.0.0.1
Host Port: 222 (or any unused port higher than 1024)
Guest IP: IP address of the guest VM
Guest Port: 22 (SSH port)


ELK Slave 1:


Name: SSH (any arbitrary unique name)
Protocol: TCP
Host IP: 127.0.0.1
Host Port: 223 (or any unused port higher than 1024)
Guest IP: IP address of the guest VM
Guest Port: 22 (SSH port)


Important note:

Remember, that we have to set up and use NAT Network. So, in case you just installed CentOS 7 as in our guide – you will need to configure your network adapter settings.

You are free to use any tool with which you feel comfortable: nmtui, nmcli or ip.

Tip:

While nmtui provides the “least effort” way of configuring network settings, I recommend to get familiar with the Network Manager console configurator – nmcli, as well as with general Linux distribution-independent ways like ip (replacement for ifconfig).

Now you are able to connect two lab servers from host system, using your favourite ssh client (putty, as example). 


user_5256_58239a6be95cf.png



Network adapters on both servers are already configured with default DHCP settings.


ELK Master 1:


SSH] Server Version OpenSSH_6.6.1
[SSH] Logged in (password)

Last login: Mon Oct 31 07:57:52 2016 from 10.0.2.2
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:29:8f:54 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.4/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 978sec preferred_lft 978sec
inet6 fe80::a00:27ff:fe29:8f54/64 scope link
valid_lft forever preferred_lft forever
[root@localhost ~]#


ELK Slave 1:


[SSH] Server Version OpenSSH_6.6.1
[SSH] Logged in (password)

Last login: Mon Oct 31 07:55:50 2016
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 08:00:27:73:f5:e8 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.5/24 brd 10.0.2.255 scope global dynamic enp0s3
valid_lft 1087sec preferred_lft 1087sec
inet6 fe80::a00:27ff:fe73:f5e8/64 scope link
valid_lft forever preferred_lft forever
[root@localhost ~]#

Now, before static network configuration, we will configure hostnames on ELK Master 1 and ELK Slave 1

Hostname changes – these are not only important steps towards lab server configurations, but they are also needed for ELK components to work correctly, and they allow you to clearly organize and categorize your network servers and any other network equipment.

Tip:

Be sure to use descriptive host names.


ELK Master 1:


[root@localhost ~]# hostnamectl set-hostname elkmaster1
[root@localhost ~]# hostnamectl status
Static hostname: elkmaster1
Icon name: computer-vm
Chassis: vm
Machine ID: 0a69fb29d55a4a4da2d671b30d497d23
Boot ID: f234f56e093c4bfd9206a3c333955651
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-327.el7.x86_64
Architecture: x86-64
[root@localhost ~]#


ELK Slave 1:

[root@localhost ~]# hostnamectl set-hostname elkslave1
[root@localhost ~]# hostnamectl status
Static hostname: elkslave1
Icon name: computer-vm
Chassis: vm
Machine ID: b0d839a6c9ee40db8910b0fdf355f300
Boot ID: b3a6967769954fa5899e233964bd1d23
Virtualization: kvm
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-327.el7.x86_64
Architecture: x86-64
[root@localhost ~]#


As this guide is about ELK Stack, and not about Linux networking, we will use simplest way to configure network in CentOS 7 from users point of view, using nmtui tool (part of NetworkManager).

We will configure network interfaces to use static IP.



ELK Master 1

On elkmaster1 host run:


[root@elkmaster1 ~]# nmtui

Edit a connection – Enter – Enter

user_5256_58239ba9d4264.png



We will edit default connection - enp0s3


user_5256_58239bdf298e0.png


You will need to set next settings for elkmaster1:

  • IPv4 Configuration: Manual
  • Addresses: 10.0.2.4/24
  • Gateway: 10.0.2.1
  • DNS servers (Google Public DNS): 8.8.8.8, 8.8.4.4
  • IPv6 Configuration: Ignore
  • Check: Automatically connect
  • Check: Availabale to all users
user_5256_58239c37ba419.png


OK – Quit


You can restart elkmaster1 and check that all network settings applied:


[root@elkmaster1 ~]# nmcli dev show enp0s3
GENERAL.DEVICE: enp0s3
GENERAL.TYPE: ethernet
GENERAL.HWADDR: 08:00:27:29:8F:54
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: enp0s3
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER: on
IP4.ADDRESS[1]: 10.0.2.4/24
IP4.GATEWAY: 10.0.2.1
IP4.DNS[1]: 8.8.8.8
IP4.DNS[2]: 8.8.4.4
IP6.ADDRESS[1]: fe80::a00:27ff:fe29:8f54/64
IP6.GATEWAY:
[root@elkmaster1 ~]#


ELK Slave 1

On elkslave1 host run:


[root@elkslave1 ~]# nmtui

Edit a connection – Enter – Enter


user_5256_58239c9b4d937.png


We will edit default connection - enp0s3

user_5256_58239cc53c57f.png


You will need to set next settings for elkslave1:

  • IPv4 Configuration: Manual
  • Addresses: 10.0.2.5/24
  • Gateway: 10.0.2.1
  • DNS servers (Google Public DNS): 8.8.8.8, 8.8.4.4
  • IPv6 Configuration: Ignore
  • Check: Automatically connect
  • Check: Availabale to all users

user_5256_58239d0a30c50.png


OK – Quit


You can restart elkslave1 and check that all network settings applied:


[root@elkslave1 ~]# nmcli dev show enp0s3
GENERAL.DEVICE: enp0s3
GENERAL.TYPE: ethernet
GENERAL.HWADDR: 08:00:27:73:F5:E8
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: enp0s3
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/0
WIRED-PROPERTIES.CARRIER: on
IP4.ADDRESS[1]: 10.0.2.5/24
IP4.GATEWAY: 10.0.2.1
IP4.DNS[1]: 8.8.8.8
IP4.DNS[2]: 8.8.4.4
IP6.ADDRESS[1]: fe80::a00:27ff:fe73:f5e8/64
IP6.GATEWAY:
[root@elkslave1 ~]#


At the end, elkmaster1 should be able to reach elkslave1 and vice versa:


[root@elkmaster1 ~]# ping -c 3 10.0.2.5
PING 10.0.2.5 (10.0.2.5) 56(84) bytes of data.
64 bytes from 10.0.2.5: icmp_seq=1 ttl=64 time=0.301 ms
64 bytes from 10.0.2.5: icmp_seq=2 ttl=64 time=0.452 ms
64 bytes from 10.0.2.5: icmp_seq=3 ttl=64 time=0.434 ms

--- 10.0.2.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.301/0.395/0.452/0.071 ms
[root@elkmaster1 ~]#


JDK 8 Installation on elkmaster1

Here, we will refer to official Elastic requirements for JVM: 


https://www.elastic.co/support/matrix#show_jvm


Check your OS version to know which packages you will need to download (x86 or x86_64):


[root@elkmaster1 ~]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
[root@localhost ~]# uname -a
Linux elkslave1 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@elkmaster1 ~]#


Install wget:


[root@elkmaster1 ~]# yum -y install wget

Download the latest available Oracle JDK 8:


[root@elkmaster1 ~]# wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm"
--2016-10-31 08:58:17-- http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm
Resolving download.oracle.com (download.oracle.com)... 77.222.148.106, 77.222.148.105
Connecting to download.oracle.com (download.oracle.com)|77.222.148.106|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://edelivery.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm [following]
--2016-10-31 08:58:17-- https://edelivery.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm
Resolving edelivery.oracle.com (edelivery.oracle.com)... 104.81.108.164, 2a02:26f0:f:290::2d3e, 2a02:26f0:f:284::2d3e
Connecting to edelivery.oracle.com (edelivery.oracle.com)|104.81.108.164|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm?AuthParam=1477918827_3ea9f75fa90cd680fb8dfe3efac8db9f [following]
--2016-10-31 08:58:25-- http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.rpm?AuthParam=1477918827_3ea9f75fa90cd680fb8dfe3efac8db9f
Connecting to download.oracle.com (download.oracle.com)|77.222.148.106|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 166040563 (158M) [application/x-redhat-package-manager]
Saving to: ‘jdk-8u111-linux-x64.rpm’

100%[=========================================================================================================================================================================>] 166,040,563 3.78MB/s in 31s

2016-10-31 08:58:57 (5.03 MB/s) - ‘jdk-8u111-linux-x64.rpm’ saved [166040563/166040563]

[root@elkmaster1 ~]#


Check that the file was downloaded


[root@elkmaster1 ~]# ls -lah jdk*
-rw-r--r--. 1 root root 159M Sep 23 15:16 jdk-8u111-linux-x64.rpm
[root@elkmaster1 ~]#


To check the control sum, go to: 


https://www.oracle.com/webfolder/s/digest/8u111checksum.html 


or use this command:


curl -s "https://www.oracle.com/webfolder/s/digest/8u111checksum.html" | awk '/jdk-8u111-linux-x64.rpm/ {print $2,$3}' | sed 's/<\/br>//'

[root@elkmaster1 ~]# curl -s "https://www.oracle.com/webfolder/s/digest/8u111checksum.html" | awk '/jdk-8u111-linux-x64.rpm/ {print $2,$3}' | sed 's/<\/br>//'
sha256: ca27da3b467547400698ae6c7b8aab1c8830e7bd36e1cc06f61c6e9dbcababf7
[root@elkmaster1 ~]#


Now, after you know the correct md5 and sha256 sums, check the downloaded file and compare:


[root@elkmaster1 ~]# sha256sum jdk-8u111-linux-x64.rpm
ca27da3b467547400698ae6c7b8aab1c8830e7bd36e1cc06f61c6e9dbcababf7 jdk-8u111-linux-x64.rpm
[root@elkmaster1 ~]#


Install the downloaded file:


[root@elkmaster1 ~]# rpm -ivh jdk-8u111-linux-x64.rpm
Preparing... ################################# [100%]
Updating / installing...
1:jdk1.8.0_111-2000:1.8.0_111-fcs ###
############################## [100%]
Unpacking JAR files...
tools.jar...
plugin.jar...
javaws.jar...
deploy.jar...
rt.jar...
jsse.jar...
charsets.jar...
localedata.jar...
[root@elkmaster1 ~]#


Feel free to reference the Oracle documentation:


http://docs.oracle.com/javase/8/docs/technotes/guides/install/linux_jre.html#BABCCAJA


Check your Java version:


[root@elkmaster1 ~]# java -version
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
[root@elkmaster1 ~]#


You may now delete the archive file that you downloaded earlier:


[root@elkmaster1 ~]# rm jdk-8u111-linux-x64.rpm
rm: remove regular file ‘jdk-8u111-linux-x64.rpm’? y
[root@elkmaster1 ~]#


Elasticsearch - Installation

Now we will install Elasticsearch 5.0 on the ELK Master 1 CentOS host. We will configure the official Elastic repository, so that we will be sure we have the latest available version of 5.x with all security and performance fixes included.

First we will need to import the Elasticsearch PGP Key:


[root@elkmaster1 ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

Tip:

How trustworthy is the developer who created the package? If the package is signed with the developer's GnuPG key (Elasticsearch BV in our case), you know that the developer really is who they say they are. Never disable packages signature check.


Now we will create the file with repository information:


[root@elkmaster1 ~]# vi /etc/yum.repos.d/elasticsearch.repo

With the following contents:


[elasticsearch-5.x]
name=Elasticsearch repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md


The repository is ready to use, so we can install elasticsearch:


[root@elkmaster1 ~]# yum -y install elasticsearch

After installation finishes, make sure that elasticsearch will start on server boot


[root@elkmaster1 ~]# systemctl daemon-reload
[root@elkmaster1 ~]# systemctl enable elasticsearch.service
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
[root@elkmaster1 ~]# systemctl start elasticsearch.service
[root@elkmaster1 ~]#


The main configuration file is located at this path: /etc/elasticsearch/elasticsearch.yml

The Systemd service file is located here: /usr/lib/systemd/system/elasticsearch.service

To override default values, add them to: /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf


Let us change the following settings in /etc/elasticsearch/elasticsearch.yml:


cluster.name: linuxacademy-elk
node.name: elkmaster1
node.attr.rack: centos7
network.host: 10.0.2.4
http.port: 9200
node.max_local_storage_nodes: 1


After doing that, we need to restart elasticsearch


[root@elkmaster1 ~]# systemctl restart elasticsearch

We also need to remove the –quiet option from ExecStart command line in the elasticsearch.service file.


[root@elkmaster1 ~]# find / -name "elasticsearch.service"
/sys/fs/cgroup/systemd/system.slice/elasticsearch.service
/etc/systemd/system/multi-user.target.wants/elasticsearch.service
/usr/lib/systemd/system/elasticsearch.service
[root@elkmaster1 ~]#


We need to edit /usr/lib/systemd/system/elasticsearch.service


ExecStart=/usr/share/elasticsearch/bin/elasticsearch \
-p ${PID_DIR}/elasticsearch
.pid \
--quiet \
-Edefault.path.logs=${LOG_DIR} \
-Edefault.path.data=${DATA_DIR} \
-Edefault.path.conf=${CONF_DIR}


Just remove the line with “--quiet \” and restart the service. Don’t forget to perform a daemon reload.


[root@elkmaster1 ~]# systemctl daemon-reload
[root@elkmaster1 ~]# systemctl restart elasticsearch.service


To tail the journal (only events related to elasticsearch):


[root@elkmaster1 ~]# journalctl --unit elasticsearch
-- Logs begin at Mon 2016-10-31 09:17:56 EDT, end at Mon 2016-10-31 09:44:11 EDT. --
Oct 31 09:35:22 elkmaster1 systemd[1]: Starting Elasticsearch...
Oct 31 09:35:22 elkmaster1 systemd[1]: Started Elasticsearch.
Oct 31 09:41:12 elkmaster1 systemd[1]: Stopping Elasticsearch...
Oct 31 09:41:12 elkmaster1 systemd[1]: Starting Elasticsearch...
Oct 31 09:41:12 elkmaster1 systemd[1]: Started Elasticsearch.
Oct 31 09:41:34 elkmaster1 systemd[1]: Stopping Elasticsearch...
Oct 31 09:41:34 elkmaster1 systemd[1]: Starting Elasticsearch...
Oct 31 09:41:34 elkmaster1 systemd[1]: Started Elasticsearch.
Oct 31 09:41:36 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:36,442][INFO ][o.e.n.Node ] [] initializing ...
Oct 31 09:41:36 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:36,558][INFO ][o.e.e.NodeEnvironment ] [TvCdZPM] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [5.3gb], net total_space [6.6gb
Oct 31 09:41:36 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:36,558][INFO ][o.e.e.NodeEnvironment ] [TvCdZPM] heap size [1.9gb], compressed ordinary object pointers [true]

[… redacted because of length …]

Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,978][INFO ][o.e.t.TransportService ] [TvCdZPM] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,982][WARN ][o.e.b.BootstrapCheck ] [TvCdZPM] max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [26214
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,172][INFO ][o.e.c.s.ClusterService ] [TvCdZPM] new_master {TvCdZPM}{TvCdZPMOR8a3TiOTmBrYKA}{Qs500KJrTDOMLfwmHUc78g}{127.0.0.1}{127.0.0.1:930
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,216][INFO ][o.e.h.HttpServer ] [TvCdZPM] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,217][INFO ][o.e.n.Node ] [TvCdZPM] started
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,237][INFO ][o.e.g.GatewayService ] [TvCdZPM] recovered [0] indices into cluster_state


To list journal entries for the elasticsearch service starting from a given time:


[root@elkmaster1 ~]# journalctl --unit elasticsearch --since  "2016-10-31 09:41:40"
-- Logs begin at Mon 2016-10-31 09:17:56 EDT, end at Mon 2016-10-31 09:44:11 EDT. --
Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,568][INFO ][o.e.n.Node ] [TvCdZPM] initialized
Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,568][INFO ][o.e.n.Node ] [TvCdZPM] starting ...
Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,978][INFO ][o.e.t.TransportService ] [TvCdZPM] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
Oct 31 09:41:41 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:41,982][WARN ][o.e.b.BootstrapCheck ] [TvCdZPM] max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [26214
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,172][INFO ][o.e.c.s.ClusterService ] [TvCdZPM] new_master {TvCdZPM}{TvCdZPMOR8a3TiOTmBrYKA}{Qs500KJrTDOMLfwmHUc78g}{127.0.0.1}{127.0.0.1:930
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,216][INFO ][o.e.h.HttpServer ] [TvCdZPM] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,217][INFO ][o.e.n.Node ] [TvCdZPM] started
Oct 31 09:41:45 elkmaster1 elasticsearch[2303]: [2016-10-31T09:41:45,237][INFO ][o.e.g.GatewayService ] [TvCdZPM] recovered [0] indices into cluster_state


For debug purposes we will also need to install netcat:


[root@elkmaster1 ~]# yum -y install nmap-ncat
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: ftp.colocall.net
* extras: ftp.colocall.net
* updates: ftp.colocall.net
Resolving Dependencies
--> Running transaction check
---> Package nmap-ncat.x86_64 2:6.40-7.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
nmap-ncat x86_64 2:6.40-7.el7 base 201 k

Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package

Total download size: 201 k
Installed size: 414 k
Downloading packages:
nmap-ncat-6.40-7.el7.x86_64.rpm | 201 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : 2:nmap-ncat-6.40-7.el7.x86_64 1/1
Verifying : 2:nmap-ncat-6.40-7.el7.x86_64 1/1

Installed:
nmap-ncat.x86_64 2:6.40-7.el7

Complete!
[root@elkmaster1 ~]#

Tip:
You can find advices to use telnet for debugging network services and checking ports availability, but I strongly advice you use netcat instead of telnet from beginning.  Although netcat seems a little bit more complicated comparing to telnet, with netcat you get a lot more options and will benefit in short term.


To check that Elasticsearch is running we will need to send an HTTP request to port 9200 on localhost


[root@elkmaster1 ~]# ncat -v localhost 9200
Ncat: Version 6.40 ( http://nmap.org/ncat )
Ncat: Connected to ::1:9200.
GET /
HTTP/1.0 404 Not Found
es.index_uuid: _na_
es.resource.type: index_or_alias
es.resource.id: bad-request
es.index: bad-request
content-type: application/json; charset=UTF-8
content-length: 367

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"bad-request","index_uuid":"_na_","index":"bad-request"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"bad-request","index_uuid":"_na_","index":"bad-request"},"status":404}


We can also check the status of our elasticsearch process followed by the most recent log data from the journal, with systemctl


[root@elkmaster1 ~]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-10-31 10:39:43 EDT; 3h 43min ago
Docs: http://www.elastic.co
Main PID: 937 (java)
CGroup: /system.slice/elasticsearch.service
└─937 /bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless...

Oct 31 10:39:59 elkmaster1 elasticsearch[937]: [2016-10-31T10:39:59,331][INFO ][o.e.n.Node ] [elkmaster1] initialized
Oct 31 10:39:59 elkmaster1 elasticsearch[937]: [2016-10-31T10:39:59,331][INFO ][o.e.n.Node ] [elkmaster1] starting ...
Oct 31 10:39:59 elkmaster1 elasticsearch[937]: [2016-10-31T10:39:59,687][INFO ][o.e.t.TransportService ] [elkmaster1] publish_address {10.0.2.4:9300}, bound_addresses {[::]:9300}
Oct 31 10:39:59 elkmaster1 elasticsearch[937]: [2016-10-31T10:39:59,691][INFO ][o.e.b.BootstrapCheck ] [elkmaster1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks

[… redacted because of length …]

Oct 31 14:12:28 elkmaster1 elasticsearch[937]: [2016-10-31T14:12:28,883][INFO ][o.e.c.m.MetaDataCreateIndexService] [elkmaster1] [.kibana] creating index, cause [api], templates [], shards [1]/[...erver, config]
Oct 31 14:13:01 elkmaster1 elasticsearch[937]: [2016-10-31T14:13:01,558][INFO ][o.e.c.m.MetaDataMappingService] [elkmaster1] [.kibana/8qsVhd-STbGLEOmkpYWDeA] create_mapping [index-pattern]
Hint: Some lines were ellipsized, use -l to show in full.
[root@elkmaster1 ~]#


As another check, we will query Elasticsearch directly:

curl -X GET http://localhost:9200/


[root@elkmaster1 ~]# curl -X GET http://localhost:9200/
{
"name" : "elkmaster1",
"cluster_name" : "linuxacademy-elk",
"cluster_uuid" : "9iTVBgpeQ4qAtUK3QStR8A",
"version" : {
"number" : "5.0.0",
"build_hash" : "253032b",
"build_date" : "2016-10-26T04:37:51.531Z",
"build_snapshot" : false,
"lucene_version" : "6.2.0"
},
"tagline" : "You Know, for Search"
}
[root@elkmaster1 ~]#


Tip:

Do not be surprised with the large number of different checks for ELK Stack components. Despite the simplicity of our current lab setup, this is a complex, integrated system that requires multiple verifications of all components, including at the initial configuration and setup stages. In the future, especially when dealing with complex ELK cluster configurations, if you neglect this simple rule, you risk spending a lot of time finding and fixing problems.

Also, testing and reading logs of all ELK components is one of the paths to the integral formation of the whole picture and the logic of the system as a whole.

That concludes it for part one of the ELK stack installation and configuration. In the next guide we will walk through installing other components of the stack including Kibana and Filebeat.


Go to Part 2.



Dmitry Korzhevin,

Crytek Lead System Administrator,

Head of Crytek CERT (Computer Emergency Response Team)

https://www.linkedin.com/in/dkorzhevin


  • post-author-pic
    Terrence C
    11-10-2016

    Can't wait for part II !!

  • post-author-pic
    Vitalii N
    12-19-2016

    Thank you for this interesting detailed guide, Dmitry! 
    Could you provide for me common commands with ncat that you use in your work for testing purposes?

  • post-author-pic
    Yash M
    01-13-2017

    Thank you Dmitry for nice tutorial......

    I am stuck in one place and not able to start services...need your help....
    getting error--

    Jan 13 09:15:45 elkmaster1 systemd[1]: [/usr/lib/systemd/system/elasticsearch.service:22] Trailing garbage, ignoring.

    Jan 13 09:15:45 elkmaster1 systemd[1]: [/usr/lib/systemd/system/elasticsearch.service:23] Unknown lvalue '-Edefault.path.logs' in section 'Service'



  • post-author-pic
    Ashraf S
    01-21-2017

    I configured ELK in my company with rsyslog instead of filebaet. 

    Checkout on my blog 

    https://arshnetworks.blogspot.ae/2017/01/elasticsearch-logstash-and-kibana-elk.html

  • post-author-pic
    Kenneth G
    02-16-2017

    Do you have any knowledge of logstash? how does logstash fit in to this stack? Or how is it different from elasticsearch?

  • post-author-pic
    Richard K
    03-30-2018

    wow thanks guys

  • post-author-pic
    Vitalii N
    03-31-2018

     @kennyg1980  Logstash is used as a filter for incoming logs before elasticsearch. For example, the workflow might be the followed: filebeat (gather logs) -> Logstash (listen for incoming messages from filebeat in the input section, then it filters each message in a filter section and finally sends it to the Elasticsearch in output section).

    Filebeat -> logstash (input -> filter -> output) -> Elasticsearch. 
     

  • post-author-pic
    Sheila H
    04-03-2018

    This is awesome. Thank you!

  • post-author-pic
    Harish M
    09-02-2018

    This is an good documentation..
    I am trying to use the steps in the link and I am stuck at the step of the port forwarding...
    After doing the IP setup and port forwarding, I am not able to connect from putty at 10.0.2.4
    The steps says host port as 127.0.0.1, however in the screen shot you have given, you are connecting to 10.0.2.4...

    am i  missing something here ?

Looking For Team Training?

Learn More