28. 04. 2023
Icinga Web 2, NetEye, Unified Monitoring
To be able to make Elasticsearch Snapshots you need shared storage mounted on all Elasticsearch Data Nodes. There are various possible file systems you can use for this: GFS, NFS, CIFS and GlusterFS.
What is GlusterFS
GlusterFS is a scalable network file system suitable for data-intensive tasks such as cloud storage and media streaming. GlusterFS is free and open source software, and can utilize common off-the-shelf hardware.
Installing GlusterFS on a NetEye 4 Cluster
While writing this post I’m using GlusterFS version 8.x, so even though 9.x has just been released in January, I’ll explain the installation for this well-tested release.
Let’s assume for this exercise that we have a 4-node NetEye 4 cluster where the nodes are:
neteye01.neteyelocal, neteye02.neteyelocal, neteye03.neteyelocal, neteye04.neteyelocal
Communication takes place over the internal TRUSTED network in this case. We also create a Distributed Replicated Volume since we have 4 nodes, and thus we can double the space of the single XFS Volume used as a basis for the Gluster volume. Also note that all communication with GlusterFS must be done over the glusterfs protocol, so writing directly to the XFS filesystem is prohibited and will break your GlusterFS volume!!!
- Create the REPO file for installing the GlusterFS packages (on all nodes)
cat >/etc/yum.repos.d/CentOS-ZZ-Gluster-8.repo <<EOM
CentOS-Gluster-8.repo
#
Please see http://wiki.centos.org/SpecialInterestGroup/Storage for more
information
[centos-gluster8]
name=CentOS-\$releasever - Gluster 8
mirrorlist=http://mirrorlist.centos.org?arch=\$basearch&release=\$releasever&repo=storage-gluster-8
baseurl=http://mirror.centos.org/\$contentdir/\$releasever/storage/\$basearch/gluster-8/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
[centos-gluster8-test]
name=CentOS-\$releasever - Gluster 8 Testing
baseurl=http://buildlogs.centos.org/centos/\$releasever/storage/\$basearch/gluster-8/
gpgcheck=0
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
EOM2. Create or download the Signature Key to use with this repo (on all nodes)
cat >/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage <<EOM
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.22 (GNU/Linux)
mQENBFTCLWABCADDHh5ktfB+78L6yxcIxwbZgaLKTp0mKvM3i2CjBrbw+xHJ4x9E
mn39rkTJf2UHOK0PgAp3FftoAFCtrSAXuanNGpEcpSxXDzxNj2QMpAbySZ2r4RpL
qxNVlB73dHuASXIMlhyV1ASpM6Me0dpaTtyKj38kRPFkWWuIUaiEQzXFgztYx7Kp
i+we0iUBfSKY47l2rbqyu9qZ8kCeMjuSuLfG5OKw+fj9zwqFJkc+LAz8IPTF4g7p
48m0m5bUPvKIIa1BfYcyqaTMxfbqjGaF1M37zF1O0TUKGQ+8VddzQmwg7GglQMt3
FqVer1WJUNPXyEgmZMzfmg7lqdPKKYaQBLk1ABEBAAG0XkNlbnRPUyBTdG9yYWdl
IFNJRyAoaHR0cDovL3dpa2kuY2VudG9zLm9yZy9TcGVjaWFsSW50ZXJlc3RHcm91
cC9TdG9yYWdlKSA8c2VjdXJpdHlAY2VudG9zLm9yZz6JATkEEwECACMFAlTCLWAC
GwMHCwkIBwMCAQYVCAIJCgsEFgIDAQIeAQIXgAAKCRDUouUL5FHltbq9B/93dtpt
lQG2mVvGik9TFgRdt+p3CPTqT1fwNzhB3iO02yJu5oM6s4FB1XqKRaKlqtvtBzyT
geAwenu74aU1hFv4uq+uETCanUaSgOvTcCn5WXUpOvlwKJV7TUjLSNRfp2dAG8Ig
d3euLnfajCE13t5BrqhTAlaMxAbGAqtzr6K9y0hUeT0ogjrscfoQSVptlcLs8d7m
P+VMR4GUfvUAws65JZxBaal4N7eIIZCWktnJ+B3dE3/tsAksGyXGLaSroPSuY18V
wksdBuscKVV49Ees0SbhvSrF5JJ07ccUt43SSFun84iNW4nuiWm2QOOKMcd182Sk
d9SDUTFu/G4s2gx7
=a0nM
-----END PGP PUBLIC KEY BLOCK-----
EOM3. Install the packages from the REPO (on all nodes)
yum install glusterfs gluster-cli glusterfs-libs glusterfs-server -y4. Start the Gluster Daemon (on all hosts)
systemctl start glusterd
systemctl enable glusterd
systemctl status glusterd5. Add the nodes to Gluster Infrastucture (on the first cluster node neteye01.neteyelocal)
gluster peer probe neteye02.neteyelocal
gluster peer probe neteye03.neteyelocal
gluster peer probe neteye04.neteyelocal
gluster peer status6. Create and mount the XFS Base Filesystems (on all nodes)
lvcreate -L10g -nlvbrick01_gv vg00
mkfs.xfs -i size=512 /dev/vg00/lvbrick01_gv
mkdir -p /data/glusterfs/brick01
cat >>/etc/fstab <<EOM
/dev/mapper/vg00-lvbrick01_gv /data/glusterfs/brick01 xfs defaults
localhost:/elastic-backup_gv /data/backup/elasticsearch glusterfs defaults 0 0
EOM
mount /data/glusterfs/brick01
mkdir /data/glusterfs/brick01/elastic-backup_gv
gluster volume create monitoring_gv replica 2 neteye01.neteyelocal:/data/glusterfs/brick01/elastic-backup_gv neteye02.neteyelocal:/data/glusterfs/brick01/elastic-backup_gv neteye03.neteyelocal:/data/glusterfs/brick01/elastic-backup_gv neteye04.neteyelocal:/data/glusterfs/brick01/elastic-backup_gv
gluster volume start elastic-backup_gv
gluster volume status
gluster volume infoAnd let’s also set a 5 second timeout for GlusterFS communication should one of the nodes go down (the default is 40 seconds):
gluster volume set elastic-backup_gv network.ping-timeout 57. Finally, mount your shared storage on ALL Nodes
mount /data/backup/elasticsearch
chown elasticsearch:elasticsearch /data/backup/elasticsearchFinal Words
So now your shared Storage Pool is set up and owned by the elasticsearch user, and you can now enter your Kibana Web Interface and configure your Elasticsearch Snapshots.
The order of the host bricks in the creation of the volume is important for this type of volume since replication is always done over 2 subsequent nodes. So for the volume we created we’ll have replication between neteye01.neteyelocal–neteye02.neteyelocal on one hand and neteye03.neteyelocal–neteye04.neteyelocal on the other. The distribution is made by GlusterFS writing the files or on one subvolume or the other.
Juergen Vigna
NetEye Solution Architect at Würth Phoenix
I have over 20 years of experience in the IT branch. After first experiences in the field of software development for public transport companies, I finally decided to join the young and growing team of Würth Phoenix. Initially, I was responsible for the internal Linux/Unix infrastructure and the management of CVS software. Afterwards, my main challenge was to establish the meanwhile well-known IT System Management Solution WÜRTHPHOENIX NetEye. As a Product Manager I started building NetEye from scratch, analyzing existing open source models, extending and finally joining them into one single powerful solution. After that, my job turned into a passion: Constant developments, customer installations and support became a matter of personal. Today I use my knowledge as a NetEye Senior Consultant as well as NetEye Solution Architect at Würth Phoenix.
Author
Latest posts by Juergen Vigna
30. 08. 2024
Log-SIEM, NetEye, Unified Monitoring
Monitor your Elasticsearch Shards Count
02. 07. 2024
NetEye
Monitor the Tasks in the Windows Task Scheduler
23. 02. 2024
Log-SIEM, NetEye, Unified Monitoring
Monitoring Logs in Elasticsearch: A Practical Example