FileCloud is a hyper-secure file storage, sharing and collaboration platform that provides a powerful set of tools for admins and users to manage their data. This includes High Availability (HA) architecture support and content management functionalities, specifically content search via FileCloud’s Solr integration.
Solr is an open-source content indexing and search application developed and distributed by Apache. This application is included with FileCloud installations.
Pacemaker and Corosync are open-source software solutions maintained by ClusterLabs. These solutions provide cluster management capabilities to client servers. Specifically, Pacemaker is a resource manager tool used on computer clusters for HA architecture, whereas Corosync supports cluster membership and messaging.
By configuring Solr HA in FileCloud with Pacemaker and Corosync, the admin can strengthen redundancy configurations, improve overall resiliency of backend software components, including quorate and resource-driven clusters, and provide fine-tuned management capabilities within and between nodes.
This step-by-step guide will outline how to manually configure Solr HA with Pacemaker and Corosync in FileCloud.
solr01 – Solr host – cluster member
solr02 – Solr host – cluster member
solr03 – quorum-device – quorum for cluster
solr-ha – proxy-ha host
NFSShare – NFS resource which can be mounted on solr01 and solr02
The example laid out in this blog post uses CentOS 7 (CentOS Linux release 7.9.2009 (Core)).
The installation instructions for Pacemaker and Corosync clusters remain the same, regardless of the Linux distributor (Ubuntu, Fedora, RedHat, or Debian).
Installation and Configuration Instructions
Step 1: Prepare the Cluster
Install all available patches using the following command:
After installing the necessary patches, reboot the system. This step must be completed for all three hosts: solr01, solr02, and solr03.
Then, the package that provides necessary nfs-client subsystems must be installed.
yum install -y nfs-utils
Next, wget must be installed.
yum install -y wget
Step 2: Install Solr and Prepare the Cluster Environment
Installing Solr in your FileCloud instance is (naturally) a critical part of configuring Solr HA. As indicated above Solr can be broken down into specific Solr hosts that are members of a cluster. These hosts must be individually configured.
Prepare Clean OS
Beginning with Solr01, prepare a clean Linux-based OS (such as the example we are using, Centos7). You may also use other operating systems according to your preference.
On the clean OS, download the FileCloud installation script: filecloud-liu.sh (official installation script).
If any issues arise related to the REMI repo, the alternative can be used: https://zerg.codelathe.com/url/qj2ny2yrvtzehjsk
Create a Folder
Create the following folder: /opt/solrfcdata
Run the Command
Mount the NFS Filesystem
The NFS filesystem should be mounted under the following:
mount -t nfs ip_nfs_server:/path/to/nfs_resource /opt/solrfcdata
Start Solr Installation
Next, start the solr component installation from using FileCloud installation script:
Follow the instructions until reaching the selection screen.
Select the “solr” option and click “enter.” The installation process may take a few minutes. Wait for confirmation that installation has been completed.
Bind Solrd to External Interface
Host: solr01, solr02
Solrd will, by default, try to bind to the localhost only. Modify the file so that solr binds to the external interface.
Modify the following file: /opt/solr/server/etc/jetty-http.xml
Change the following line in the file.
<Set name="host"><Property name="jetty.host" default="127.0.0.1" /></Set>
<Set name="host"><Property name="jetty.host" default="0.0.0.0" /></Set>
Change System Daemon Control to System
Solr was started with the FileCloud installation. Before proceeding, stop the Solr service.
Host: solr01, solr02
Remove the following file: /etc/init,d/solr
Create a new file:
Edit this new file and copy the contents specified below to this file:
### Beginning of File ### [Unit] Description=Apache SOLR [Service] User=solr LimitNOFILE=65000 LimitNPROC=65000 Type=forking Restart=no ExecStart=/opt/solr/bin/solr start ExecStop=/opt/solr/bin/solr stop ### End of File ###
Save the file before continuing.
Verify New Service Definition is Working
Host: solr01, solr02
systemctl daemon-reload systemctl stop solrd
It should not return any errors. Start the service:
systemctl start solrd systemctl status solrd
Remove Folder Contents
Critical Note: THIS COMMAND SHOULD BE ISSUED ONLY ON solr02.
systemctl stop solrd rm -rf /opt/solrfcdata/*
Update Firewall Rules
Complete this step whenever needed, as in the below example on CentOS.
Host: solr01, solr02
firewall-cmd --permanent --add-port 8983/tcp firewall-cmd --reload
With these steps completed, the Solr installation has been carried out to successfully prepare the environment for HA clusters.
Step 3: Set Up Pacemaker
Host: solr01, solr02, solr03
Edit /etc/hosts File
Add the entries for all 3 cluster nodes, so that the file reads as follows:
coresponding_ip solr01 coresponding_ip solr02 coresponding_ip solr03
File: cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.101.59 solr01 192.168.101.60 solr02 192.168.101.61 solr03
Install Cluster Packages
hosts: solr01 and solr02
yum -y install pacemaker pcs corosync-qdevice sbd
Enable and Start the Main Cluster Daemon
hosts: solr01 and solr02
systemctl start pcsd systemctl enable pcsd
Update Passwords for the Cluster User
hosts: solr01, solr02
Set the same password for all hosts for the hacluster user.
Provide the hacluster user with the login credentials, as these will be necessary in later steps.
Open Network Traffic on Firewall
hosts: solr01 and solr02
firewall-cmd --add-service=high-availability –permanent firewall-cmd --reload
Authorize Cluster Nodes
pcs cluster auth solr01 solr02
Password: “secret_password” set in the previous step.
solr01 Authorized solr02 Authorized
Create Initial Cluster Instance
pcs cluster setup --name solr_cluster solr01 solr02
Start and Enable Cluster Instance
pcs cluster start --all pcs cluster enable --all
Step 4: Set Up QDevice – Quorum Node
Install Software Required for Quorum-only Cluster Node
Install the required software on solr03 (quorum-only cluster node).
yum install pcs corosync-qnetd
Start and Enable the PCSD Daemon
systemctl enable pcsd.service systemctl start pcsd.service
Configure QDevice (Quorum Mechanism)
pcs qdevice setup model net --enable –start
Open Firewall Traffic
Open the firewall traffic (if required – below example on CentOS)
firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability
Set the Password for HA Cluster User
Set the password for the hacluster user on solr03.
Provide the password to the HA cluster user. This password should be the same password used for solr01 and solr02.
Authenticate QDevice Host in the Cluster
pcs cluster auth solr03
Add Quorum Device to the Cluster and Verify
pcs quorum device add model net host=solr03 algorithm=lms
pcs quorum status
Quorum information ------------------ Date: Wed Aug 3 10:27:26 2022 Quorum provider: corosync_votequorum Nodes: 2 Node ID: 1 Ring ID: 2/9 Quorate: Yes Votequorum information ---------------------- Expected votes: 3 Highest expected: 3 Total votes: 3 Quorum: 2 Flags: Quorate Qdevice Membership information ---------------------- Nodeid Votes Qdevice Name 2 1 A,V,NMW solr02 1 1 A,V,NMW solr01 (local) 0 1 Qdevice
Step 5: Install Soft-Watchdog
The module softwatchdog should load automatically after rebooting the system.
Host: solr01, solr02
echo softdog > /etc/modules-load.d/watchdog.conf
Reboot solr01 and solr02 to Activate Watchdog
Host: solr01, solr02
Carry out the reboots in sequence:
- reboot solr01 and wait until it comes back
- reboot solr02
Step 6: Enable SBD Mechanism in the Cluster
Host: solr01, solr02
pcs stonith sbd enable
Restart Cluster so pkt 1 Takes Effect
pcs cluster stop --all pcs cluster start --all
Verify the SBD Mechanism
pcs stonith sbd status
SBD STATUS <node name>: <installed> | <enabled> | <running> solr01: YES | YES | YES solr02: YES | YES | YES
Step 7: Create Cluster Resources
Create Cluster Resource with NFSMount
pcs resource create NFSMount Filesystem device=192.168.101.70:/mnt/rhvmnfs/solrnfs directory=/opt/solrfcdata fstype=nfs --group solr
The parameter device should point to the nfs server and nfs share being used in the configuration.
Cluster name: solr_cluster Stack: corosync Current DC: solr01 (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum Last updated: Wed Aug 3 12:22:36 2022 Last change: Wed Aug 3 12:20:35 2022 by root via cibadmin on solr01 2 nodes configured 1 resource instance configured Online: [ solr01 solr02 ] Full list of resources: Resource Group: solr NFSMount (ocf::heartbeat:Filesystem): Started solr01 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled sbd: active/enabled
Change the Recovery Strategy for the NFSMount Resource
pcs resource update NFSMount meta on-fail=fence
Create Cluster Resource – solrd
pcs resource create solrd systemd:solrd --group solr
Cluster name: solr_cluster Stack: corosync Current DC: solr01 (version 1.1.23-1.el7_9.1-9acf116022) - partition with quorum Last updated: Wed Aug 3 12:25:45 2022 Last change: Wed Aug 3 12:25:22 2022 by root via cibadmin on solr01 2 nodes configured 2 resource instances configured Online: [ solr01 solr02 ] Full list of resources: Resource Group: solr NFSMount (ocf::heartbeat:Filesystem): Started solr01 solrd (systemd:solrd): Started solr02 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled sbd: active/enabled
Set Additional Cluster Parameters
pcs property set stonith-watchdog-timeout=36 pcs property set no-quorum-policy=suicide
Step 8: Configure haproxy on Dedicated Host
Install haproxy on Clean OS
Our example uses CentOS.
yum install -y haproxy
Configure the haproxy
Configure the haproxy to redirect to the active solr node.
backup file: /etc/haproxy/haproxy.cfg
mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg_bck
Create an Empty File
Add the content below into the empty file.
#### beginning of /etc/haproxy/haproxy.cfg ### global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend solr_front *:8983 default_backend solr_back backend static balance roundrobin server static 127.0.0.1:4331 check backend solr_back server solr01 solr01:8983 check server solr02 solr02:8983 check #### beginning of /etc/haproxy/haproxy.cfg ###
Ensure that parameters solr01/solr02 point to the full DNS name or to the IP of the cluster nodes.
systemctl enable haproxy systemctl start haproxy
Solr service will be available on host solr-ha on port 8983 – independent of where it is really running, either on solr01 or solr02.
Congratulations! If you followed these step-by-step instructions, you will have successfully configured Solr with high availability along with Pacemaker and Corosync. This configuration will serve to improve redundancy and security for your critical data.
For any questions on Solr or High-Availability architecture, schedule a consultation or configuration support session.
Article written by Marek Frueauff, Solutions Architect
Edited by Katie Gerhardt, Junior Product Marketing Manager
Appendix – Glossary of Terms
Below are the key terms used in this article, listed in alphabetical order.
|Cluster||The group of servers or other IT systems, which primary purpose is to realize similar or exactly the same function to achieve one or both of the outcome’s: High Availability or Load Balance.|
|Cluster Quorum||Server or other system that is part of the cluster and performs a particular role: verify which production cluster nodes (servers) can be communicated and their health status. In cluster members are missing, the cluster quorum system decides if the remaining servers can operate and continue providing services or if it should be treated as unhealthy. There is a risk of the split brain situation. The main purpose of the cluster quorum system is to avoid the split brain scenario.|
|Corosync||Corosync is a typical part of High Availability architecture set up in Linux or Unix systems and usually exists alongside pacemaker. Corosync is the communication engine responsible for keeping cluster nodes (servers) in sync state.|
|Firewall||Software or hardware which can inspect and manipulate based on the multiple rules the network traffic. The modern firewalls implementations can operate on multiple network layers (usually from 3 to 7) including the network frame content inspection.|
|Firewall-cmd||The modern Linux build in firewall software implementation.|
|nfs||Network File System – is the filesystem which by design is network related. It is common method to share file resources in the unix environment. Due to very long history related to this technology it has been implemented almost on all possible operating systems and became very popular and commonly used.|
|Pacemaker||Open-source software involved in cluster resource management and part of a typical High Availability setup on Linux systems to provide modern functions and cluster management resources.|
|Proxy||Software or hardware solution that provides a gateway between two networks separated by design. A proxy is usually installed between the public Internet and a local network and allows some communications between those network segments based on predefined rules. A proxy can also be used for other purposes, like load balancing: for example redirecting incoming connections from one network to multiple hosts in another network segment.|
|Proxy-HA||The specific implementation of the proxy mechanism to provide High Availability service, which is usually correlated with a single host (server). In our example proxy-ha is used to verify where services are currently running (on which cluster servers) and redirect all incoming requests to the active node.|
|Resource Group||A logical organization unit within the pacemaker cluster implementation that enables control of the dependencies between particular resources managed by the cluster. For example, the nfs server that shares files must be started after the filesystem where the files resists and additionally on the same cluster node (server) – this control can be easily achieved using Resource Groups.|
|QDevice||The software implementation of the quorum functionality in the pacemaker cluster setup. This kind of functionality is being installed on the cluster host, which will perform the quorum role only, and will never provide any other services.|
|SBD||Stonith Block Device – by design this the implementation of the additional communication and stonith mechanism on top of shared block device between cluster nodes (servers). In some cases, sbd can be used in the diskless mode (as in our example). To operate in this mode, the watchdog mechanism needs to be enabled/installed.|
|Solr||Advanced and open-source search and indexing system maintained and developed by Apache. This mechanism is a part of the standard FileCloud installation.|
|Split Brain||Very dangerous scenario in all cluster environments in which a node or nodes loses the ability to communicate with the rest of the node population due to an environment malfunction (most often due to lost network connectivity). In this situation, a separated node may “think” that it is the “last man standing” and calls up all cluster resources to begin providing all services. This resource demand is repeated by all cluster nodes, leading to disagreement on which node should remain active and which services the cluster should provide. Each cluster implementation has multiple built-in mechanisms to prevent this situation, which can easily lead to data corruption. One such mechanism is stonith, which is activated as soon as the node is loses its “quorate” status –indicating a high probability that the node is not visible by the rest of the environment.|
|Stonith||Shut The Other Node in The Head is a mechanism that allows an immediate restart (without any shut down procedure) of any node in the cluster. This mechanism is extremely important to prevent potential data corruption by the wrong cluster node behavior.|
|SystemV||The name of the former Linux approach to starting and stopping system services (daemons).|
|SystemD||The name of the modern Linux approach to starting and stopping system services (daemons) and much more. Each modern Linux distribution now uses systemd as the main mechanism to manage system services.|
|Watchdog||The software or hardware mechanism that works like a delayed bomb detonator. The watchdog is periodically pinged by the system (approximately every 5 seconds) to reset the countdown procedure. If the countdown reaches 0, watchdog will reset the operating system immediately. Watchdog is used with Pacemaker in clusters to ensure that nodes remain recognized within the cluster community. In the event of a lost connection (which is the typical reason behind the Split Brain scenario), Watchdog enables an immediate reboot of the node.|