Setup clustering on RHEL

To setup clustering via PCS on RHEL

 

Create cluster authentication

pcs cluster auth server1 server2 -u hacluster

 

Setup & initialize cluster and nodes

pcs cluster setup –name clusterName server1 server2

 

Set STONITH 

pcs property set stonith-enabled=true

 

 

 

 

Setting cluster parameters

 

Create ipmi users for clustering

ipmitool -U fence -I lanplus -P ‘fencePassword’ -H 192.168.0.1 sensor -L Administrator

 

iDRAC IP for IPMI

server1 (192.1.1.1)

server2 (192.2.2.2)

 

Create pcs stonith entity

pcs stonith create server1 fence_ipmilan pcmk_host_list=server1 delay=30 ipaddr=192.1.1.x login=fence passwd=password  privlvl=Operator lanplus=1 op monitor interval=60s

pcs stonith create server2 fence_ipmilan pcmk_host_list=server2 ipaddr=192.0.0.x login=fence passwd=password privlvl=Operator lanplus=1 op monitor interval=60s

 

Configure corosync

edit /etc/corosync/corosync.conf to have rrp_mode:active & add new ring#_addr to make redundancy

restart corosync

“corosync-cfgtool -s” to check

 

Create Virtual IP Address

pcs resource create virtual_ip ocf:heartbeat:IPaddr2 ip=192.1.1.x cidr_netmask=32 nic=bond0 op monitor interval=30s

 

Create ping gateway

pcs resource create ping_gateway_db ocf:pacemaker:ping host_list=192.1.1.1

 

Create LVM

pcs resource create vg_serviceName LVM volgrpname=serviceName exclusive=true

 

Edit the lvm.conf file to mount LVM

use_lvmetad = 0 on all nodes

locking_type = 1 on all nodes

 

        volume_list = [ “local_volume_group” ]

 

# systemctl stop    lvm2-lvmetad

# systemctl disable lvm2-lvmetad

# systemctl mask lvm2-lvmetad.socket

 

# cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.$(date +%m-%d-%H%M%S).bak

# dracut -f -v

 

Oracle Filesystems

pcs resource create fs_serviceName Filesystem device=”/dev/mapper/service-oracle” directory=”/oracle/serviceName” force_unmount=”true” fstype=”ext4″ run_fsck=”no”

pcs resource create fs_oracle_data1 Filesystem device=”/dev/mapper/serviceName-oradata1″ directory=”/oracle/serviceName/oradata1″ force_unmount=”true” fstype=”ext4″ run_fsck=”no”

pcs resource create fs_oracle_data2 Filesystem device=”/dev/mapper/serviceName-oradata2″ directory=”/oracle/serviceName/oradata2″ force_unmount=”true” fstype=”ext4″ run_fsck=”no”

 

Oracle instance & listener

pcs resource create db-oracle-instance oracle sid=”Oracle_ID” home=/oracle/serviceName/19_64 monuser=”monuser” monpassword=”password” user=”serviceUser” monprofile=”default”

 

*serious issue if home= variable has / at the end

*https://dbvisit.com/blog/connected-to-an-idle-instance-but-database-is-running

 

pcs resource create db-oracle-listener ocf:heartbeat:oralsnr home=/oracle/serviceName/19_64 listener=LISTENER_serviceName sid=”Oracle_ID” tns_admin=/oracle/serviceName/19_64/network/admin user=”serviceUser” op methods interval=0s timeout=5 monitor interval=10 timeout=30 start interval=0s timeout=120 stop interval=0s timeout=120

 

Resource group

pcs resource group add group_name resource_id…

pcs resource group add service-db-vip ping_gateway_db vg_serviceName fs_gspnb fs_oracle_data1 fs_oracle_data2  db-oracle-instance db-oracle-listener

 

Add constraint

location add <id> <resource> <node> <score>

pcs constraint location add server1 ServiceName server1 20000

pcs constraint location add server2 ServiceName server2 10000

 

location add <id> <resource> <node> <score>

pcs constraint location add location-server2-server1-INFINITY server2 server1 INFINITY

pcs constraint location add location-server1-server2-INFINITY server1 server2 INFINITY

 

Set up logical volume management

volume_list attribute should be commented on /etc/lvm/lvm.conf

#volume_list=[~~~~~]

 

Clean up failure messages

pcs resource cleanup [resource id]

 

During & After system migration

disable stonith during configuration to prevent fencing

pcs property set stonith-enabled=false

 

set fence reaction to panic after migration

pcs property set fence-reaction=panic

 

Show constraint

pcs constraint show –full

 

Add stickiness on resource meta

stickiness is value to indicate how much the resource prefers to stay where it is

stickiness is like the “cost” of moving resource groups

to see current score on each resource group

pcs constraint show –full

EX:  Resource: GSPNB

    Enabled on: gspnbldb (score:20000) (id:gspnb1)

    Enabled on: gspndev (score:10000) (id:gspnb2)

 

to set the stickiness

pcs resource meta <ResourceGroup> resource-stickiness=[NumberValue] priority=[NumberValue]

pcs resource meta GSPNB resource-stickiness=15000 priority=100

 

this will allow the resource to move when “move” command is issued, otherwise it will stick to failover node

calculation:

score of 20000 > score of 10000 so it’ll stay at gspnbldb

once you issue failover or move command, it will create ban constraint to not allow moving back

“move” option on pcs is to “move the  resource  off  the  node  it  is currently running on by creating a -INFINITY location constraint to ban the node” (it ain’t coming back…)

the stickiness cost of 15000 can be added to help move 10000+15000 > 20000 to push failover node to move back to original node

 

Remove constraint

you need to remove constraint that may have been created during resource group move OR failover (it will not failback otherwise)

pcs constraint remove <constraint_id>

 

To test fencing

it will shutdown the node specified and remove node from clustering system

pcs stonith fence [NodeName]

 

 

 

*cheat sheet on RedHat Linux Guru