+++ /dev/null
-Valid as of 1.0.66, may/will change in the future
-
-
-RECMASTER
-=========
-Recovery Master, this is one of the nodes in the cluster that has been designated to
-be the "recovery master".
-The recovery master is responsible for performing full checks of cluster and cluster node consistency and is also responsible for performing the actual database recovery procedure.
-
-Only one node at a time can be the recovery master.
-This is ensured by CTDB using a lock on a single file in the shared gpfs filesystem:
- /etc/sysconfig/ctdb :
- ...
- # Options to ctdbd. This is read by /etc/init.d/ctdb
- # you must specify the location of a shared lock file across all the
- # nodes. This must be on shared storage
- # there is no default here
- CTDB_RECOVERY_LOCK=/gpfs/.ctdb/shared
- ...
-
-In order to prevent that two nodes become recovery master at the same time (==split brain)
-CTDB here relies on GPFS that GPFS will guarantee coherent locking across the cluster.
-Thus CTDB relies on that GPFS MUST only allow one ctdb process on one node to take out and
-hold this lock.
-
-The recovery master is designated through an election process.
-
-
-VNNMAP
-======
-The VNNMAP is a list of all nodes in the cluster that is currently part of the cluster
-and participates in hosting the cluster databases.
-All nodes that are CONNECTED but not BANNED be present in the VNNMAP.
-
-The VNNMAP is the list of LMASTERS for the cluster as reported by 'ctdb status' "
- ...
- Size:3
- hash:0 lmaster:0
- hash:1 lmaster:1
- hash:2 lmaster:2
- ...
-
-
-CLUSTER MONITORING
-==================
-All nodes in the cluster monitor its own health and its own consistency regards to the
-recovery master. How and what the nodes monitor for differs between the node which is
-the recovery master and normal nodes.
-This monitoring it to ensure that the cluster is healthy and consistent.
-This is not related to monitoring of inidividual node health, a.k.a. eventscript monitroing.
-
-At the end of each step in the process are listed some of the most common and important
-error messages that can be generated during that step.
-
-
-NORMAL NODE CLUSTER MONITORING
-------------------------------
-Monitoring is performed in the dedicated recovery daemon process.
-The implementation can be found in server/ctdb_recoverd.c:monitor_cluster()
-This is an overview of the more important tasks during monitoring.
-These tests are to verify that the local node is consistent with the recovery master.
-
-Once every second the following monitoring loop is performed :
-
-1, Verify that the parent ctdb daemon on the local node is still running.
- If it is not, the recovery daemon logs an error and terminates.
- "CTDB daemon is no longer available. Shutting down recovery daemon"
-
-2, Check if any of the nodes has been recorded to have misbehaved too many times.
- If so we ban the node and log a message :
- "Node %u has caused %u failures in %.0f seconds - banning it for %u seconds"
-
-3, Check that there is a recovery master.
- If not we initiate a clusterwide election process and log :
- "Initial recovery master set - forcing election"
- and we restart monitoring from 1.
-
-4, Verify that recovery daemon and the local ctdb daemon agreed on all the
- node BANNING flags.
- If the recovery daemon and the local ctdb daemon disagrees on these flags we update
- the local ctdb daemon, logs one of two messages and restarts monitoring from 1 again.
- "Local ctdb daemon on recmaster does not think this node is BANNED but the recovery master disagrees. Unbanning the node"
- "Local ctdb daemon on non-recmaster does not think this node is BANNED but the recovery master disagrees. Re-banning the node"
-
-5, Verify that the node designated to be recovery master exists in the local list of all nodes.
- If the recovery master is not in the list of all cluster nodes a new recovery master
- election is triggered and monitoring restarts from 1.
- "Recmaster node %u not in list. Force reelection"
-
-6, Check if the recovery master has become disconnected.
- If is has, log an error message, force a new election and restart monitoring from 1.
- "Recmaster node %u is disconnected. Force reelection"
-
-7, Read the node flags off the recovery master and verify that it has not become banned.
- If is has, log an error message, force a new election and restart monitoring from 1.
- "Recmaster node %u no longer available. Force reelection"
-
-8, Verify that the recmaster and the local node agrees on the flags (BANNED/DISABLED/...)
- for the local node.
- If there is an inconsistency, push the flags for the local node out to all other nodes.
- "Recmaster disagrees on our flags flags:0x%x recmaster_flags:0x%x Broadcasting out flags."
-
-9, Verify that the local node hosts all public ip addresses it should host and that it does
- NOT host any public addresses it should not host.
- If there is an inconsistency we log an error, trigger a recovery to occur and restart
- monitoring from 1 again.
- "Public address '%s' is missing and we should serve this ip"
- "We are still serving a public address '%s' that we should not be serving."
-
-These are all the checks we perform during monitoring for a normal node.
-These tests are performed on all nodes in the cluster which is why it is optimized to perform
-as few network calls to other nodes as possible.
-Each node only performs 1 call to the recovery master in each loop and to no other nodes.
-
-RECOVERY MASTER CLUSTER MONITORING
------------------------------------
-The recovery master performs a much more extensive test. In addition to tests 1-9 above
-the recovery master also performs the following tests:
-
-10, Read the list of nodes and flags from all other CONNECTED nodes in the cluster.
- If there is a failure to read this list from one of the nodes, then log an
- error, mark this node as a candidate to become BANNED and restart monitoring from 1.
- "Unable to get nodemap from remote node %u"
-
-11, Verify that the local recovery master and the remote node agrees on the flags
- for the remote node. If there is a inconsistency for the BANNING flag,
- log an error, trigger a new recmaster election and restart monitoring from 1.
- "Remote node %u had different BANNED flags 0x%x, local had 0x%x - trigger a re-election"
- "Remote node %u had flags 0x%x, local had 0x%x - updating local"
-
-12, Verify that the local recovery master and the remote node agrees on the flags
- for the remote node. If one of the flags other than the BANNING flag was inconsistent,
- just update the set of flags for the local recovery daemon, log an information message
- and continue monitoring.
- "Remote node %u had flags 0x%x, local had 0x%x - updating local"
-
-13, Read the list of public ip addresses from all of the CONNECTED nodes and merge into a
- single clusterwide list.
- If we fail to read the list of ips from a node, log an error and restart monitoring from 1.
- "Failed to read public ips from node : %u"
-
-14, Verify that all other nodes agree that this node is the recovery master.
- If one of the other nodes discgrees this is the recovery master, log an error,
- force a new election and restart monitoring from 1.
- "Node %d does not agree we are the recmaster. Need a new recmaster election"
-
-15, Check if the previous attempt to run a recovery failed, and if it did, try a new recovery.
- After the recovery, restart monitoring from 1.
- "Starting do_recovery"
-
-16, Verify that all CONNECTED nodes in the cluster are in recovery mode NORMAL.
- If one of the nodes were in recovery mode ACTIVE, force a new recovery and restart
- monitoring from 1.
- "Node:%u was in recovery mode. Start recovery process"
-
-17, Verify that the filehandle to the recovery lock file is valid.
- If it is not, this may mean a split brain and is a critical error.
- Try a new recovery and restart monitoring from 1.
- "recovery master doesn't have the recovery lock"
-
-18, Verify that GPFS allows us to read from the recovery lock file.
- If not there is a critical GPFS issue and we may have a split brain.
- Try forcing a new recovery and restart monitoring from 1.
- "failed read from recovery_lock_fd - %s"
-
-19, Read the list of all nodes and flags from all CONNECTED nodes in the cluster.
- If fail to read the nodemap from one of the remote nodes, log an error and restart
- monitoring from 1.
- "Unable to get nodemap from remote node %u"
-
-20, If the nodemap differs between the local node and the remote node, log an error
- and force a recovery.
- This would happen if the /etc/ctdb/nodes file differs across nodes in the cluster.
- It is unlikely that the recovery will rectify the situation.
- This is a critical error, it is most likely the entire cluster will be unavailable
- until the files are fixed or have became banned.
- "Remote node:%u has different node count. %u vs %u of the local node"
-
-21, If a remote node disagrees on the content of the nodes list, try a recovery and restart
- monitoring from 1.
- It is unlikely that the recovery will rectify the situation.
- This is a critical error, it is most likely the entire cluster will be unavailable
- until the files are fixed or have became banned.
- "Remote node:%u has different nodemap pnn for %d (%u vs %u)."
-
-22, If a remote node disgrees on the node flags in the list, try a recovery to re-sync
- the flags and restart monitoring from 1.
- "Remote node:%u has different nodemap flag for %d (0x%x vs 0x%x)"
-
-23, Verify that all active nodes are part of the VNNMAP.
- If not, this would be a new node that has become CONNECTED but does not yet participate
- in the cluster.
- Perform a recovery to merge the new node to the cluster and restart monitoring from 1.
- "The vnnmap count is different from the number of active nodes. %u vs %u"
- or
- "Node %u is active in the nodemap but did not exist in the vnnmap"
-
-24, Read the VNNMAP from all CONNECTED nodes.
- Verify that all nodes have the same VNNMAP content and that all nodes are in the same
- generation instance of the databases.
- If not, force a recovery to re-synchronize the vnnmap and the databases across the cluster
- and restart monitoring from 1.
- "Remote node %u has different generation of vnnmap. %u vs %u (ours)"
- "Remote node %u has different size of vnnmap. %u vs %u (ours)"
- "Remote node %u has different vnnmap."
-
-25, If there has been changes to the cluster that requires a reallocation of public ip
- addresses. On all nodes run the "startrecovery" event. Run "releaseip" and "takeip"
- events to reassign the ips across the cluster and finally run the "recovered" event.
-
-Finished monitoring, continue monitoring from 1.
-
-
-CLUSTER RECOVERY
-================
-Recoveries are driven by the recovery daemon on the node that is currently the recovery
-master.
-Most of the logging that is performed during recovery is only logged on the node that
-is the recovery master.
-Make sure to find which node is the recovery master and check the log for that node.
-
-Example log entries that start in column 1 are expected to be present in the
-log. Example log entries that are indented 3 columns are optional and might
-only be present if an error occurred.
-
-
-1, Log that recovery has been initiated.
-"Starting do_recovery"
-
- It might log an informational message :
-"New recovery culprit %u".
- This is only semi-accurate and might may not mean that there is any problem
- at all with the node indicated.
-
-
-2, Check if a node has caused too many failed recoveries and if so ban it from
- the cluster, giving the other nodes in the cluster a chance to recovery
- operation.
- "Node %u has caused %u recoveries in %.0f seconds - banning it for %u seconds"
-
-
-3, Verify that the recovery daemon can lock the recovery lock file.
- At this stage this should be recovery master.
- If this operation fails it means we have a split brain and have to abort recovery.
- "("ctdb_recovery_lock: Unable to open %s - (%s)"
- "ctdb_recovery_lock: Failed to get recovery lock on '%s'"
- "Unable to get recovery lock - aborting recovery"
-"ctdb_recovery_lock: Got recovery lock on '%s'"
-
-
-4, Log which node caused the recovery to be initiated.
- This is a semi-accurate information message only.
- This line does NOT mean that there has to be something wrong with the node listed.
-"Recovery initiated due to problem with node %u"
-
-
-5, Pull the names of all databases from all nodes and verify that these databases also
- exists locally.
- If a database is missing locally, just create it.
- It is not an error if a database is missing locally. Databases are created on demand and
- this could happen if it was one database which samba has never tried to access on the
- local node.
-
-
-6, Check the list of databases on each remote node and create any databases that may be missing
- on the remote node.
-"Recovery - created remote databases"
-
-
-7, Set recovery mode to ACTIVE on all remote nodes.
-
-
-8, Run the "startrecovery" eventscript on all nodes.
-
- At this stage you will also get a few additional log entries, these are not
- from the recovery daemon but from the main ctdb daemon due to running
- the eventscript :
-"startrecovery eventscript has been invoked"
-"Monitoring has been disabled"
-"Executing event script ...
-...
-
-
-9, Create a new generation id and update the generation id and the VNNMAP on the local node
- only.
- This guarantees that the generation id will now be inconsistent across the cluster and
- that if recovery fails a new recovery is attempted in the next iteration of the monitoring
- loop.
-
-
-10, Start a TDB TRANSACTION on all nodes for all databases.
- This is to ensure that if recovery is aborted or fails that we do not
- modify any databases on only some of the nodes.
-"started transactions on all nodes"
-
-
-11, For each database, pull the content from all CONNECTED nodes and merge it into
- the TDB databases on the local node.
- This merges the records from the remote nodes based on their serial numbers so we
- only keep the most recent record found.
-"Recovery - pulled remote database 0x%x"
-
-
-12, For each database, perform a fast TDB WIPE operation to delete the entire TDB under the
- transaction started above.
-
-
-13, For each database, drop all empty records.
- Force the DMASTER field of all records to point to the recovery master.
- Push the database out to all other nodes.
-
- The PUSH process lists some additional log entries for each database of the
- form :
-"Recovery - pulled remote database 0x..."
-"Recovery - pushed remote database 0x... of size ..."
-
-
-14, Commit all changes to all TDB databases.
-"Recovery - starting database commits"
-"Recovery - committed databases"
-
-
-15, Create a new VNNMAP of all CONNECTED nodes, create a new generation number
- and piush this new VNNMAP out to all nodes.
-"Recovery - updated vnnmap"
-
-
-16, Update all nodes that the local node is the recovery master.
-"Recovery - updated recmaster"
-
-
-17, synchronize node flags across the cluster.
-"Recovery - updated flags"
-
-18, Change recovery mode back to NORMAL.
-"Recovery - disabled recovery mode"
-
-
-19, Re-allocate all public ip addresses across the cluster.
-"Deterministic IPs enabled. Resetting all ip allocations"
-
- If the IP address allocation on the local node changes you might get
- "Takeover of IP 10.0.0.201/24 on interface eth0"
- "Release of IP 10.0.0.204/24 on interface eth0"
-
-"Recovery - takeip finished"
-
-
-20, Run the "recovered" eventscript on all nodes.
-"Recovery - finished the recovered event"
-
- You will also get an entry from the local ctdb daemon itself that it has
- switched back to recovery mode NORMAL.
-"Recovery has finished"
-
-
-21, Broadcast a message to all samba daemons in the cluster that the databases have been
- recovered. Samba will now do some additional checking/cleanup of the content in the stored
- records.
-
-"Recovery complete"
-
-
-22. Finished. At this stage a 10 second timeout (ctdb listvars : rerecoverytimeout) is
- initiated. The cluster will not allow a new recovery to be performed until this timeout
- has expired.
-
-"New recoveries suppressed for the rerecovery timeout"
-"Rerecovery timeout elapsed. Recovery reactivated."
-
-
-
-
-
-
-
-Example : RECOVERY LOG ON RECMASTER
-====================================
-2008/12/01 09:57:28.110732 [ 4933]: 10.0.0.21:4379: node 10.0.0.24:4379 is dead: 2 connected
-2008/12/01 09:57:28.110838 [ 4933]: Tearing down connection to dead node :3
-2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:2682 The vnnmap count is different from the number of active nodes. 4 vs 3
-2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1327 Starting do_recovery
-2008/12/01 09:57:28.967297 [ 4935]: ctdb_recovery_lock: Got recovery lock on '/gpfs/.ctdb/shared'
-2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1355 Recovery initiated due to problem with node 0
-2008/12/01 09:57:28.967297 [ 4935]: server/ctdb_recoverd.c:1381 Recovery - created remote databases
-2008/12/01 09:57:28.973543 [ 4933]: server/ctdb_recover.c:589 Recovery mode set to ACTIVE
-2008/12/01 09:57:28.974823 [ 4933]: server/ctdb_recover.c:904 startrecovery eventscript has been invoked
-2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1431 started transactions on all nodes
-2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x42fe72c5
-2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x42fe72c5 of size 0
-2008/12/01 09:57:29.187264 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x1421fb78
-2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x1421fb78 of size 0
-2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0xc0bdde6a
-2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0xc0bdde6a of size 0
-2008/12/01 09:57:29.197262 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x17055d90
-2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x17055d90 of size 8
-2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x7bbbd26c
-2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x7bbbd26c of size 1
-2008/12/01 09:57:29.207261 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0xf2a58948
-2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0xf2a58948 of size 51
-2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x92380e87
-2008/12/01 09:57:29.217259 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x92380e87 of size 17
-2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x63501287
-2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x63501287 of size 1
-2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0xe98e08b6
-2008/12/01 09:57:29.227258 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0xe98e08b6 of size 4
-2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0x2672a57f
-2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0x2672a57f of size 28
-2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1268 Recovery - pulled remote database 0xb775fff6
-2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1230 Recovery - pushed remote database 0xb775fff6 of size 6
-2008/12/01 09:57:29.237256 [ 4935]: server/ctdb_recoverd.c:1440 Recovery - starting database commits
-2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1452 Recovery - committed databases
-2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1502 Recovery - updated vnnmap
-2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1511 Recovery - updated recmaster
-2008/12/01 09:57:29.297247 [ 4935]: server/ctdb_recoverd.c:1522 Recovery - updated flags
-2008/12/01 09:57:29.305235 [ 4933]: server/ctdb_recover.c:589 Recovery mode set to NORMAL
-2008/12/01 09:57:29.307245 [ 4935]: server/ctdb_recoverd.c:1531 Recovery - disabled recovery mode
-2008/12/01 09:57:29.307245 [ 4935]: Deterministic IPs enabled. Resetting all ip allocations
-2008/12/01 09:57:29.311071 [ 4933]: takeoverip called for an ip '10.0.0.201' that is not a public address
-2008/12/01 09:57:29.311186 [ 4933]: takeoverip called for an ip '10.0.0.202' that is not a public address
-2008/12/01 09:57:29.311204 [ 4933]: takeoverip called for an ip '10.0.0.203' that is not a public address
-2008/12/01 09:57:29.311299 [ 4933]: takeoverip called for an ip '10.0.0.204' that is not a public address
-2008/12/01 09:57:29.537210 [ 4935]: server/ctdb_recoverd.c:1542 Recovery - takeip finished
-2008/12/01 09:57:29.545404 [ 4933]: Recovery has finished
-2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1551 Recovery - finished the recovered event
-2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1557 Recovery complete
-2008/12/01 09:57:29.807169 [ 4935]: server/ctdb_recoverd.c:1565 New recoveries suppressed for the rerecovery timeout
-2008/12/01 09:57:39.815648 [ 4935]: server/ctdb_recoverd.c:1567 Rerecovery timeout elapsed. Recovery reactivated.
-
-
-
-
-
-
-
-