ctdb-recovery: Update timeout and number of retries during recovery

The timeout RecoverTimeout (default 120) is used for control messages
sent during the recovery.  If any of the nodes does not respond to any
of the recovery control messages for RecoverTimeout seconds, then it
will cause a failure of recovery of a database.  Recovery helper will
retry the recovery for a database 5 times.

In the worst case, if a database could not be recovered within 5 attempts,
a total of 600 seconds would have passed.  During this time period other
timeouts will be triggered causing unnecessary failures as follows:

1. During the recovery, even though recoverd is processing events,
   it does not send a ping message to ctdb daemon.  If a ping message is
   not received for RecdPingTimeout (default 60) seconds, then ctdb will
   count it as unresponsive recovery daemon.  If the recovery daemon
   fails for RecdFailCount (default 10) times, then ctdb daemon will
   restart recovery daemon.  So after 600 seconds, ctdb daemon will
   restart recovery daemon.

2. If ctdb daemon stays in recovery for RecoveryDropAllIPs (default 120),
   then it will drop all the public addresses.  This will cause all
   SMB client to be disconnected unnecessarily.  The released public
   addresses will not be taken over till the recovery is complete.

To avoid dropping of IPs and restarting recovery daemon during a delayed
recovery, adjust RecoverTimeout to 30 seconds and limit number of
retries for recovering a database to 3.  If we don't hear from a node
for more than 25 seconds, then the node is considered disconnected.
So 30 seconds is sufficient timeout for controls during recovery.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Jun  6 08:49:15 CEST 2016 on sn-devel-144

diff --git a/ctdb/server/ctdb_recovery_helper.c b/ctdb/server/ctdb_recovery_helper.c
index 0720d0e2ca6806f0e57b29760e1d444e470c29f4..d54f32db04ed17aec246e94f6ae5eedc5b9486b2 100644 (file)
--- a/ctdb/server/ctdb_recovery_helper.c
+++ b/ctdb/server/ctdb_recovery_helper.c
@@ -34,9 +34,9 @@
 #include "protocol/protocol_api.h"
 #include "client/client.h"
 
-static int recover_timeout = 120;
+static int recover_timeout = 30;
 
-#define NUM_RETRIES    5
+#define NUM_RETRIES    3
 
 #define TIMEOUT()      timeval_current_ofs(recover_timeout, 0)
 

diff --git a/ctdb/server/ctdb_tunables.c b/ctdb/server/ctdb_tunables.c
index 83b57f7ed2be9d43ad6665110a99a7626a00c60f..9c1e4a9d8e11d8ecb0c8842a1166229f5d4c1d08 100644 (file)
--- a/ctdb/server/ctdb_tunables.c
+++ b/ctdb/server/ctdb_tunables.c
@@ -41,7 +41,7 @@ static const struct {
        { "TraverseTimeout",     20, offsetof(struct ctdb_tunable_list, traverse_timeout), false },
        { "KeepaliveInterval",    5,  offsetof(struct ctdb_tunable_list, keepalive_interval), false },
        { "KeepaliveLimit",       5,  offsetof(struct ctdb_tunable_list, keepalive_limit), false },
-       { "RecoverTimeout",     120,  offsetof(struct ctdb_tunable_list, recover_timeout), false },
+       { "RecoverTimeout",      30,  offsetof(struct ctdb_tunable_list, recover_timeout), false },
        { "RecoverInterval",      1,  offsetof(struct ctdb_tunable_list, recover_interval), false },
        { "ElectionTimeout",      3,  offsetof(struct ctdb_tunable_list, election_timeout), false },
        { "TakeoverTimeout",      9,  offsetof(struct ctdb_tunable_list, takeover_timeout), false },