fixed problem with looping ctdb recoveries

After a node failure, GPFS can get into a state where non-blocking
fcntl() locks can take a long time. This means to the ctdb set_recmode
test timing out, which leads to a recovery failure, and a new
recovery. The recovery loop can last a long time.

The fix is to consider a fcntl timeout as a success of this test. The
test is to see that we can't lock the shared reclock file, so a
timeout is fine for a success.

diff --git a/server/ctdb_recover.c b/server/ctdb_recover.c
index c8b0ba066abce420520a9d7d3ef1e1e7eab65453..39b73acff56bd12d09d746edcbe7270df4c59455 100644 (file)
--- a/server/ctdb_recover.c
+++ b/server/ctdb_recover.c
@@ -477,7 +477,14 @@ static void ctdb_set_recmode_timeout(struct event_context *ev, struct timed_even
        struct ctdb_set_recmode_state *state = talloc_get_type(private_data, 
                                           struct ctdb_set_recmode_state);
 
-       ctdb_request_control_reply(state->ctdb, state->c, NULL, -1, "timeout in ctdb_set_recmode");
+       /* we consider this a success, not a failure, as we failed to
+          set the recovery lock which is what we wanted.  This can be
+          caused by the cluster filesystem being very slow to
+          arbitrate locks immediately after a node failure.       
+        */
+       DEBUG(DEBUG_NOTICE,(__location__ " set_recmode timeout - allowing recmode set\n"));
+       state->ctdb->recovery_mode = state->recmode;
+       ctdb_request_control_reply(state->ctdb, state->c, NULL, 0, NULL);
        talloc_free(state);
 }
 
@@ -643,7 +650,7 @@ int32_t ctdb_control_set_recmode(struct ctdb_context *ctdb,
        talloc_set_destructor(state, set_recmode_destructor);
 
        state->te = event_add_timed(ctdb->ev, state, timeval_current_ofs(3, 0),
-                       ctdb_set_recmode_timeout, state);
+                                   ctdb_set_recmode_timeout, state);
 
        state->fde = event_add_fd(ctdb->ev, state, state->fd[0],
                                EVENT_FD_READ|EVENT_FD_AUTOCLOSE,