Ronnie Sahlberg [Mon, 31 Jan 2011 06:48:22 +0000 (17:48 +1100)]
New version 1.2.19
Ronnie Sahlberg [Mon, 31 Jan 2011 06:40:26 +0000 (17:40 +1100)]
If the node is stopped, put a log entry in /var/log/* to indicate this is why we never become ready
Ronnie Sahlberg [Mon, 24 Jan 2011 00:42:50 +0000 (11:42 +1100)]
LockWait congestion.
Add a dlist to track all active lockwait child processes.
Everytime creating a new lockwait handle, check if there is already an
active lockwait process for this database/key and if so,
send the new request straight to the overflow queue.
This means we will only have one active lockwaic child process for a certain key,
even if there were thousands of fetch-lock requests for this key.
When the lockwait processing finishes for the original request, the processing in d_overflow() will automagically process all remaining keys as well.
Add back a --nosetsched argument to make it easier to run under gdb
Ronnie Sahlberg [Sun, 23 Jan 2011 22:43:45 +0000 (09:43 +1100)]
Compile fix
Rusty Russell [Fri, 21 Jan 2011 10:47:02 +0000 (21:17 +1030)]
ctdb_lockwait: create overflow queue.
Once we have more than 200 children waiting on a particular db, don't create
any more. Just put them on an overflow queue, and when a child gets a lock
search that queue to see if others were after the same lock (they probably
were).
Ronnie Sahlberg [Sun, 23 Jan 2011 20:39:33 +0000 (07:39 +1100)]
Add a new test tool that fetch locks a record and then blocks until it receives
user input to unlock the record again.
Ronnie Sahlberg [Thu, 20 Jan 2011 23:56:56 +0000 (10:56 +1100)]
60.nfs
Dont update the statd settings that often.
When we have very many nodes and very many ips, this would generate
a lot of unnessecary load on the system
Ronnie Sahlberg [Tue, 18 Jan 2011 21:00:36 +0000 (08:00 +1100)]
TDB : Fix for a deadlock with transaction lock and lockall/lockallmark
causing ctdbd hangs
Ronnie Sahlberg [Tue, 18 Jan 2011 02:33:24 +0000 (13:33 +1100)]
ctdb: hold transaction locks during freeze, mark during recover.
Make the ctdb parent "mark" the transaction lock once the child process
has frozen/locked the entire database.
This stops the ctdb daemon from using a blocking fcntl() locking on the tdb during the
read traverse during recovery.
CQ
1021388
Rusty Russell [Tue, 18 Jan 2011 00:17:11 +0000 (10:47 +1030)]
tdb: expose transaction lock infrastructure for ctdb
tdb_traverse_read() grabs the transaction lock. This can cause ctdbd
(which uses it) to block when it should not; expose mark and normal
variants of this lock, so ctdbd's child (the recovery daemon) can
acquire it and the ctdbd parent can mark it was held.
Ronnie Sahlberg [Mon, 17 Jan 2011 01:05:43 +0000 (12:05 +1100)]
New version 1.2.17
Ronnie Sahlberg [Mon, 17 Jan 2011 01:00:18 +0000 (12:00 +1100)]
change Christinas previous patch to only perform the check/logging
if we are the main ctdb daemon.
Other daemons/child processes are not guaranteed to get events on regular basis
so those should not be checked.
Christian Ambach [Fri, 14 Jan 2011 12:55:28 +0000 (13:55 +0100)]
improve timing issue detections
the original "Time jumped" messages are too coarse to interpret
exactly what was going wrong inside of CTDB.
This patch removes the original logs and adds two other logs that
differentiate between the time it took to work on an event and
the time it took to get the next event.
Ronnie Sahlberg [Fri, 14 Jan 2011 06:35:31 +0000 (17:35 +1100)]
LIBCTDB: add support for traverse
Ronnie Sahlberg [Thu, 13 Jan 2011 22:46:04 +0000 (09:46 +1100)]
We can not always rely on the recovery daemon pinging us in a timely manner
so we need a "ticker" in the main ctdbd daemon too to ensure we get at least one event to process every second.
This will improve the accuracy of "Time jumped" messages and remove false positives when the recovery daemon is "slow".
Ronnie Sahlberg [Thu, 13 Jan 2011 05:17:43 +0000 (16:17 +1100)]
ADDIP failure
Found during automatic regression testing.
We do not allow the takeip/releaseip events to be executed during a recovery.
All of "ctdb addip, ctdb delip, ctdb moveip" use and force these events to
trigger to perform the ip assignments required.
If these commands collide with a recovery, these commands could fail since we do
not allow takeip/releaseip events to trigger during the recovery.
While it is easy to just try running hte command again, this is suboptimal for script use.
Change these commands to retry these operations a few times until either successfull or until we give up.
This makes the commands much easier to use in scripts.
Ronnie Sahlberg [Wed, 12 Jan 2011 22:35:37 +0000 (09:35 +1100)]
IPALLOCATION : If the node is held pinned down in "init" state
by external services failing to start, or blocking CTDBD from finishing the startup phase,
we can encounter a situation where we have not yet fully initialized, but a
remote recovery master tries to release a certain ip clusterwide.
In this situation the node that is pinned down in init/startup phase
would fail to perform the release of the ip address since we are not yet fully operational and not yet host any valid interfaces.
In this situation, we just need to remain unhealthy, there is on need to
also ban the node.
Remove the autobanning for this condition and just let the node remain in
unhealthy mode.
Banning is overkill in this situation when the system is broken and just
draws attention to ctdbd instead of the root cause.
Martin Schwenke [Tue, 11 Jan 2011 06:13:57 +0000 (17:13 +1100)]
Eventscripts: lower the fail/restart limits for nfsd.
We were potentially leaving a node unable to serve requests for too
long.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 11 Jan 2011 06:13:06 +0000 (17:13 +1100)]
Eventscripts: use "startstop_nfs restart" to reconfigure NFS.
This was defaulting to just "service nfs restart", which doesn't have
the workarounds we need.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 11 Jan 2011 06:12:03 +0000 (17:12 +1100)]
Eventscripts: only autostart during a monitor event.
Otherwise we might short-circuit events that are run only once and
actually need to do something.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 11 Jan 2011 06:10:55 +0000 (17:10 +1100)]
Eventscripts: print a message when reconfiguring a service.
Otherwise there can be strange error messages from services
stopping/starting, without any context.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 11 Jan 2011 06:06:48 +0000 (17:06 +1100)]
Eventscripts: work around NFS restart failure under load.
"service nfs restart" can fail. To stop nfsd it sends a SIGINT and
nfsd might take a while to process it if the system is loaded.
Starting nfsd may then fail because resources are still in use.
This does some /proc magic to tell nfsd to do no more processing. It
then runs service stop, kills nfsd with SIGKILL, and then runs service
start. This is much less likely to fail.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Tue, 11 Jan 2011 05:17:06 +0000 (16:17 +1100)]
TYPO
Ronnie Sahlberg [Tue, 11 Jan 2011 05:15:41 +0000 (16:15 +1100)]
STATD is 100027 not
1000247
Ronnie Sahlberg [Mon, 10 Jan 2011 20:37:17 +0000 (07:37 +1100)]
LIBCTDB uninitialized inqueue element
From Michael Anderson,
initialize the inqueue element of the ctdb structure to NULL,
else it might be used uninitialized and cause a segv.
Ronnie Sahlberg [Mon, 10 Jan 2011 05:51:56 +0000 (16:51 +1100)]
recoverd: avoid triggering a full recovery if just some ip allocation
has failed.
We dont need to rebuild the databases in this situation, we just
need to try again to sort out the ip address allocations.
Ronnie Sahlberg [Mon, 10 Jan 2011 02:57:49 +0000 (13:57 +1100)]
Add ctdb_fork(0 which will fork a child process and drop the real-time
scheduler for the child.
Use ctdb_fork() from callers where we dont want the child to be running
at real-time privilege.
Ronnie Sahlberg [Mon, 10 Jan 2011 02:35:39 +0000 (13:35 +1100)]
Revert scheduling back to use real-time processes
Revert this patch:
commit
482c302d46e2162d0cf552f8456bc49573ae729d
We may need to use real-time processes for the main daemon and the recovery daemon to handle the cases where systems come under very high loads.
Ronnie Sahlberg [Thu, 6 Jan 2011 04:42:45 +0000 (15:42 +1100)]
60.nfs Check if we have rpc.statd and if not, skip checking for statd
availability at all (since we cant restart it, there is not point checking
if it is alive)
Ronnie Sahlberg [Tue, 21 Dec 2010 23:39:25 +0000 (10:39 +1100)]
New version 1.2.16
- 50.samba dont run serverid wipe in the background in case it
is so slow to start that samba manages to come up before it finishes.
- 60.nfs wait 10 intervals before trying to restart lockd.
flag the node unhealthy after 15 failures.
CQ S1021266
- 41.httpd httpd can sometimes be slow, wait 5 intervals before we try to
restart it and 10 intervals before we flag the node unhealthy.
Ronnie Sahlberg [Tue, 21 Dec 2010 23:27:53 +0000 (10:27 +1100)]
41.HTTPD
Httpd can be very slow to start on some platforms,
wait 5 monitor intervals before we try to restart it if
it has not bound to port 80 yet.
After 10 failed intervals, flag the node as unhealthy.
Ronnie Sahlberg [Tue, 21 Dec 2010 23:09:35 +0000 (10:09 +1100)]
60.nfs
Try to restart LOCKD after 10 failures and
flag the node as unhealthy after 15 failures
Ronnie Sahlberg [Tue, 21 Dec 2010 23:05:40 +0000 (10:05 +1100)]
Dont run net serverid wipe in the background
Ronnie Sahlberg [Tue, 14 Dec 2010 10:17:14 +0000 (21:17 +1100)]
50.samba
Net serverid wipe can take a bit of time sometimes so background it.
Only perform auto start/stop of the managed service on the monitor event
Ronnie Sahlberg [Mon, 13 Dec 2010 03:20:37 +0000 (14:20 +1100)]
New version 1.2.15
* Mon Dec 13 2010 : Version 1.2.15
- Add two new debugging commands "ctdb readkey/writekey"
- idtree overflow bugfix
- only run "serverid wipe" when we are actually running samba
- libctdb, add roper input queueing so we can support calling
sync functions from an async callback
- lvs updates
- addip, always wait across at least one ip reallocation, making the
command slower, but making it easier to use in scripts
Ronnie Sahlberg [Mon, 13 Dec 2010 01:39:01 +0000 (12:39 +1100)]
Revert "server: when we migrate off a record with data, set the MIGRATED_WITH_DATA flag"
This reverts commit
c63dab9763d45fd4f9be77b9c9f463bd457de808.
Ronnie Sahlberg [Mon, 13 Dec 2010 01:38:39 +0000 (12:38 +1100)]
Revert "Add a new header flag for "migrated with data" and set this to 1"
This reverts commit
d22e7e47a7f3d450bbbc2267322dadbdbf192e84.
Ronnie Sahlberg [Mon, 13 Dec 2010 01:06:01 +0000 (12:06 +1100)]
ctdb addip:
After finishing "ctdb addip" wait for an implicit "iptakeover" to complete
the assignment to a node.
This makes it more wasteful and timeconsuming when adding multiple ips
at once, or the same ip to multiple nodes,
but makes it easier to script the use of this command.
Ronnie Sahlberg [Sun, 12 Dec 2010 08:38:39 +0000 (19:38 +1100)]
LVS
update lvs configuration on ipreallocated events too
Ronnie Sahlberg [Sun, 12 Dec 2010 03:22:20 +0000 (14:22 +1100)]
When assigning the single-public-ip during startup,
flag the interface as initially being "link ok"
so that we can add it and startup.
The eventscript can later drop the flag if required
Ronnie Sahlberg [Fri, 10 Dec 2010 03:18:28 +0000 (14:18 +1100)]
libctdb
fix a compile problem after renaming a structure field
Ronnie Sahlberg [Fri, 10 Dec 2010 02:39:18 +0000 (13:39 +1100)]
LibCTDB
Add an input queue where we keep received pdus we have not yet processed
This allows us to perform SYNC calls from an ASYNC callback
Ronnie Sahlberg [Wed, 8 Dec 2010 00:08:19 +0000 (11:08 +1100)]
only run "serverid wipe" if we are actually running samba.
we dont need to run this on systems where we do run winbind but not samba
Rusty Russell [Mon, 6 Dec 2010 03:22:38 +0000 (13:52 +1030)]
idtree: fix overflow for v. large ids on allocation and removal
(Imported from SAMBA commit
09a6538969ac).
Chris Cowan tracked down a SEGV in sub_alloc: idp->level can actually
be equal to 7 (MAX_LEVEL) there, as it can be in sub_remove.
(We unfairly blamed a shift of a signed var for this crash in commit
2db1987f5a3a).
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Ronnie Sahlberg [Mon, 6 Dec 2010 05:09:38 +0000 (16:09 +1100)]
Add a new header flag for "migrated with data" and set this to 1
when we migrate a non-empty record onto the node
or a non-empty record off the node
When we migrate a record back to the lmaster and yield the dmaster role,
inspect this flag if if it is still not set, we can delete the record from
the local database as soon as we have migrated it back to the lmaster.
Ronnie Sahlberg [Mon, 6 Dec 2010 05:07:55 +0000 (16:07 +1100)]
add new command line functions
ctdb readkey <dbid> <key>
ctdb writekey <dbid> <key> <value>
these are mainly intended for debugging of databases and dmaster migration issues
Ronnie Sahlberg [Mon, 6 Dec 2010 05:06:20 +0000 (16:06 +1100)]
add a new ctdb_ltdb function to delete a record in a normal database
Michael Adam [Fri, 3 Dec 2010 14:21:51 +0000 (15:21 +0100)]
server: when we migrate off a record with data, set the MIGRATED_WITH_DATA flag
Ronnie Sahlberg [Mon, 6 Dec 2010 07:34:49 +0000 (18:34 +1100)]
new version 1.2.14
Ronnie Sahlberg [Mon, 6 Dec 2010 02:08:53 +0000 (13:08 +1100)]
Add two new flags for the ltdb header.
One of which signals that the record has never been migrated to/from a node
while containing data.
This property "has never been migrated while non-zero" is important later
to provide heuristics on which records we might be able to purge
from the tdb files cheaply, i.e. without having to rely on the full-blown
database vacuum.
These records are belived to be very common and the pattern would look like
this :
1, no record exists at all.
2, client opens a file
3, samba requests the record for this file
4, an empty record is created on the LMASTER
5, the empty record is migrated to the DMASTER
6, samba writes a <sharemode> to the record locally and the record grows
7, client finishes working the file and closes the file
8, samba removes the sharemode and the record becomes empty again.
9, much later : vacuuming will delete the record
At stage 8, since the record has never been migrated onto a node wile being
non-zero it would be safe, and much more efficient to just delete the record
completely from the database and hand it back to the LMASTER.
The flags occupy the same uint32_t as was previously used for laccessor/lacount
in the header. For now, make sure the flags only define/use the top 16 bits
of this field so that we are sure we dont collide with bits set to one
from previous generations of the ctdb cluster database prior to this
change in semantics of this word.
This is a rework of Michaels patch :
commit
2af1a47cbe1a608496c8caf3eb0c990eb7259a0d
Author: Michael Adam <obnox@samba.org>
Date: Tue Nov 30 17:00:54 2010 +0100
add a DEFAULT record flag and a MIGRATED_WITH_DATA record flag.
Ronnie Sahlberg [Mon, 6 Dec 2010 02:04:44 +0000 (13:04 +1100)]
change one of the reserved words in the ctdb ltdb header to be a flags field
for now, try avoiding using bits in the low16 bits as flags since this may
collide with laccessor/lacount values from previous versions of the cluster
databases
Ronnie Sahlberg [Mon, 29 Nov 2010 02:07:59 +0000 (13:07 +1100)]
Remove LACOUNT and LACCESSOR and migrate the records immediately.
This concept didnt work out and it is really just as expensive as a full migration
anyway, without the benefit of caching the data for subsequence accesses.
Now, migrate the records immediately on first access.
This will be combined with a "cheap vacuum-lite" for special empty records to
prevent growth of databases.
Later extensions to mimic read-only behaviour of records will include proper shared read-only locking of database records, making the laccessor/lacount read-only access to the data obsolete anyway.
By removing this special case and handling of lacount laccessor makes the codapath where shared read-only locking will be be implemented simpler, and frees up space in the ctdb_ltdb header for use by vacuuming flags as well as read-only locking flags.
Ronnie Sahlberg [Mon, 6 Dec 2010 00:30:24 +0000 (11:30 +1100)]
Add 60.ganesha to what gets installed by make install as well as by the RPM
Ronnie Sahlberg [Mon, 6 Dec 2010 00:26:43 +0000 (11:26 +1100)]
add a missing part of the import of the previous ganesha patch
Chandra Seetharaman [Fri, 3 Dec 2010 23:26:22 +0000 (15:26 -0800)]
make changes to ctdb event scripts to support NFS-Ganesha.
make changes to ctdb event scripts to support NFS-Ganesha.
Signed-off-by: Chandra Seetharaman <sekharan@us.ibm.com>
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
Ronnie Sahlberg [Fri, 3 Dec 2010 02:28:35 +0000 (13:28 +1100)]
during ip allocation, there are failure modes where a node might hold a ip address
but thinks it is still unassigned (-1).
add code to the recovery daemon to detect this case and trigger a reallocation
so that the ip gets covered
and change the takeip code to allow for this condition, taking on an ip address that is
already hosted.
cq s1021073
Ronnie Sahlberg [Thu, 2 Dec 2010 19:08:44 +0000 (06:08 +1100)]
new version 1.2.13
Ronnie Sahlberg [Thu, 2 Dec 2010 19:07:03 +0000 (06:07 +1100)]
dont try starting samba through the "init" event
Ronnie Sahlberg [Mon, 29 Nov 2010 08:31:05 +0000 (19:31 +1100)]
new version 1.2.12
Ronnie Sahlberg [Mon, 29 Nov 2010 01:39:14 +0000 (12:39 +1100)]
When we are no longer the natgw master, dont put the natgw ip on loopback.
We put the ip on loopback just to make sure we would still interoperate with
non-standard configurations on unix-KDC, that are configured to verify the optional
HostAddresses field.
This is not required for AD, since AD does not use this field, and is replaced in
unix land with other/better mechanisms than this "dodgy" check.
This makes it "easier" for applications that have bound to the natgw address
to detect a socket problem and try to reconnect/recover if the ip address
is completely missing from the system.
At the same time, use the winbind specific hook that exists to explicitely tell winbindd : this address is gone, so if you have bound to it, this is a good time to close and rebind your socket.
cq
1020333
Ronnie Sahlberg [Mon, 22 Nov 2010 09:57:27 +0000 (20:57 +1100)]
new version 1.2.11
Ronnie Sahlberg [Thu, 18 Nov 2010 04:40:19 +0000 (15:40 +1100)]
update autostart/stop to work for samba
Ronnie Sahlberg [Thu, 18 Nov 2010 03:15:18 +0000 (14:15 +1100)]
add an explicit _is_managed_service to iscsi eventscript
Ronnie Sahlberg [Thu, 18 Nov 2010 02:52:46 +0000 (13:52 +1100)]
Dont pollute the logs with a "file not found" message
CQ S1020745
Martin Schwenke [Thu, 18 Nov 2010 02:23:40 +0000 (13:23 +1100)]
60.nfs eventscript should do nothing if NFS isn't managed by CTDB.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Thu, 18 Nov 2010 00:27:10 +0000 (11:27 +1100)]
Eventscript functions - catch failures in ctdb_service_start().
ctdb_service_start() currently succeeds if ctdb_counter_init()
succeeds.
This changes it to fail when a service start fails.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Thu, 18 Nov 2010 00:04:52 +0000 (11:04 +1100)]
50.samba eventscript should stop/start services when they become (un)managed.
When the value of $CTDB_MANAGES_SAMBA or $CTDB_MANAGES_WINBIND (or
corresponding changes are made to $CTDB_MANAGED_VERSIONS), the
associated service should be started or stopped as necessary.
This add calls to ctdb_start_stop_service() to manage
starting/stopping samba and winbind.
An associated cleanup is made to the initial checks that one of
$CTDB_MANAGES_SAMBA or $CTDB_MANAGES_WINBIND is set, replacing them
with calls to is_ctdb_managed_service().
To handle the winbind cases ctdb_start_stop_service() and
is_ctdb_managed_service() are updated to take an optional service name
parameter.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Wed, 17 Nov 2010 02:50:56 +0000 (13:50 +1100)]
add a new support function ctdb_check_counter_equal()
update nfs to try to restart the service after 10 consecutive failures
and to flag the node unhealthy after 15
add similar function to mountd
Martin Schwenke [Tue, 31 Aug 2010 07:40:40 +0000 (17:40 +1000)]
Eventscripts: make loadconfig() function hookable by the test suite.
Rename loadconfig() to _loadconfig(). Add a new loadconfig() that
simply calls _loadconfig().
This makes it easy for the test suite to override loadconfig().
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 16 Nov 2010 08:42:31 +0000 (19:42 +1100)]
Make a time comparison in 60.nfs eventscript more readable.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 16 Nov 2010 08:31:18 +0000 (19:31 +1100)]
60.nfs only fails or warns after 10 consecutive nfsd/statd failures.
These failures are sometimes the result of slow restarts so we want to
avoid dirtying the logs or marking a node unhealthy because of them,
unless they are excessive.
For these 2 cases we use the existing fail counting code but hack a
temporary service_name in a subshell to allow separate fail counts.
We also update ctdb_check_rpc() so that it captures the error output
from rpcinfo and we add a message including the service name to the
beginning. The error is printed to stdout but is also stored in
ctdb_check_rpc_out to allow it to be conditionally used by the caller.
This function also now returns non-zero rather than exiting on
failure.
Other direct rpcinfo calls are relaced by called to ctdb_check_rpc()
for consistency.
Option handling code for service restarts is cleaned up so that fits
in 80 columns. A more informative restart messageis now used in all
cases, printing the exact command being used to start a service.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Tue, 12 Oct 2010 00:10:38 +0000 (11:10 +1100)]
Test suite: fix typo in ctdb ping test grep pattern.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Wed, 6 Oct 2010 05:32:22 +0000 (16:32 +1100)]
Test suite: match changed output for ctdb ping to disconnected node.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 15 Oct 2010 04:09:08 +0000 (15:09 +1100)]
Test suite: make statistics test cope with changes to statistics output.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Mon, 15 Nov 2010 05:47:22 +0000 (16:47 +1100)]
New version 1.2.10
* Mon Nov 15 2010 : Version 1.2.10
- Make sure to initialize the statistics start time to current time
instead of leaving it to point to start of epoch.
CQ : S1020838
- Create a new tunable DisableIPFailover that is used to tell ctdb
to not check any ip allocation at all and never do any failover
This can be used to stop/restart individual nodes without causing
any ip failovers to happen.
Ronnie Sahlberg [Mon, 15 Nov 2010 05:30:44 +0000 (16:30 +1100)]
initialize the statistics to the current time, not start of epoch
this makes "ctdb statistics" show correct "start of starts collection"
Ronnie Sahlberg [Wed, 10 Nov 2010 03:47:28 +0000 (14:47 +1100)]
Dont exit the update ip function if the old and new interfaces are the same
since if they are the same for whatever reason this triggers the system
to go into an infinite loop and is unrobust
The scriptds have been changed instead to be able to cope with this
situation for enhanced robustness
During takeover_run and when merging all ip allocations across the cluster
try to kepe track of when and which node currently hosts an ip address
so that we avoid extra ip failovers between nodes
Ronnie Sahlberg [Wed, 10 Nov 2010 03:46:45 +0000 (14:46 +1100)]
change the takeover script timeout to 9 seconds from 5
Ronnie Sahlberg [Wed, 10 Nov 2010 03:46:05 +0000 (14:46 +1100)]
Dont check remote ip allocation if public ip mgmt is disabled
Ronnie Sahlberg [Wed, 10 Nov 2010 03:45:43 +0000 (14:45 +1100)]
this stuff is just so fragile that it will enter infinite recovery and fail loops
on any kind of tiny unexpected error
unconditionally try to remove ip addresses from both old and new interface
before trying to add it to the new interface to make it less
fragile
Ronnie Sahlberg [Wed, 10 Nov 2010 03:40:43 +0000 (14:40 +1100)]
delete from old interface before adding to new interface
this stops the script from failing with an error if
both interfaces are specified as the same, which otherwise breaks and leads to an infinite recovery loop
Ronnie Sahlberg [Wed, 10 Nov 2010 01:59:25 +0000 (12:59 +1100)]
delay loading the public ip address file until after we have started the transport and discovered ouw own pnn number
Ronnie Sahlberg [Wed, 10 Nov 2010 01:11:11 +0000 (12:11 +1100)]
when we load the public address file, at the same time check if we are already hosting the public address, if so, set ourselves up as the pnn for that address
Ronnie Sahlberg [Wed, 10 Nov 2010 01:06:05 +0000 (12:06 +1100)]
dont check the public ip assignment or if even we are hosting them and shouldnt
when public ips have been disabled
Ronnie Sahlberg [Tue, 9 Nov 2010 04:19:06 +0000 (15:19 +1100)]
Add a new tunable : DisableIPFailover that when set to non 0
will stopp any ip reallocations at all from happening.
Ronnie Sahlberg [Tue, 9 Nov 2010 01:59:05 +0000 (12:59 +1100)]
change the default for how long to waqit before dropping all ips to 120 seconds
Ronnie Sahlberg [Tue, 9 Nov 2010 01:56:02 +0000 (12:56 +1100)]
dont delete all ips from the system during the initial "init" event
leave any ips as they are and let the recovery daemon remove them as required
Ronnie Sahlberg [Tue, 9 Nov 2010 01:55:20 +0000 (12:55 +1100)]
when creating/adding a public ip, set the initial interface to be the first interface specified
Ronnie Sahlberg [Tue, 2 Nov 2010 09:11:09 +0000 (20:11 +1100)]
New version 1.2.9
* Tue Nov 2 2010 : Version 1.2.9
- Drop loglevels on several items and remove spam from the messages file
- Both nfs and nfslock can fail so restart both if there is a problem
Ronnie Sahlberg [Thu, 28 Oct 2010 02:43:57 +0000 (13:43 +1100)]
Both nfs and nfslock scripts can fail under redhat in very rare situations.
Ctdb can also be configured to ignore checking for knfsd and if it is alive.
In that situation, no attempt will be made to restart nfs, and sicne nfs is not running, lockd can not be restarted either.
To workaround this, everytime we try to restart the lockmanager, also try to restart nfsd
Ronnie Sahlberg [Thu, 28 Oct 2010 02:38:34 +0000 (13:38 +1100)]
during shutdown there is a window after we have stopped TCP and disconnected from all other nodes but before we have stopped all processing.
During this window we may still hit asynchronous events that will fail because we can not send/receive packets from other nodes.
These messages are logged as ... Transport is DOWN. To help indicate that they are benign messages related to the process of shutting down.
These messages spam the syslog during normal shutdown, so this patch will drop the loglevel of these messages to DEBUG, so that they will not appear in or spam the syslog.
Ronnie Sahlberg [Thu, 28 Oct 2010 02:36:24 +0000 (13:36 +1100)]
When shuttind down, we always unconditionally try to remove the natgw address
even if we are not currently the natgw master.
This adds extra reliability in case we have stopped previously without removing it proper,
but does add spam messages to syslog everytime we shutdowm.
Remove these spam messages from pulluting the syslog upon normal shutdown
Ronnie Sahlberg [Thu, 28 Oct 2010 02:34:33 +0000 (13:34 +1100)]
Redirect the output from 00.ctdb pfetch to stdout.
Normally, the config.tdb database would not exist, so we do not need
to spam syslog with a "config.tdb does not exist" message every time we start ctdb
Ronnie Sahlberg [Thu, 28 Oct 2010 02:32:29 +0000 (13:32 +1100)]
Drop the loglevel of the "reqid wrap" developer debug message to DEBUG
so that we dont spam the logs with this normal benign message.
Ronnie Sahlberg [Mon, 25 Oct 2010 08:49:19 +0000 (19:49 +1100)]
new version 1.2.8
Ronnie Sahlberg [Mon, 25 Oct 2010 00:31:12 +0000 (11:31 +1100)]
Add support to create TDB databases using the new jenkins hash.
SRVID for the control to attach to a database is used to pass
tdb flags from samba to ctdb when samba attached to a database.
This has been used earlier for TDB_NOSYNC flag.
Add TDB_INCOMPATIBLE_HASH as a supported tdb flag to store in the
SRVID field when attaching to a database.
This allows samba to control if ctdb should create databases using the
new jenkins hash, or using the old hash.
This only affects new databases when they are initially created.
Existing databases remain using the old hash when attached to.
Ronnie Sahlberg [Mon, 18 Oct 2010 04:58:03 +0000 (15:58 +1100)]
New version 1.2.7
- Dont monitor GPFS filesystems in 62.cnfs
- If tdb_open() fails, print errno to make troubleshooting easier
- Try restarting RPC.LOCKD if it failed to start
- Remove a dbug message
- Make sure the statd state directory exists before trying to touch files in
Ronnie Sahlberg [Mon, 18 Oct 2010 00:57:38 +0000 (11:57 +1100)]
remove checking for filesystems and filesystem health from the cnfs script.
remove the gpfsmount and gpfsumount entry points
Ronnie Sahlberg [Wed, 13 Oct 2010 22:49:23 +0000 (09:49 +1100)]
If tdb_open() fails when trying to open the vacuuming database,
print errno so we get some idea of why this failed.
Ronnie Sahlberg [Wed, 13 Oct 2010 21:12:41 +0000 (08:12 +1100)]
try to restart NFS LOCKD if it failed to start