Ronnie Sahlberg [Tue, 2 Dec 2008 03:08:10 +0000 (14:08 +1100)]
print the list of valid debug level literals when an invalid debug level
is specified in 'ctdb setdebug'
Ronnie Sahlberg [Tue, 2 Dec 2008 02:26:30 +0000 (13:26 +1100)]
redesign how reloadnodes is implemented.
modify the transport methods to allow to restart individual connections
and set up destructors properly.
only tear down/set-up tcp connections to nodes removed from the cluster
or nodes added to the cluster.
Leave tcp connections to unchanged nodes connected.
make "ctdb reloadnodes" explicitely cause a recovery of the cluster once
the files have been realoaded
root [Fri, 28 Nov 2008 00:29:43 +0000 (11:29 +1100)]
debuglevel is a signed int, not usnigned.
Ronnie Sahlberg [Thu, 27 Nov 2008 22:52:26 +0000 (09:52 +1100)]
make it possible to delete an ip from all nodes at once using
"ctdb delip x.x.x.x -n all"
This is not as straightforward as one might think since during the
delete process we don not want the ip to be bouncing from one node to
another as node by node deletes it.
Thus we first delete the ip from all connected nodes which are not
currently hosting it.
After this we delete the ip from the node which is hosting it.
Ronnie Sahlberg [Mon, 24 Nov 2008 08:06:02 +0000 (19:06 +1100)]
inew version 1.0.66
ddwq
Ronnie Sahlberg [Fri, 21 Nov 2008 05:24:12 +0000 (16:24 +1100)]
allow to change the recmaster even the database is not frozen
Ronnie Sahlberg [Fri, 21 Nov 2008 00:30:32 +0000 (11:30 +1100)]
remove two variables no longer used from the example sysconfig file
Andrew Tridgell [Thu, 20 Nov 2008 21:05:59 +0000 (08:05 +1100)]
fixed problem with looping ctdb recoveries
After a node failure, GPFS can get into a state where non-blocking
fcntl() locks can take a long time. This means to the ctdb set_recmode
test timing out, which leads to a recovery failure, and a new
recovery. The recovery loop can last a long time.
The fix is to consider a fcntl timeout as a success of this test. The
test is to see that we can't lock the shared reclock file, so a
timeout is fine for a success.
Andrew Tridgell [Thu, 20 Nov 2008 10:23:26 +0000 (21:23 +1100)]
Merge commit 'ronnie/master'
Ronnie Sahlberg [Thu, 20 Nov 2008 05:39:56 +0000 (16:39 +1100)]
dont override/change CTDB_BASE if it is already set by the shell
Ronnie Sahlberg [Thu, 20 Nov 2008 02:35:08 +0000 (13:35 +1100)]
Keepalive packets were only sent every KeepaliveInterval if the socket
had been completely idle during that interval.
If we had been sending other packets such as Messages, Calls or Controls
there wouldnt be any need for an explicit keepalive and thus we didnt
send one.
This does make it somewhat awkward when analyzing traces since it is
non-intuitive when keepalives are sent and when they are not sent.
Change the keepalive logic to always send a keepalive regardless of
whether the link is idle or not.
Ronnie Sahlberg [Wed, 19 Nov 2008 03:43:46 +0000 (14:43 +1100)]
reqrite the handling of flag updates across the cluster to eliminate a
race between the ctdb tool and the recovery daemon both at once
trying to push flag changes across the cluster.
Ronnie Sahlberg [Wed, 12 Nov 2008 23:55:20 +0000 (10:55 +1100)]
new version 1.0.65
update the example sysconfig file. the default log level is 2, not 0
Ronnie Sahlberg [Tue, 11 Nov 2008 03:49:30 +0000 (14:49 +1100)]
add a CTDB_SOCKET variable that can be used to override the default
/tmp/ctdb.socket
Ronnie Sahlberg [Mon, 3 Nov 2008 10:54:52 +0000 (21:54 +1100)]
we actually need a ctdb_db variable
Ronnie Sahlberg [Thu, 30 Oct 2008 02:34:10 +0000 (13:34 +1100)]
latency is measured in us, not ms
use an explicit ctdb_db variable instead of dereferencing state
Ronnie Sahlberg [Thu, 30 Oct 2008 01:49:53 +0000 (12:49 +1100)]
add control and logging of very high latencies.
log the type of operation and the database name for all latencies higher
than a treshold
Ronnie Sahlberg [Wed, 22 Oct 2008 00:06:18 +0000 (11:06 +1100)]
new version 1.0.64
Ronnie Sahlberg [Wed, 22 Oct 2008 00:04:41 +0000 (11:04 +1100)]
add a context and a timed event so that once we have been in recovery
mode for too long we drop all public ip addresses
Ronnie Sahlberg [Sun, 19 Oct 2008 22:47:54 +0000 (09:47 +1100)]
new version 1.0.63
Ronnie Sahlberg [Sun, 19 Oct 2008 22:45:15 +0000 (09:45 +1100)]
dont log "running periodic cleanup" ...
Ronnie Sahlberg [Fri, 17 Oct 2008 10:38:42 +0000 (21:38 +1100)]
null out the pointer before we reload the nodes file
Ronnie Sahlberg [Fri, 17 Oct 2008 10:18:06 +0000 (21:18 +1100)]
when we reload the nodes file, we may need to reload the nodes file
inside the recovery daemon as well.
Ronnie Sahlberg [Thu, 16 Oct 2008 22:02:03 +0000 (09:02 +1100)]
make it possible to set the script log level in CTDB sysconfig
Ronnie Sahlberg [Thu, 16 Oct 2008 20:56:12 +0000 (07:56 +1100)]
specify a "script log level" on the commandline to set under which log
level any/all output from eventscripts will be logged as
Ronnie Sahlberg [Thu, 16 Oct 2008 06:59:55 +0000 (17:59 +1100)]
new version 1.0.62
Ronnie Sahlberg [Thu, 16 Oct 2008 06:57:50 +0000 (17:57 +1100)]
allow multiple eventscripts using the same prefix.
this eases the pain for users that use out of tree eventscripts
Andrew Tridgell [Thu, 16 Oct 2008 01:58:25 +0000 (12:58 +1100)]
Merge commit 'ronnie/master'
Ronnie Sahlberg [Wed, 15 Oct 2008 05:40:44 +0000 (16:40 +1100)]
new version 1.0.61
Ronnie Sahlberg [Wed, 15 Oct 2008 05:29:09 +0000 (16:29 +1100)]
install the new multipath monitoring event script
Ronnie Sahlberg [Wed, 15 Oct 2008 05:27:33 +0000 (16:27 +1100)]
add an eventscript to monitor that the multipath devices are healthy
Ronnie Sahlberg [Tue, 14 Oct 2008 21:33:37 +0000 (08:33 +1100)]
we must also check the status returned from the get tickles control to
determine whether it was successful or not
Ronnie Sahlberg [Tue, 14 Oct 2008 16:02:09 +0000 (03:02 +1100)]
lower the loglevel for the informational message that a TCP_ADD opeation
described an ip address not known to be a public address.
This could happen if someone for genuine reasons accesses a share
through a static ip address.
It can also happen if non homogenous public address configurations are
used and when a tcp description is pushed out to a different node that
does not server/know the specific ip address.
Ronnie Sahlberg [Tue, 14 Oct 2008 14:49:19 +0000 (01:49 +1100)]
change ip route add to route add -net since this works more reliably
update the makefile and rpm to install 99.routing
Ronnie Sahlberg [Tue, 14 Oct 2008 14:32:46 +0000 (01:32 +1100)]
new version 1.0.60
Ronnie Sahlberg [Tue, 14 Oct 2008 14:23:57 +0000 (01:23 +1100)]
verify that the nodes we try to ban/unban are operational and print an
error to the user othervise.
Ronnie Sahlberg [Tue, 14 Oct 2008 14:08:29 +0000 (01:08 +1100)]
Revert "from Mathieu Parent <math.parent@gmail.com>"
This reverts commit
dc9cd4779db4a89697731e4cf415be51067a07c1.
Conflicts:
Ronnie Sahlberg [Tue, 14 Oct 2008 13:24:44 +0000 (00:24 +1100)]
update the client side of getnodemap and getpublicips controls to
fallback to the old-style ipv4-only controls if the new-style ipv4/ipv6
control fails.
this allows a 1.0.59+ (ipv4/ipv6) ctdb daemon being recmaster to be
compatible with
pre-1.0.59 versions of ctdb that are ipv4 only.
Ronnie Sahlberg [Mon, 13 Oct 2008 23:40:29 +0000 (10:40 +1100)]
update TAKEIP/RELEASEIP/GETPUBLICIP/GETNODEMAP controls so we retain an
older ipv4-only version of these controls.
We need this so that we are backwardcompatible with old versions of ctdb
and so that we can interoperate with a ipv4-only recmaster during a
rolling upgrade.
Ronnie Sahlberg [Sun, 12 Oct 2008 21:27:33 +0000 (08:27 +1100)]
from Mathieu Parent <math.parent@gmail.com>
Hi,
I have attached a patch necessary as debian log dir (/var/log) is not
a subdir of VARDIR (/var/lib on rpm systems, /var/lib/ctdb on debian).
As I don't know much about autotools and friends, this patch may be
hacky.
This is part of the process to minimize diff between distributions.
Ronnie Sahlberg [Sun, 12 Oct 2008 21:21:20 +0000 (08:21 +1100)]
From Mathieu Parent
patch to make debian systems log the package versions in
ctdb_diagnostics
Andrew Tridgell [Thu, 9 Oct 2008 07:45:12 +0000 (18:45 +1100)]
added some more gpfs commands per-filesystem
Ronnie Sahlberg [Tue, 7 Oct 2008 08:34:34 +0000 (19:34 +1100)]
skip empty lines in the public addresses file, not skip all non-empty
lines
Ronnie Sahlberg [Tue, 7 Oct 2008 08:25:10 +0000 (19:25 +1100)]
from Michael Adams : allow #-style comments in the nodes and public
addresses file
Ronnie Sahlberg [Tue, 7 Oct 2008 07:23:12 +0000 (18:23 +1100)]
new version 1.0.59
Ronnie Sahlberg [Tue, 7 Oct 2008 07:14:44 +0000 (18:14 +1100)]
remove an unused variable
Ronnie Sahlberg [Tue, 7 Oct 2008 07:12:54 +0000 (18:12 +1100)]
When we reload the nodes file
instead of shutting down/restarting the entire tcp layer
just bounce all outgoing connections and reconnect
Ronnie Sahlberg [Tue, 7 Oct 2008 00:03:30 +0000 (11:03 +1100)]
add a new eventscript : 99.routing that is used to add static routes to
interfaces when they are activated (an ip address is added during
takeip)
Andrew Tridgell [Tue, 30 Sep 2008 14:16:17 +0000 (07:16 -0700)]
The author of the upstream code asked for this code to be GPLv2+ not GPLv3
Andrew Tridgell [Tue, 30 Sep 2008 14:09:06 +0000 (07:09 -0700)]
merged a bugfix for the idtree code from the Linux kernel. This
matches commit
7aae6dd80e265aa9402ed507caaff4a5dba55069 in the kernel.
Many thanks to Jim Houston for pointing out this fix to us
Ronnie Sahlberg [Mon, 22 Sep 2008 15:38:28 +0000 (01:38 +1000)]
Check that a database exists first before we dump its content (and
implicitely also create it) using 'ctdb catdb'
Andrew Tridgell [Wed, 17 Sep 2008 11:00:04 +0000 (21:00 +1000)]
expanded ctdb_diagnostics based on recent experience
Ronnie Sahlberg [Wed, 17 Sep 2008 04:24:12 +0000 (14:24 +1000)]
use the correct tunable failcount not timeout
Ronnie Sahlberg [Wed, 17 Sep 2008 04:17:41 +0000 (14:17 +1000)]
The ctdb daemon keeps track of whether the recovery process is running
correctly by measuring how long it was since the last successful
communication with the recovery daemon was recorded.
After a certain timeout the ctdb daemon would deem the recovery daemon
as inoperable and shut down.
If the system clock is suddenly changed forward by many (60 or more)
seconds this could cause the timeout to trigger prematurely/immediately
where ctdb would incorrectly think that more than 60 seconds had passed
since last successful communications and thus abort.
Instead of cehcking for one timeout occuring, only deem the recovery
daemon to be "down" and trigger a shutdown if communications have
timedout for three intervals in a row.
Ronnie Sahlberg [Mon, 15 Sep 2008 23:00:48 +0000 (09:00 +1000)]
fix a slow memory leak in the recovery daemon in the error paths for the
memdump function
Ronnie Sahlberg [Mon, 15 Sep 2008 21:55:57 +0000 (07:55 +1000)]
fix some slow memory leaks in the vacuuming handler in the recovery
daemon
Ronnie Sahlberg [Mon, 15 Sep 2008 20:50:28 +0000 (06:50 +1000)]
From Volker L
Fix a slow memory leak in the recovery daemon if there is a recoery
triggered during the public ip reassignment process
Ronnie Sahlberg [Sun, 14 Sep 2008 21:04:26 +0000 (07:04 +1000)]
updates to the precompiled documentation
Martin Schwenke [Fri, 12 Sep 2008 08:20:52 +0000 (18:20 +1000)]
Document the new descriptive node specifications.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 12 Sep 2008 06:55:18 +0000 (16:55 +1000)]
onnode changes. "ok" is an alias for "healthy", "con" is an alias for
"connected". Allow "rm" or "recmaster" to be a nodespec for the
recovery master. Better error handling for interaction with ctdb
client.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 12 Sep 2008 08:21:51 +0000 (18:21 +1000)]
Merge commit 'origin/master' into for-ronnie
Ronnie Sahlberg [Fri, 12 Sep 2008 02:06:53 +0000 (12:06 +1000)]
i add a new ctdb command "ctdb recmaster"
this shows the node id of hte current recmaster
Martin Schwenke [Fri, 12 Sep 2008 01:22:50 +0000 (11:22 +1000)]
Changes to onnode. Add "healthy" and "connected" as possible
nodespecs. Since we're now explicitly using bash, use local variables
when sensible.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Martin Schwenke [Fri, 12 Sep 2008 01:26:25 +0000 (11:26 +1000)]
Merge commit 'origin/master' into for-ronnie
Martin Schwenke [Fri, 12 Sep 2008 00:36:15 +0000 (10:36 +1000)]
Minor documentation fixes.
Signed-off-by: Martin Schwenke <martin@meltin.net>
Ronnie Sahlberg [Tue, 9 Sep 2008 03:59:48 +0000 (13:59 +1000)]
lower the debuglevel when logging unknown idr in responses
Ronnie Sahlberg [Tue, 9 Sep 2008 03:55:31 +0000 (13:55 +1000)]
lower the debug level for when printing that the nodeflags have changed
Ronnie Sahlberg [Tue, 9 Sep 2008 03:44:46 +0000 (13:44 +1000)]
additional monitoring between the two daemons.
we currently only monitor that the dameons are running by kill(0, pid)
and verifying the the domain socket between them is ok.
this is not sufficient since we can have a situation where the recovery
daemon is hung.
this new code monitors that the recovery daemon is operating.
if the recovery hangs, we log this and shut down the main daemon
Ronnie Sahlberg [Sun, 7 Sep 2008 22:57:42 +0000 (08:57 +1000)]
From C Cowan.
Patch to make AIX compile with the new ipv6 additions.
Ronnie Sahlberg [Fri, 29 Aug 2008 02:26:02 +0000 (12:26 +1000)]
zero out the address structure to keep valgrind happy
Ronnie Sahlberg [Wed, 27 Aug 2008 00:26:34 +0000 (10:26 +1000)]
new version 1.0.58
Ronnie Sahlberg [Wed, 27 Aug 2008 00:24:35 +0000 (10:24 +1000)]
rename ctdb_tcp_client back to the original name ctdb_control_tcp
Ronnie Sahlberg [Mon, 25 Aug 2008 00:13:18 +0000 (10:13 +1000)]
From Abhijith Das <adas@redhat.com>:
Fixup the initscript sdo it passes rpm-lint
Ronnie Sahlberg [Mon, 25 Aug 2008 00:03:16 +0000 (10:03 +1000)]
Add a "reload" option to the initscript.
Ronnie Sahlberg [Sun, 24 Aug 2008 23:41:08 +0000 (09:41 +1000)]
add a link to my webpage
Ronnie Sahlberg [Sun, 24 Aug 2008 22:52:29 +0000 (08:52 +1000)]
version 1.0.57 : initial ipv6 support
Ronnie Sahlberg [Thu, 21 Aug 2008 23:25:47 +0000 (09:25 +1000)]
Do not fail the takeip event if the "ip addr add ..." command failed.
Let the event complete successfully. the local recovery daemon will check that we have the address and reissue takip othervise.
There are several reasons why "ip addr add " can fail, one is a misconfiguration
anothe ris that for ipv6 the stack is a lot more picky than for ipv4. for examplke this WILL fail in ipv6 if there is a duplicate ip address on the network.
thus this check could cause rolling-recoveries which is why it has to go
Ronnie Sahlberg [Thu, 21 Aug 2008 23:09:08 +0000 (09:09 +1000)]
when we collect all ip addresses and sort them for the "ctdb ip -n all" output we must look at more than just the first 4 bytes of the sockaddr address or ipv6 wont work
Ronnie Sahlberg [Wed, 20 Aug 2008 02:50:50 +0000 (12:50 +1000)]
When we harvest all tcp connections to kill off after a takeip/releaseip event we must also harvest the ipv4 connections which may be presented in ::ff:xxxx:xxxx form by netstat
Ronnie Sahlberg [Wed, 20 Aug 2008 02:02:54 +0000 (12:02 +1000)]
we must canonicalize the sockaddr structures in killtcp so that we do the necessary downgrade if required
Ronnie Sahlberg [Wed, 20 Aug 2008 01:58:27 +0000 (11:58 +1000)]
make the function to canonicalize a sockaddr structure public
Ronnie Sahlberg [Wed, 20 Aug 2008 01:52:36 +0000 (11:52 +1000)]
when we compare ip addresses in ctdb_same_ip we must first canonicalize the addresses so that we realize that 127.0.0.1:22 is really the same thing as ::ffff:127.0.0.1:22
Downgrade all AF_INET6 ::ffff:xxxx:xxxx sockaddresses into AF_INET ones
Ronnie Sahlberg [Tue, 19 Aug 2008 23:47:00 +0000 (09:47 +1000)]
update the socketkiller in the eventscripts to be able to handle ipv6
Ronnie Sahlberg [Tue, 19 Aug 2008 23:23:31 +0000 (09:23 +1000)]
fix a bug in the tcp socketkiller for ipv6
Ronnie Sahlberg [Tue, 19 Aug 2008 08:24:08 +0000 (18:24 +1000)]
fix the ipv6 checksum calculation for pseudoheader so that it actually works
add support to send ipv6 "gratious arp" aka neighbor solicitation packets from ctdb
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
Ronnie Sahlberg [Tue, 19 Aug 2008 04:58:57 +0000 (14:58 +1000)]
remove a file we dont need
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
Ronnie Sahlberg [Tue, 19 Aug 2008 04:58:29 +0000 (14:58 +1000)]
initial ipv6 patch
Signed-off-by: Ronnie Sahlberg <ronniesahlberg@gmail.com>
Ronnie Sahlberg [Thu, 14 Aug 2008 00:57:08 +0000 (10:57 +1000)]
use a local tdb_traverse instead of a ctdb_pulldb to lessen the impact of the system while performing a database backup
Ronnie Sahlberg [Wed, 13 Aug 2008 23:52:23 +0000 (09:52 +1000)]
only freeze the local node when doing a backup and not the entire cluster
Ronnie Sahlberg [Wed, 13 Aug 2008 22:36:39 +0000 (08:36 +1000)]
store the database name, not the backup filename in the database header
Ronnie Sahlberg [Wed, 13 Aug 2008 22:35:19 +0000 (08:35 +1000)]
Encode a file version number in the database backup header
Encode the database name in the header so we dont need to provide the database
name when doing a restore
Encode a timestamp in the header telling us when the backup was created
Ronnie Sahlberg [Wed, 13 Aug 2008 12:03:29 +0000 (22:03 +1000)]
Add two new ctdb commands :
ctdb backupdb : which will copy a database out from ctdb and write it to a file
ctdb restoredb : which will read a database backup from a file and write it into ctdb
Andrew Tridgell [Mon, 11 Aug 2008 14:10:48 +0000 (00:10 +1000)]
fixed merge
Andrew Tridgell [Mon, 11 Aug 2008 13:52:46 +0000 (23:52 +1000)]
up release version
Ronnie Sahlberg [Mon, 11 Aug 2008 13:50:42 +0000 (23:50 +1000)]
new version 1.0.56
Andrew Tridgell [Mon, 11 Aug 2008 13:33:46 +0000 (23:33 +1000)]
Merge commit 'ronnie/master'
Andrew Tridgell [Mon, 11 Aug 2008 13:33:05 +0000 (23:33 +1000)]
fixed a memory leak in the recovery daemon
thanks to vl for spotting this
Ronnie Sahlberg [Mon, 11 Aug 2008 00:36:38 +0000 (10:36 +1000)]
fix the date soe rpmbuild works
Ronnie Sahlberg [Mon, 11 Aug 2008 00:33:22 +0000 (10:33 +1000)]
new version 1.0.55
Andrew Tridgell [Fri, 8 Aug 2008 12:06:39 +0000 (22:06 +1000)]
fixed send of release IP message