vacuum: fix delete list counts in delete_marshall_traverse_first

when bumping skipped, decrement left, so the sum is correct

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: fix possible cause for delelete_list processing counts left records > 0

We need to have left records == 0 at the end of the delete list processing.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: systematize counters into various structs

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: remove unused counter vdata->total

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: make ctdb_process_delete_list() void

The overall return code was not really used anyways.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: make ctdb_process_vacuum_fetch_lists() void.

This constantly returns 0 anyways.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: make ctdb_vacuum_traverse_db() void.

Failure in traversal of the DB should not
prevent further processing.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: don't stop in process_vacuum_fetch_lists when sending to one node fails.

We should try to continue vacuuming as much as possible.
Failure to send records to one lmaster doesn't mean the
others will fail too.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: catch and log errors to traverse the delete list in ctdb_process_delete_list()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: catch and log error of traverse in ctdb_process_delete_queue()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: use tdb_parse_record instead of tdb_fetch in delete_marshall_traverse_first()

Spare malloc and free.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: simplify delete_marshall_traverse_first: use tdb_null

we know anyways the record to store is empty at this point.
So skip pointer calculations.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: remove VacuumLimit criterion for triggering a repack

With the new vacuuming, we consider it an error if there are
records left for deletion after processing the various lists.
All records that can be deleted should have been deleted by
tdb_delete calls.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: treat value 0 of tunable VacuumLimit as turning off repacking

I.e. no number of records found to delete will trigger the
repacking.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: add consistency check for counts at end of process_delete_list()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: log error if records are left for deletion after ctdb_process_delete_list()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: use tdb_parse_record instead of tdb_fetch in delete_record_traverse()

Spare malloc and free.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: update comment for ctdb_process_delete_queue

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: rename ctdb_vacuum_db_fast --> ctdb_process_delete_queue

This describes more precisely what this does.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: update comment for ctdb_vacuum_traverse_db

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: rename ctdb_vacuum_db_full --> ctdb_vacuum_traverse_db

This describes more precisely what it actually is, nowadays.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: change full db traverse vacuuming to fill delete queue

This lets the "fast vacuum" delete queue traverse do the actual work.

On the positive side, we note that this lets the "full vacuuming"
treat the records that have never been migrated with data correctly.
These had previously been added to the delete list for complicated
cross-node deletion instead of directly deleting them.

On the other hand side, there might be a slight overhead
since the records are read again in the delete queu traverse,
but this is OK because this change is in preparation of
untangling the db traverse altogether from the vacuum run,
making it independent.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: run the fast vacuum after the db traverse

This in preparation of modifying the db traverse to
fill the delete_queue that is processed by the fast
vacuum run, instead of filling the same lists as the
fast vacuum run for further processing.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: rename private->private_data in repack_traverse

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: rename private->private_data in vacuum_traverse

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: extract check for full vacuum run out of ctdb_vacuum_db_full()

This is more consistent.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: add consistency check for counts to ctdb_vacuum_db_fast()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: use tdb_parse_record instead of tdb_fetch in delete_queue_traverse()

this spares malloc and free

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: simplify delete_record_traverse() - free treats NULL

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: simplify delete_queue_traverse() - free treats NULL pointers.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: reduce indentation in delete_queue_traverse

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: treat value 0 of tunable RepackLimit as turned off.

I.e. when RepackLimit is set to 0, no size of the freelist
should trigger a repack in vacuuming.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: fix treatment of remaining records and statistics in delete_record_traverse()

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: cast freelist_size in comparison.

At this point, it is >= 0 anyways.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: improve output of delete list statistics

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: move retrieval of freelist to after vacuum run

The fast vacuum run may have increased the freelist size.

Signed-off-by: Michael Adam <obnox@samba.org>

vacuum: fix debug message typo in add_record_to_delete_list()

Signed-off-by: Michael Adam <obnox@samba.org>

doc: Update NEWS

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Signed-off-by: Martin Schwenke <martin@meltin.net>

doc: Fix usage string for ctdb readkey/writekey

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Fri Jan 31 07:52:46 CET 2014 on sn-devel-104

(Imported from commit 35eb6cb521d54708f0bbba515f645327846b4e70)

daemon: Return negative status only if there are known errors

If event script does not exist or does not have execute permissions, then
return negative errno to distinguish from the exit errors of event script.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 1566790e5a738f12db1dfb519589c1842d74b8e5)

tests/eventscripts: Avoid errors on broken pipe

ctdb_get_my_public_addresses() attempts to echo things and this causes
an error if head has taken the first line and the pipe is closed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jan 31 05:30:38 CET 2014 on sn-devel-104

(Imported from commit b112a3317cbedc73a6e17b3f711fec84f0d41d4e)

tests/eventscripts: Improve ip command stub secondary handling

It should support primary and secondaries per network instead of per
interface.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 1640f36d5831b2575d117fac335f3324ceefa9f8)

daemon: reloadips must register state of asynchronous controls

Otherwise ctdb_client_async_wait() is a no-op.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit e5778cc172eb9fab6382f1c600326f6cc99b9162)

tests: in the stub "ip link show" command use echo instead of cat

This case of "ip link show" does not break autobuild with
"Broken pipe" messages, but let's be consistent.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Nov 28 09:23:03 CET 2013 on sn-devel-104

(Imported from commit e2db9c524f40f8771ae19b2be47a56f7a9d887af)

test: remove unused ip2ipmask from integration.bash

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit fd5e8905a09875d13ef109133edd361a82cf8e1e)

tests:76_ctdb_pdb_recovery: change from using ctdb pstore to ctdb ptrans.

This removes the requirement to create a temporary file
and hence makes this test runnable against local daemons
and against a real cluster without further changes.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit e281cfa8db4a2506f9016718373cdc80f4aa9c1f)

tests:76_ctdb_pdb_recovery: fix a typo in a message

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 30dead171f82b5da31cbcbab88eaa70a896d9c55)

tests:76_ctdb_pdb_recovery: fix a typo in a message

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 3e083f96ff02cbf419513e16a200e8d4d0c2c227)

tests: in the stub ip command, avoid broken pipe by using echo instead of cat

This fixes running "make autotest" from autobuild, since
it prevents irritating error output in delete_ip_from_iface()
when calling ip addr list ... | grep -Fq "inet ..." .

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 70f469e05e279e29980df2af10dd89c53001b236)

tests/integration: Update NFS tickles test and supporting code

This currently requires an eventscript to be dynamically installed.
This eventscript is only used to help determine when a monitor event
has occurred.  This code is horrible and fragile.

A better way is to just monitor the output of "ctdb scriptstatus".
When changes it changes then a monitor event has occurred.

Also remove the old code that checks for tickle information in shared
storage.  CTDB hasn't done things this way for a long time.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>
(Imported from commit ef0e8cc1928dbd12c862a5e96710471ce3b4d023)

eventscripts: Do not mark node unhealthy if no fs is available

Signed-off-by: Srikrishan Malik <srimalik@in.ibm.com>
Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Jan 30 11:18:19 CET 2014 on sn-devel-104

(Imported from commit 164ee000df2a3ffc91690c60d08e4ea7ff1a33f2)

daemon: Simplify listing event scripts using scandir

Instead of using RB tree for sorting the script names (incorrectly since
it's only using the leading numbers in the script name), use scandir
with alphasort.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Tue Jan 21 06:41:25 CET 2014 on sn-devel-104

(Imported from commit eee450fec2f7cb5f45c47162fd5b7c0717978598)

daemon: Do not run monitor event if any other event is already running

Any currently running monitor events are cancelled if any other events
are scheduled.  However, this does not stop monitor events to be run
when other events are already running.

Keep track of the number of active events and schedule monitor event
only if there are no active events.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit cbffbb7c2f406fc1d8ebad3c531cc2757232690e)

eventscripts: Move all eventscript state under $CTDB_VARDIR/state

Services can be flagged for reconfigure when they release IPs at
shutdown.  The flag is never removed and the service is prematurely
reconfigured during the first "ipreallocated" event, before any IPs
are hosted and before the "startup" event has actually started the
services.

$CTDB_VARDIR/state directly contained the service state subdirectories
and is already removed in the "init" event.  Just push the service
state subdirectories down a level and put everything else in a
subdirectory.

This way all the eventscript state gets cleaned up every time CTDB
starts up.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Fri Jan 17 09:58:26 CET 2014 on sn-devel-104

(Imported from commit b7bfe46636d07c71f83daff884ec339c9b4aee72)

daemon: Untangle serialisation of 1st recovery -> startup -> monitor

At the moment ctdb_check_healthy() is overloaded to wait until the
first recovery is complete, handle the "startup" event and also
actually handle monitoring.  This is untidy and hard to follow.

Instead, have the daemon explicitly wait for 1st recovery after the
"setup" event.  When first recovery is complete, schedule a function
to handle the "startup" event.  When the "startup" event succeeds then
explicitly enable monitoring.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit e6304d1e1adc86fc9c1199feb7b4802614fbc70f)

eventscripts: Print a count if killing TCP connections times out

Also update related test

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 50e00b3e5224d53df0f3cc882e71737f928e01cd)

eventscripts: Reconfigure lock should be released quickly

Currently the lock is held until the corresponding eventscript
completes, since the process still exists.  If the regular part of an
eventscript hangs then the lock might unnecessarily be held for a long
time.  The pathological case is when a monitor event gets stuck in
D-wait state and the script times out but can't be killed so the lock
is still held.  This can cause an unwanted monitor replay.

Change this so that the lock is released immediately after the
reconfiguration is complete.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 8eb20c23476d390bb8a12ba01c9f06e7ac4a1453)

recoverd: Do not refuse disabling takeover runs on inactive nodes

Failure might be expected when disabling takeover runs on banned
nodes, since they might be suffering from performance problems or
similar.  More broadly, administrators who reconfigure a cluster that
isn't in a happy state aren't necessarily doing something sensible.

However, allowing takeover runs to be disabled on inactive nodes stops
reconfiguration of stopped nodes.  This is probaby an unreasonable
limitation, so drop it.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit e77d5f99e396d71c1d354b3f8dc5ddf9ba5c5ee9)

recoverd: Ignore failed ipreallocated controls to inactive nodes

Currently timeouts for controls to inactive nodes can cause banning
credits to be applied.  This should not happen.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit a955d0bedce888597633c0c88082f29e1d26e503)

daemon: Remove ctdb_fork_with_logging()

This function has been replaced with ctdb_vfork_with_logging().

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Thu Jan 16 04:05:35 CET 2014 on sn-devel-104

(Imported from commit a92fd11ad1ccc904a999a254d249bbdc74f08f84)

tests: Set CTDB_EVENT_HELPER when running with local daemons

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit dd98b9df6651054dabefdf439735042a78cfea2e)

daemon: Remove unused code to run eventscripts

Eventscripts are now executed using a helper.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit 97575e1ba0b7fecef2b26f2da1c0d8cb769a37a8)

daemon: Replace ctdb_fork_with_logging with ctdb_vfork_with_logging (part 2)

Use ctdb_event_helper to run debug-hung-script.sh.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit 18c1f432102f1a5093927be9276d001180539e50)

daemon: Replace ctdb_fork_with_logging with ctdb_vfork_with_logging (part 1)

Use ctdb_event_helper to run eventscripts.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit d86662a925a072eb0374ad7743f4cf95c447bebb)

daemon: Add helper process to execute event scripts

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit 69324b61f0669022c7204ee08a4c7104865d4e9b)

daemon: Add ctdb_vfork_with_logging()

This will be used to spawn lightweight helper processes to run
eventscripts.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit 2879404388ed04af199a7e4451605b4435e8cc23)

daemon: No need to call event scripts with CTDB_CALLED_BY_USER

This was added to support external monitoring using CTDB event scripts.
However, it was never used.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit 7aa20ccb5c747707fca349e9e0847cd0fca8c839)

daemon: Deprecate RELOAD and STATUS events

These events have never been used.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit bafa467021b7b2f17c61904b9f70f695a4395921)

common: mkdir_p should not try to create .

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Martin Schwenke <martin@meltin.net>
(Imported from commit b8c6bcc365ce08ddc0ebf51c002d53c08f144981)

eventscripts: Do not reconfigure in "monitor" events

"monitor" events can be cancelled.  If a reconfigure action does a
service restart then the "monitor" event can be cancelled at the
inconvenient moment after the service is stopped.  In this case the
service stays down and the node may become unhealthy (depending on
whether there are any repair actions in the monitor event).

A long time ago we did service reconfiguration in "monitor" events
following failovers.  Service reconfiguration was then moved to the
"ipreallocated" event.  However, reconfiguration in "monitor" events
has been kept as a last resort in case an "ipreallocate" event does
not occur.  The only important case that this covers is "ctdb
deleteip", where "releaseip" events are generated without a
corresponding "ipreallocated".  Therefore, IPs can be deleted without
running the required service reconfiguration.

The supported way of removing IP addresses is now via "ctdb
reloadips", which always causes a takeover run with a corresponding
"ipreallocate" event.

This means that service reconfiguration in "monitor" events is no
longer required and should be removed because it is unsafe.

Also update the associated tests.  Make the first confirm that the
monitor event no longer does reconfiguration.  Change the others to
test that monitor status is correctly replayed when something else is
doing a reconfigure and currently holds the reconfigure lock.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Dec 17 06:32:35 CET 2013 on sn-devel-104

(Imported from commit fdccaab2a9a1b9d7eebcd7a4d121dbf68ea48dcd)

packaging:RPM: don't run autogen.

autogen is already run in maketarball.sh which generates
the tarball for the RPM.

This way, we don't have a rpm build dependency on autoconf.
Recent changes introduced a dependency into autoconf
version >= 2.60, so this fix allows the generated
source RPM to be built also on older platforms.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
Autobuild-User(master): Martin Schwenke <martins@samba.org>
Autobuild-Date(master): Mon Dec  9 05:47:00 CET 2013 on sn-devel-104

(Imported from commit c65ad56d40c2ac286dc9d726119d04384981d0b3)

packaging:RPM: package the new manpages

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 7dbb068aa7e77f34377e762bbd65cb7ca72b85b4)

build: install the new manpages

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
(Imported from commit 0e8340229b0efa6291218a24865e52acb24bb12c)

Update NEWS

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

scripts: Be careful when generating unique pids for stack traces

sort expects the data to be line based, so make it so.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

config: Simplify the default CTDB configuration file

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-programmed-with: Martin Schwenke <martin@meltin.net>

scripts: Replace hard-coded /var/ctdb with CTDB_VARDIR

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

scripts: Set defaults for CTDB_DBDIR and CTDB_DBDIR_PERSISTENT

If these configuration variables are not defined, then there should
a default fallback.  This is a workaround till CTDB compile time
configuration can be accessed at runtime.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

eventscripts: Perform share check before NFS RPC checks in 60.ganesha

If NFS RPC checks do restart Ganesha, then it's possible that share
check can fail prematurely.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

tools/ctdb: Improve error checking when parsing node string

If a node isn't numeric then it is silently converted to 0.

Signed-off-by: Martin Schwenke <martin@meltin.net>

recoverd: Only respond to currently queued ipreallocated requests

Otherwise new requests can come in during the latter parts of the
takeover run when the IP allocation algorithm has already run, and the
new requests will be dequeued even though they haven't really be
processed.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

scripts: Add an early exit to statd-callout's notify case

If $statd_state is empty then the loop will run once and print
spurious errors.

Signed-off-by: Martin Schwenke <martin@meltin.net>

eventscripts: Remove the nfs_statd_update() call from 60.ganesha

Signed-off-by: Martin Schwenke <martin@meltin.net>

tests/integration: Neaten up some of the persistent database tests

Signed-off-by: Martin Schwenke <martin@meltin.net>

tools/ctdb: Fix tstore command to generate ltdb header internally

This fixes an alignment discrepancy on 32-bit vs 64-bit platforms.

  sizeof(struct ctdb_ltdb_header) = 20  (32-bit)
                                  = 24  (64-bit)

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

tests/takeover: Fix bogus test description

Signed-off-by: Martin Schwenke <martin@meltin.net>

tests/simple: User sleep_for() instead of sleep

Progress...

Signed-off-by: Martin Schwenke <martin@meltin.net>

tests/simple: Update persistent DB tests

* Low level DB checks should ignore the sequence number record.

* A restart is needed after messing with the RecoverPDBBySeqNum
  tunable.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

recoverd: For persistent databases a sequence number of 0 is valid

Otherwise recovery ends up done by RSN when it is unnecessary.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

locking: Use vfork instead of fork to exec helpers

There is a significant overhead using fork() over vfork(), specially
when the child process execs a helper.  The overhead is in memory space
and time.

    # strace -c ./test_fork 1024 200
    count=1024, size=204800, total=200M
    failed fork=0
    time for fork() = 4879.597000 us
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
    100.00    4.543321        3304      1375       375 clone
      0.00    0.000071           0      1033           mmap
      0.00    0.000000           0         1           read
      0.00    0.000000           0         3           write
      0.00    0.000000           0         2           open
      0.00    0.000000           0         2           close
      0.00    0.000000           0         3           fstat
      0.00    0.000000           0         3           mprotect
      0.00    0.000000           0         1           munmap
      0.00    0.000000           0         3           brk
      0.00    0.000000           0         1         1 access
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
    ------ ----------- ----------- --------- --------- ----------------
    100.00    4.543392                  2429       376 total

    # strace -c ./test_vfork 1024 200
    count=1024, size=204800, total=200M
    failed fork=0
    time for fork() = 82.041000 us
    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
     96.47    0.001204           1      1000           vfork
      3.53    0.000044           0      1033           mmap
      0.00    0.000000           0         1           read
      0.00    0.000000           0         3           write
      0.00    0.000000           0         2           open
      0.00    0.000000           0         2           close
      0.00    0.000000           0         3           fstat
      0.00    0.000000           0         3           mprotect
      0.00    0.000000           0         1           munmap
      0.00    0.000000           0         3           brk
      0.00    0.000000           0         1         1 access
      0.00    0.000000           0         1           execve
      0.00    0.000000           0         1           arch_prctl
    ------ ----------- ----------- --------- --------- ----------------
    100.00    0.001248                  2054         1 total

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

common: Refactor code to keep track of child processes

This code can then be used to track child processes created with vfork().

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

scripts: Run a single instance of debug_locks.sh at a give time

This prevents spamming of logs if multiple lock requests are waiting
and keep timing out.

Also, improve the logging format with separators.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

locking: Update current lock statistics when lock is scheduled

When a child process is created for a lock request, the current locks
statistics should be updated immediately.  This will provide accurate
information on number of active lock requests.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

locking: Do not merge multiple lock requests to avoid unfair scheduling

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

locking: Implement active lock requests limit per database

This limit was currently a global limit and not per database.  This
prevents any database freeze lock requests from getting scheduled if
the global limit was reached.

Only individual record requests should be limited and database freeze
requests should always get scheduled.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

scripts: Rewrite statd-callout to avoid 10 minute lag

This is naive and assumes no performance problems when updating
persistent DBs.  It also does no error handling.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

client: Treat empty __db_sequence_number__ record as 0

This fixes the issue of transaction commit failing due to an empty
__db_sequence_number__ record in persistent database left by previous
cancelled transaction.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

doc: Update ctdb.1 - primarily to add pdelete/pfetch/pstore/ptrans

Also:

* More <refentryinfo> above <refmeta> to make the XML valid.

* Describe DB argument in introduction and use it for database
  commands.

* Remove unnecessary format="linespecific" from <screen> tags, since
  it will not be allowed in DocBook 5.0.

* Sort the items in "INTERNAL COMMANDS".

* Update/simplify some command descriptions.

Signed-off-by: Martin Schwenke <martin@meltin.net>

tools/ctdb: New ptrans command

Also add test.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>

onnode: New -i option to stop stdin from being closed

This can be useful for piping data to onnode in certain circumstances.

There are now also enough command-line options that they should
definitely be alphabetically ordered.

Signed-off-by: Martin Schwenke <martin@meltin.net>