1 <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>ctdb</title><meta name="generator" content="DocBook XSL Stylesheets V1.79.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="refentry"><a name="ctdb.1"></a><div class="titlepage"></div><div class="refnamediv"><h2>Name</h2><p>ctdb — CTDB management utility</p></div><div class="refsynopsisdiv"><h2>Synopsis</h2><div class="cmdsynopsis"><p><code class="command">ctdb</code> [<em class="replaceable"><code>OPTION</code></em>...] {<em class="replaceable"><code>COMMAND</code></em>} [<em class="replaceable"><code>COMMAND-ARGS</code></em>]</p></div></div><div class="refsect1"><a name="idm31"></a><h2>DESCRIPTION</h2><p>
2 ctdb is a utility to view and manage a CTDB cluster.
4 The following terms are used when referring to nodes in a
6 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term">PNN</span></dt><dd><p>
7 Physical Node Number. The physical node number is an
8 integer that describes the node in the cluster. The
9 first node has physical node number 0. in a cluster.
10 </p></dd><dt><span class="term">PNN-LIST</span></dt><dd><p>
11 This is either a single PNN, a comma-separate list of PNNs
13 </p></dd></dl></div><p>
15 Commands that reference a database use the following terms:
16 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term">DB</span></dt><dd><p>
17 This is either a database name, such as
18 <code class="filename">locking.tdb</code> or a database ID such
20 </p></dd><dt><span class="term">DB-LIST</span></dt><dd><p>
21 A space separated list of at least one
22 <em class="parameter"><code>DB</code></em>.
23 </p></dd></dl></div><p>
24 </p></div><div class="refsect1"><a name="idm56"></a><h2>OPTIONS</h2><div class="variablelist"><dl class="variablelist"><dt><span class="term">-n <em class="parameter"><code>PNN</code></em></span></dt><dd><p>
25 The node specified by PNN should be queried for the
26 requested information. Default is to query the daemon
27 running on the local host.
28 </p></dd><dt><span class="term">-Y</span></dt><dd><p>
29 Produce output in machine readable form for easier parsing
30 by scripts. This uses a field delimiter of ':'. Not all
31 commands support this option.
32 </p></dd><dt><span class="term">-x <em class="parameter"><code>SEPARATOR</code></em></span></dt><dd><p>
33 Use SEPARATOR to delimit fields in machine readable output.
35 </p></dd><dt><span class="term">-X</span></dt><dd><p>
36 Produce output in machine readable form for easier parsing
37 by scripts. This uses a field delimiter of '|'. Not all
38 commands support this option.
40 This is equivalent to "-x|" and avoids some shell quoting
42 </p></dd><dt><span class="term">-t <em class="parameter"><code>TIMEOUT</code></em></span></dt><dd><p>
43 Indicates that ctdb should wait up to TIMEOUT seconds for
44 a response to most commands sent to the CTDB daemon. The
45 default is 10 seconds.
46 </p></dd><dt><span class="term">-T <em class="parameter"><code>TIMELIMIT</code></em></span></dt><dd><p>
47 Indicates that TIMELIMIT is the maximum run time (in
48 seconds) for the ctdb command. When TIMELIMIT is exceeded
49 the ctdb command will terminate with an error. The default
51 </p></dd><dt><span class="term">-? --help</span></dt><dd><p>
52 Print some help text to the screen.
53 </p></dd><dt><span class="term">--usage</span></dt><dd><p>
54 Print useage information to the screen.
55 </p></dd><dt><span class="term">-d --debug=<em class="parameter"><code>DEBUGLEVEL</code></em></span></dt><dd><p>
56 Change the debug level for the command. Default is NOTICE.
57 </p></dd><dt><span class="term">--socket=<em class="parameter"><code>FILENAME</code></em></span></dt><dd><p>
58 Specify that FILENAME is the name of the Unix domain
59 socket to use when connecting to the local CTDB
60 daemon. The default is
61 <code class="filename">/usr/local/var/run/ctdb/ctdbd.socket</code>.
62 </p></dd></dl></div></div><div class="refsect1"><a name="idm107"></a><h2>ADMINISTRATIVE COMMANDS</h2><p>
63 These are commands used to monitor and administer a CTDB cluster.
64 </p><div class="refsect2"><a name="idm110"></a><h3>pnn</h3><p>
65 This command displays the PNN of the current node.
66 </p></div><div class="refsect2"><a name="idm113"></a><h3>status</h3><p>
67 This command shows the current status of all CTDB nodes based
68 on information from the queried node.
70 Note: If the the queried node is INACTIVE then the status
72 </p><div class="refsect3"><a name="idm117"></a><h4>Node status</h4><p>
73 This includes the number of physical nodes and the status of
74 each node. See <span class="citerefentry"><span class="refentrytitle">ctdb</span>(7)</span> for information
76 </p></div><div class="refsect3"><a name="idm123"></a><h4>Generation</h4><p>
77 The generation id is a number that indicates the current generation
78 of a cluster instance. Each time a cluster goes through a
79 reconfiguration or a recovery its generation id will be changed.
81 This number does not have any particular meaning other than
82 to keep track of when a cluster has gone through a
83 recovery. It is a random number that represents the current
84 instance of a ctdb cluster and its databases. The CTDB
85 daemon uses this number internally to be able to tell when
86 commands to operate on the cluster and the databases was
87 issued in a different generation of the cluster, to ensure
88 that commands that operate on the databases will not survive
89 across a cluster database recovery. After a recovery, all
90 old outstanding commands will automatically become invalid.
92 Sometimes this number will be shown as "INVALID". This only means that
93 the ctdbd daemon has started but it has not yet merged with the cluster through a recovery.
94 All nodes start with generation "INVALID" and are not assigned a real
95 generation id until they have successfully been merged with a cluster
97 </p></div><div class="refsect3"><a name="idm128"></a><h4>Virtual Node Number (VNN) map</h4><p>
98 Consists of the number of virtual nodes and mapping from
99 virtual node numbers to physical node numbers. Only nodes
100 that are participating in the VNN map can become lmaster for
102 </p></div><div class="refsect3"><a name="idm131"></a><h4>Recovery mode</h4><p>
103 This is the current recovery mode of the cluster. There are two possible modes:
105 NORMAL - The cluster is fully operational.
107 RECOVERY - The cluster databases have all been frozen, pausing all services while the cluster awaits a recovery process to complete. A recovery process should finish within seconds. If a cluster is stuck in the RECOVERY state this would indicate a cluster malfunction which needs to be investigated.
109 Once the recovery master detects an inconsistency, for example a node
110 becomes disconnected/connected, the recovery daemon will trigger a
111 cluster recovery process, where all databases are remerged across the
112 cluster. When this process starts, the recovery master will first
113 "freeze" all databases to prevent applications such as samba from
114 accessing the databases and it will also mark the recovery mode as
117 When the CTDB daemon starts up, it will start in RECOVERY
118 mode. Once the node has been merged into a cluster and all
119 databases have been recovered, the node mode will change into
120 NORMAL mode and the databases will be "thawed", allowing samba
121 to access the databases again.
122 </p></div><div class="refsect3"><a name="idm138"></a><h4>Recovery master</h4><p>
123 This is the cluster node that is currently designated as the recovery master. This node is responsible of monitoring the consistency of the cluster and to perform the actual recovery process when reqired.
125 Only one node at a time can be the designated recovery master. Which
126 node is designated the recovery master is decided by an election
127 process in the recovery daemons running on each node.
128 </p></div><div class="refsect3"><a name="idm142"></a><h4>Example</h4><pre class="screen">
131 pnn:0 192.168.2.200 OK (THIS NODE)
132 pnn:1 192.168.2.201 OK
133 pnn:2 192.168.2.202 OK
134 pnn:3 192.168.2.203 OK
135 Generation:1362079228
141 Recovery mode:NORMAL (0)
143 </pre></div></div><div class="refsect2"><a name="idm145"></a><h3>nodestatus [<span class="optional"><em class="parameter"><code>PNN-LIST</code></em></span>]</h3><p>
144 This command is similar to the <span class="command"><strong>status</strong></span>
145 command. It displays the "node status" subset of output. The
146 main differences are:
147 </p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p>
148 The exit code is the bitwise-OR of the flags for each
149 specified node, while <span class="command"><strong>ctdb status</strong></span> exits
150 with 0 if it was able to retrieve status for all nodes.
151 </p></li><li class="listitem"><p>
152 <span class="command"><strong>ctdb status</strong></span> provides status information
153 for all nodes. <span class="command"><strong>ctdb nodestatus</strong></span>
154 defaults to providing status for only the current node.
155 If PNN-LIST is provided then status is given for
156 the indicated node(s).
157 </p></li></ul></div><p>
158 A common invocation in scripts is <span class="command"><strong>ctdb nodestatus
159 all</strong></span> to check whether all nodes in a cluster are
161 </p><div class="refsect3"><a name="idm161"></a><h4>Example</h4><pre class="screen">
163 pnn:0 10.0.0.30 OK (THIS NODE)
165 # ctdb nodestatus all
167 pnn:0 10.0.0.30 OK (THIS NODE)
169 </pre></div></div><div class="refsect2"><a name="idm164"></a><h3>recmaster</h3><p>
170 This command shows the pnn of the node which is currently the recmaster.
172 Note: If the the queried node is INACTIVE then the status
173 might not be current.
174 </p></div><div class="refsect2"><a name="idm168"></a><h3>uptime</h3><p>
175 This command shows the uptime for the ctdb daemon. When the last recovery or ip-failover completed and how long it took. If the "duration" is shown as a negative number, this indicates that there is a recovery/failover in progress and it started that many seconds ago.
176 </p><div class="refsect3"><a name="idm171"></a><h4>Example</h4><pre class="screen">
178 Current time of node : Thu Oct 29 10:38:54 2009
179 Ctdbd start time : (000 16:54:28) Wed Oct 28 17:44:26 2009
180 Time of last recovery/failover: (000 16:53:31) Wed Oct 28 17:45:23 2009
181 Duration of last recovery/failover: 2.248552 seconds
182 </pre></div></div><div class="refsect2"><a name="idm174"></a><h3>listnodes</h3><p>
183 This command shows lists the ip addresses of all the nodes in the cluster.
184 </p><div class="refsect3"><a name="idm177"></a><h4>Example</h4><pre class="screen">
190 </pre></div></div><div class="refsect2"><a name="idm180"></a><h3>natgw {master|list|status}</h3><p>
191 This command shows different aspects of NAT gateway status.
192 For an overview of CTDB's NAT gateway functionality please see
193 the <em class="citetitle">NAT GATEWAY</em> section in
194 <span class="citerefentry"><span class="refentrytitle">ctdb</span>(7)</span>.
195 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term">master</span></dt><dd><p>
196 Show the PNN and private IP address of the current NAT
200 </p><pre class="screen">
202 </pre></dd><dt><span class="term">list</span></dt><dd><p>
203 List the private IP addresses of nodes in the current
204 NAT gateway group, annotating the master node.
207 </p><pre class="screen">
212 </pre></dd><dt><span class="term">status</span></dt><dd><p>
213 List the nodes in the current NAT gateway group and
217 </p><pre class="screen">
218 pnn:0 192.168.2.200 UNHEALTHY (THIS NODE)
219 pnn:1 192.168.2.201 OK
220 pnn:2 192.168.2.202 OK
221 pnn:3 192.168.2.203 OK
222 </pre></dd></dl></div></div><div class="refsect2"><a name="idm206"></a><h3>ping</h3><p>
223 This command will "ping" specified CTDB nodes in the cluster
224 to verify that they are running.
225 </p><div class="refsect3"><a name="idm209"></a><h4>Example</h4><pre class="screen">
227 response from 0 time=0.000054 sec (3 clients)
228 </pre></div></div><div class="refsect2"><a name="idm212"></a><h3>ifaces</h3><p>
229 This command will display the list of network interfaces, which could
230 host public addresses, along with their status.
231 </p><div class="refsect3"><a name="idm215"></a><h4>Example</h4><pre class="screen">
234 name:eth5 link:up references:2
235 name:eth4 link:down references:0
236 name:eth3 link:up references:1
237 name:eth2 link:up references:1
240 |Name|LinkStatus|References|
245 </pre></div></div><div class="refsect2"><a name="idm218"></a><h3>ip</h3><p>
246 This command will display the list of public addresses that are provided by the cluster and which physical node is currently serving this ip. By default this command will ONLY show those public addresses that are known to the node itself. To see the full list of all public ips across the cluster you must use "ctdb ip all".
247 </p><div class="refsect3"><a name="idm221"></a><h4>Example</h4><pre class="screen">
250 172.31.91.82 node[1] active[] available[eth2,eth3] configured[eth2,eth3]
251 172.31.91.83 node[0] active[eth3] available[eth2,eth3] configured[eth2,eth3]
252 172.31.91.84 node[1] active[] available[eth2,eth3] configured[eth2,eth3]
253 172.31.91.85 node[0] active[eth2] available[eth2,eth3] configured[eth2,eth3]
254 172.31.92.82 node[1] active[] available[eth5] configured[eth4,eth5]
255 172.31.92.83 node[0] active[eth5] available[eth5] configured[eth4,eth5]
256 172.31.92.84 node[1] active[] available[eth5] configured[eth4,eth5]
257 172.31.92.85 node[0] active[eth5] available[eth5] configured[eth4,eth5]
260 |Public IP|Node|ActiveInterface|AvailableInterfaces|ConfiguredInterfaces|
261 |172.31.91.82|1||eth2,eth3|eth2,eth3|
262 |172.31.91.83|0|eth3|eth2,eth3|eth2,eth3|
263 |172.31.91.84|1||eth2,eth3|eth2,eth3|
264 |172.31.91.85|0|eth2|eth2,eth3|eth2,eth3|
265 |172.31.92.82|1||eth5|eth4,eth5|
266 |172.31.92.83|0|eth5|eth5|eth4,eth5|
267 |172.31.92.84|1||eth5|eth4,eth5|
268 |172.31.92.85|0|eth5|eth5|eth4,eth5|
269 </pre></div></div><div class="refsect2"><a name="idm224"></a><h3>ipinfo <em class="parameter"><code>IP</code></em></h3><p>
270 This command will display details about the specified public addresses.
271 </p><div class="refsect3"><a name="idm228"></a><h4>Example</h4><pre class="screen">
272 # ctdb ipinfo 172.31.92.85
273 Public IP[172.31.92.85] info on node 0
277 Interface[1]: Name:eth4 Link:down References:0
278 Interface[2]: Name:eth5 Link:up References:2 (active)
279 </pre></div></div><div class="refsect2"><a name="idm231"></a><h3>event run|status|script list|script enable|script disable</h3><p>
280 This command is used to control event daemon and to inspect
281 status of various events.
282 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term">run <em class="parameter"><code>EVENT</code></em> <em class="parameter"><code>TIMEOUT</code></em> [<span class="optional"><em class="parameter"><code>ARGUMENTS</code></em></span>] </span></dt><dd><p>
283 This command can be used to manually run specified EVENT
284 with optional ARGUMENTS. The event will be allowed to run
285 a maximum of TIMEOUT seconds. If TIMEOUT is 0, then there
286 is no time limit for running the event.
287 </p></dd><dt><span class="term">status [<span class="optional"><em class="parameter"><code>EVENT</code></em></span>] [<span class="optional">lastrun|lastpass|lastfail</span>]</span></dt><dd><p>
288 This command displays the last execution status of the
289 specified EVENT. If no event is specified, then the status
290 of last executed monitor event will be displayed.
292 To see the last successful execution of the event, lastpass
293 can be specified. Similarly lastfail can be specified
294 to see the last unsuccessful execution of the event.
295 The optional lastrun can be specified to query the last
296 execution of the event.
298 The command will terminate with the exit status
299 corresponding to the overall status of event that is
300 displayed. If lastpass is specified, then the command will
301 always terminate with 0. If lastfail is specified then the
302 command will always terminate with non-zero exit status.
303 If lastrun is specified, then the command will terminate
304 with 0 or not depending on if the last execution of the
305 event was successful or not.
307 The output is the list of event scripts executed.
308 Each line shows the name, status, duration and start time
312 </p><pre class="screen">
313 00.ctdb OK 0.014 Sat Dec 17 19:39:11 2016
314 01.reclock OK 0.013 Sat Dec 17 19:39:11 2016
315 05.system OK 0.029 Sat Dec 17 19:39:11 2016
316 06.nfs OK 0.014 Sat Dec 17 19:39:11 2016
318 10.interface OK 0.037 Sat Dec 17 19:39:11 2016
319 11.natgw OK 0.011 Sat Dec 17 19:39:11 2016
320 11.routing OK 0.007 Sat Dec 17 19:39:11 2016
321 13.per_ip_routing OK 0.007 Sat Dec 17 19:39:11 2016
322 20.multipathd OK 0.007 Sat Dec 17 19:39:11 2016
323 31.clamd OK 0.007 Sat Dec 17 19:39:11 2016
324 40.vsftpd OK 0.013 Sat Dec 17 19:39:11 2016
325 41.httpd OK 0.018 Sat Dec 17 19:39:11 2016
326 49.winbind OK 0.023 Sat Dec 17 19:39:11 2016
327 50.samba OK 0.100 Sat Dec 17 19:39:12 2016
328 60.nfs OK 0.376 Sat Dec 17 19:39:12 2016
329 70.iscsi OK 0.009 Sat Dec 17 19:39:12 2016
330 91.lvs OK 0.007 Sat Dec 17 19:39:12 2016
331 99.timeout OK 0.007 Sat Dec 17 19:39:12 2016
332 </pre></dd><dt><span class="term">script list</span></dt><dd><p>
333 List the available event scripts.
336 </p><pre class="screen">
356 </pre></dd><dt><span class="term">script enable <em class="parameter"><code>SCRIPT</code></em></span></dt><dd><p>
357 Enable the specified event SCRIPT. Only enabled scripts will be
358 executed when running any event.
359 </p></dd><dt><span class="term">script disable <em class="parameter"><code>SCRIPT</code></em></span></dt><dd><p>
360 Disable the specified event SCRIPT. This will prevent the script
361 from executing when running any event.
362 </p></dd></dl></div></div><div class="refsect2"><a name="idm271"></a><h3>scriptstatus</h3><p>
363 This command displays which event scripts where run in the previous
364 monitoring cycle and the result of each script. If a script
365 failed with an error, causing the node to become unhealthy,
366 the output from that script is also shown.
368 This command is deprecated. It's provided for backward
369 compatibility. In place of <span class="command"><strong>ctdb scriptstatus</strong></span>,
370 use <span class="command"><strong>ctdb event status</strong></span>.
371 </p><div class="refsect3"><a name="idm277"></a><h4>Example</h4><pre class="screen">
373 00.ctdb OK 0.011 Sat Dec 17 19:40:46 2016
374 01.reclock OK 0.010 Sat Dec 17 19:40:46 2016
375 05.system OK 0.030 Sat Dec 17 19:40:46 2016
376 06.nfs OK 0.014 Sat Dec 17 19:40:46 2016
378 10.interface OK 0.041 Sat Dec 17 19:40:46 2016
379 11.natgw OK 0.008 Sat Dec 17 19:40:46 2016
380 11.routing OK 0.007 Sat Dec 17 19:40:46 2016
381 13.per_ip_routing OK 0.007 Sat Dec 17 19:40:46 2016
382 20.multipathd OK 0.007 Sat Dec 17 19:40:46 2016
383 31.clamd OK 0.007 Sat Dec 17 19:40:46 2016
384 40.vsftpd OK 0.013 Sat Dec 17 19:40:46 2016
385 41.httpd OK 0.015 Sat Dec 17 19:40:46 2016
386 49.winbind OK 0.022 Sat Dec 17 19:40:46 2016
387 50.samba ERROR 0.077 Sat Dec 17 19:40:46 2016
388 OUTPUT: ERROR: samba tcp port 445 is not responding
389 </pre></div></div><div class="refsect2"><a name="idm280"></a><h3>listvars</h3><p>
390 List all tuneable variables, except the values of the obsolete tunables
391 like VacuumMinInterval. The obsolete tunables can be retrieved only
392 explicitly with the "ctdb getvar" command.
393 </p><div class="refsect3"><a name="idm283"></a><h4>Example</h4><pre class="screen">
395 SeqnumInterval = 1000
398 KeepaliveInterval = 5
405 TickleUpdateInterval = 20
406 EventScriptTimeout = 30
407 MonitorTimeoutCount = 20
408 RecoveryGracePeriod = 120
409 RecoveryBanPeriod = 300
410 DatabaseHashSize = 100001
412 RerecoveryTimeout = 10
415 DisableIPFailover = 0
416 VerboseMemoryNames = 0
420 RecLockLatencyMs = 1000
421 RecoveryDropAllIPs = 120
423 VacuumMaxRunTime = 120
426 VacuumFastPathCount = 60
427 MaxQueueDropMsg = 1000000
428 AllowUnhealthyDBRead = 0
429 StatHistoryInterval = 1
430 DeferredAttachTO = 120
431 AllowClientDBAttach = 1
432 RecoverPDBBySeqNum = 1
433 DeferredRebalanceOnNodeAdd = 300
435 HopcountMakeSticky = 50
439 DBRecordCountWarn = 100000
440 DBRecordSizeWarn = 10000000
441 DBSizeWarn = 100000000
442 PullDBPreallocation = 10485760
443 NoIPHostOnAllDisabled = 0
445 LockProcessesPerDB = 200
446 RecBufferSizeLimit = 1000000
447 QueueBufferSize = 1024
449 </pre></div></div><div class="refsect2"><a name="idm286"></a><h3>getvar <em class="parameter"><code>NAME</code></em></h3><p>
450 Get the runtime value of a tuneable variable.
451 </p><div class="refsect3"><a name="idm290"></a><h4>Example</h4><pre class="screen">
452 # ctdb getvar MonitorInterval
454 </pre></div></div><div class="refsect2"><a name="idm293"></a><h3>setvar <em class="parameter"><code>NAME</code></em> <em class="parameter"><code>VALUE</code></em></h3><p>
455 Set the runtime value of a tuneable variable.
456 </p><div class="refsect3"><a name="idm298"></a><h4>Example</h4><pre class="screen">
457 # ctdb setvar MonitorInterval 20
458 </pre></div></div><div class="refsect2"><a name="idm301"></a><h3>lvs {master|list|status}</h3><p>
459 This command shows different aspects of LVS status. For an
460 overview of CTDB's LVS functionality please see the
461 <em class="citetitle">LVS</em> section in
462 <span class="citerefentry"><span class="refentrytitle">ctdb</span>(7)</span>.
463 </p><div class="variablelist"><dl class="variablelist"><dt><span class="term">master</span></dt><dd><p>
464 Shows the PNN of the current LVS master node.
467 </p><pre class="screen">
469 </pre></dd><dt><span class="term">list</span></dt><dd><p>
470 Lists the currently usable LVS nodes.
473 </p><pre class="screen">
476 </pre></dd><dt><span class="term">status</span></dt><dd><p>
477 List the nodes in the current LVS group and their status.
480 </p><pre class="screen">
481 pnn:0 10.0.0.11 UNHEALTHY (THIS NODE)
482 pnn:1 10.0.0.12 UNHEALTHY
485 </pre></dd></dl></div></div><div class="refsect2"><a name="idm327"></a><h3>getcapabilities</h3><p>
486 This command shows the capabilities of the current node. See
487 the <em class="citetitle">CAPABILITIES</em> section in
488 <span class="citerefentry"><span class="refentrytitle">ctdb</span>(7)</span> for more details.
491 </p><pre class="screen">
494 </pre></div><div class="refsect2"><a name="idm336"></a><h3>statistics</h3><p>
495 Collect statistics from the CTDB daemon about
496 how many calls it has served. Information about
497 various fields in statistics can be found in
498 <span class="citerefentry"><span class="refentrytitle">ctdb-statistics</span>(7)</span>.
499 </p><div class="refsect3"><a name="idm342"></a><h4>Example</h4><pre class="screen">
502 Current time of statistics : Tue Mar 8 15:18:51 2016
503 Statistics collected since : (003 21:31:32) Fri Mar 4 17:47:19 2016
508 client_packets_sent 8170534
509 client_packets_recv 7166132
510 node_packets_sent 16549998
511 node_packets_recv 5244418
512 keepalive_packets_sent 201969
513 keepalive_packets_recv 201969
522 reply_control 6831284
539 pending_childwrite_calls 0
542 total_ro_delegations 0
544 hop_count_buckets: 8 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0
545 lock_buckets: 0 0 8 0 0 0 0 0 0 0 0 0 0 0 0 0
546 locks_latency MIN/AVG/MAX 0.010005/0.010418/0.011010 sec out of 8
547 reclock_ctdbd MIN/AVG/MAX 0.002538/0.002538/0.002538 sec out of 1
548 reclock_recd MIN/AVG/MAX 0.000000/0.000000/0.000000 sec out of 0
549 call_latency MIN/AVG/MAX 0.000044/0.002142/0.011702 sec out of 15
550 childwrite_latency MIN/AVG/MAX 0.000000/0.000000/0.000000 sec out of 0
551 </pre></div></div><div class="refsect2"><a name="idm345"></a><h3>statisticsreset</h3><p>
552 This command is used to clear all statistics counters in a node.
554 Example: ctdb statisticsreset
555 </p></div><div class="refsect2"><a name="idm349"></a><h3>dbstatistics <em class="parameter"><code>DB</code></em></h3><p>
556 Display statistics about the database DB. Information
557 about various fields in dbstatistics can be found in
558 <span class="citerefentry"><span class="refentrytitle">ctdb-statistics</span>(7)</span>.
559 </p><div class="refsect3"><a name="idm356"></a><h4>Example</h4><pre class="screen">
560 # ctdb dbstatistics locking.tdb
561 DB Statistics: locking.tdb
569 hop_count_buckets: 28087 2 1 0 0 0 0 0 0 0 0 0 0 0 0 0
570 lock_buckets: 0 14188 38 76 32 19 3 0 0 0 0 0 0 0 0 0
571 locks_latency MIN/AVG/MAX 0.001066/0.012686/4.202292 sec out of 14356
572 vacuum_latency MIN/AVG/MAX 0.000472/0.002207/15.243570 sec out of 224530
574 Count:8 Key:ff5bd7cb3ee3822edc1f0000000000000000000000000000
575 </pre></div></div><div class="refsect2"><a name="idm359"></a><h3>getreclock</h3><p>
576 Show details of the recovery lock, if any.
579 </p><pre class="screen">
580 /clusterfs/.ctdb/recovery.lock
581 </pre></div><div class="refsect2"><a name="idm364"></a><h3>getdebug</h3><p>
582 Get the current debug level for the node. the debug level controls what information is written to the log file.
584 The debug levels are mapped to the corresponding syslog levels.
585 When a debug level is set, only those messages at that level and higher
586 levels will be printed.
588 The list of debug levels from highest to lowest are :
590 ERROR WARNING NOTICE INFO DEBUG
591 </p></div><div class="refsect2"><a name="idm370"></a><h3>setdebug <em class="parameter"><code>DEBUGLEVEL</code></em></h3><p>
592 Set the debug level of a node. This controls what information will be logged.
594 The debuglevel is one of ERROR WARNING NOTICE INFO DEBUG
595 </p></div><div class="refsect2"><a name="idm375"></a><h3>getpid</h3><p>
596 This command will return the process id of the ctdb daemon.
597 </p></div><div class="refsect2"><a name="idm378"></a><h3>disable</h3><p>
598 This command is used to administratively disable a node in the cluster.
599 A disabled node will still participate in the cluster and host
600 clustered TDB records but its public ip address has been taken over by
601 a different node and it no longer hosts any services.
602 </p></div><div class="refsect2"><a name="idm381"></a><h3>enable</h3><p>
603 Re-enable a node that has been administratively disabled.
604 </p></div><div class="refsect2"><a name="idm384"></a><h3>stop</h3><p>
605 This command is used to administratively STOP a node in the cluster.
606 A STOPPED node is connected to the cluster but will not host any
607 public ip addresse, nor does it participate in the VNNMAP.
608 The difference between a DISABLED node and a STOPPED node is that
609 a STOPPED node does not host any parts of the database which means
610 that a recovery is required to stop/continue nodes.
611 </p></div><div class="refsect2"><a name="idm387"></a><h3>continue</h3><p>
612 Re-start a node that has been administratively stopped.
613 </p></div><div class="refsect2"><a name="idm390"></a><h3>addip <em class="parameter"><code>IPADDR</code></em>/<em class="parameter"><code>mask</code></em> <em class="parameter"><code>IFACE</code></em></h3><p>
614 This command is used to add a new public ip to a node
615 during runtime. It should be followed by a <span class="command"><strong>ctdb
616 ipreallocate</strong></span>. This allows public addresses to be
617 added to a cluster without having to restart the ctdb daemons.
619 Note that this only updates the runtime instance of ctdb. Any
620 changes will be lost next time ctdb is restarted and the public
621 addresses file is re-read. If you want this change to be
622 permanent you must also update the public addresses file manually.
623 </p></div><div class="refsect2"><a name="idm398"></a><h3>delip <em class="parameter"><code>IPADDR</code></em></h3><p>
624 This command flags IPADDR for deletion from a node at runtime.
625 It should be followed by a <span class="command"><strong>ctdb
626 ipreallocate</strong></span>. If IPADDR is currently hosted by the
627 node it is being removed from, this ensures that the IP will
628 first be failed over to another node, if possible, and that it
629 is then actually removed.
631 Note that this only updates the runtime instance of CTDB. Any
632 changes will be lost next time CTDB is restarted and the
633 public addresses file is re-read. If you want this change to
634 be permanent you must also update the public addresses file
636 </p></div><div class="refsect2"><a name="idm404"></a><h3>moveip <em class="parameter"><code>IPADDR</code></em> <em class="parameter"><code>PNN</code></em></h3><p>
637 This command can be used to manually fail a public ip address to a
640 In order to manually override the "automatic" distribution of public
641 ip addresses that ctdb normally provides, this command only works
642 when you have changed the tunables for the daemon to:
644 IPAllocAlgorithm != 0
647 </p></div><div class="refsect2"><a name="idm412"></a><h3>shutdown</h3><p>
648 This command will shutdown a specific CTDB daemon.
649 </p></div><div class="refsect2"><a name="idm415"></a><h3>setlmasterrole on|off</h3><p>
650 This command is used ot enable/disable the LMASTER capability for a node at runtime. This capability determines whether or not a node can be used as an LMASTER for records in the database. A node that does not have the LMASTER capability will not show up in the vnnmap.
652 Nodes will by default have this capability, but it can be stripped off nodes by the setting in the sysconfig file or by using this command.
654 Once this setting has been enabled/disabled, you need to perform a recovery for it to take effect.
656 See also "ctdb getcapabilities"
657 </p></div><div class="refsect2"><a name="idm421"></a><h3>setrecmasterrole on|off</h3><p>
658 This command is used ot enable/disable the RECMASTER capability for a node at runtime. This capability determines whether or not a node can be used as an RECMASTER for the cluster. A node that does not have the RECMASTER capability can not win a recmaster election. A node that already is the recmaster for the cluster when the capability is stripped off the node will remain the recmaster until the next cluster election.
660 Nodes will by default have this capability, but it can be stripped off nodes by the setting in the sysconfig file or by using this command.
662 See also "ctdb getcapabilities"
663 </p></div><div class="refsect2"><a name="idm426"></a><h3>reloadnodes</h3><p>
664 This command is used when adding new nodes, or removing
665 existing nodes from an existing cluster.
667 Procedure to add nodes:
668 </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>
669 To expand an existing cluster, first ensure with
670 <span class="command"><strong>ctdb status</strong></span> that all nodes are up and
671 running and that they are all healthy. Do not try to
672 expand a cluster unless it is completely healthy!
673 </p></li><li class="listitem"><p>
674 On all nodes, edit <code class="filename">/usr/local/etc/ctdb/nodes</code>
675 and <span class="emphasis"><em>add the new nodes at the end of this
677 </p></li><li class="listitem"><p>
678 Verify that all the nodes have identical
679 <code class="filename">/usr/local/etc/ctdb/nodes</code> files after adding
681 </p></li><li class="listitem"><p>
682 Run <span class="command"><strong>ctdb reloadnodes</strong></span> to force all nodes
683 to reload the nodes file.
684 </p></li><li class="listitem"><p>
685 Use <span class="command"><strong>ctdb status</strong></span> on all nodes and verify
686 that they now show the additional nodes.
687 </p></li><li class="listitem"><p>
688 Install and configure the new node and bring it online.
689 </p></li></ol></div><p>
690 Procedure to remove nodes:
691 </p><div class="orderedlist"><ol class="orderedlist" type="1"><li class="listitem"><p>
692 To remove nodes from an existing cluster, first ensure
693 with <span class="command"><strong>ctdb status</strong></span> that all nodes, except
694 the node to be deleted, are up and running and that they
695 are all healthy. Do not try to remove nodes from a
696 cluster unless the cluster is completely healthy!
697 </p></li><li class="listitem"><p>
698 Shutdown and power off the node to be removed.
699 </p></li><li class="listitem"><p>
700 On all other nodes, edit the
701 <code class="filename">/usr/local/etc/ctdb/nodes</code> file and
702 <span class="emphasis"><em>comment out</em></span> the nodes to be removed.
703 <span class="emphasis"><em>Do not delete the lines for the deleted
704 nodes</em></span>, just comment them out by adding a '#' at
705 the beginning of the lines.
706 </p></li><li class="listitem"><p>
707 Run <span class="command"><strong>ctdb reloadnodes</strong></span> to force all nodes
708 to reload the nodes file.
709 </p></li><li class="listitem"><p>
710 Use <span class="command"><strong>ctdb status</strong></span> on all nodes and verify
711 that the deleted nodes are no longer listed.
712 </p></li></ol></div></div><div class="refsect2"><a name="idm467"></a><h3>
714 [<span class="optional"><em class="parameter"><code>PNN-LIST</code></em></span>]
716 This command reloads the public addresses configuration file
717 on the specified nodes. When it completes addresses will be
718 reconfigured and reassigned across the cluster as necessary.
720 This command is currently unable to make changes to the
721 netmask or interfaces associated with existing addresses.
722 Such changes must be made in 2 steps by deleting addresses in
723 question and re-adding then. Unfortunately this will disrupt
724 connections to the changed addresses.
725 </p></div><div class="refsect2"><a name="idm473"></a><h3>getdbmap</h3><p>
726 This command lists all clustered TDB databases that the CTDB
727 daemon has attached to. Some databases are flagged as PERSISTENT,
728 this means that the database stores data persistently and the
729 data will remain across reboots. One example of such a database
730 is secrets.tdb where information about how the cluster was joined
731 to the domain is stored. Some database are flagged as REPLICATED,
732 this means that the data in that database is replicated across all
733 the nodes. But the data will not remain across reboots. This
734 type of database is used by CTDB to store it's internal state.
736 If a PERSISTENT database is not in a healthy state the database
737 is flagged as UNHEALTHY. If there's at least one completely
738 healthy node running in the cluster, it's possible that the
739 content is restored by a recovery run automaticly. Otherwise an
740 administrator needs to analyze the problem.
742 See also "ctdb getdbstatus", "ctdb backupdb", "ctdb restoredb",
743 "ctdb dumpbackup", "ctdb wipedb", "ctdb setvar AllowUnhealthyDBRead 1"
744 and (if samba or tdb-utils are installed) "tdbtool check".
746 Most databases are not persistent and only store the state
747 information that the currently running samba daemons need. These
748 databases are always wiped when ctdb/samba starts and when a
750 </p><div class="refsect3"><a name="idm479"></a><h4>Example</h4><pre class="screen">
752 Number of databases:10
753 dbid:0x435d3410 name:notify.tdb path:/usr/local/var/lib/ctdb/notify.tdb.0
754 dbid:0x42fe72c5 name:locking.tdb path:/usr/local/var/lib/ctdb/locking.tdb.0
755 dbid:0x1421fb78 name:brlock.tdb path:/usr/local/var/lib/ctdb/brlock.tdb.0
756 dbid:0x17055d90 name:connections.tdb path:/usr/local/var/lib/ctdb/connections.tdb.0
757 dbid:0xc0bdde6a name:sessionid.tdb path:/usr/local/var/lib/ctdb/sessionid.tdb.0
758 dbid:0x122224da name:test.tdb path:/usr/local/var/lib/ctdb/test.tdb.0
759 dbid:0x2672a57f name:idmap2.tdb path:/usr/local/var/lib/ctdb/persistent/idmap2.tdb.0 PERSISTENT
760 dbid:0xb775fff6 name:secrets.tdb path:/usr/local/var/lib/ctdb/persistent/secrets.tdb.0 PERSISTENT
761 dbid:0xe98e08b6 name:group_mapping.tdb path:/usr/local/var/lib/ctdb/persistent/group_mapping.tdb.0 PERSISTENT
762 dbid:0x7bbbd26c name:passdb.tdb path:/usr/local/var/lib/ctdb/persistent/passdb.tdb.0 PERSISTENT
764 # ctdb getdbmap # example for unhealthy database
765 Number of databases:1
766 dbid:0xb775fff6 name:secrets.tdb path:/usr/local/var/lib/ctdb/persistent/secrets.tdb.0 PERSISTENT UNHEALTHY
769 |ID|Name|Path|Persistent|Unhealthy|
770 |0x7bbbd26c|passdb.tdb|/usr/local/var/lib/ctdb/persistent/passdb.tdb.0|1|0|
771 </pre></div></div><div class="refsect2"><a name="idm482"></a><h3>
773 <em class="parameter"><code>DB</code></em>
774 <em class="parameter"><code>FILE</code></em>
776 Copy the contents of database DB to FILE. FILE can later be
777 read back using <span class="command"><strong>restoredb</strong></span>. This is mainly
778 useful for backing up persistent databases such as
779 <code class="filename">secrets.tdb</code> and similar.
780 </p></div><div class="refsect2"><a name="idm489"></a><h3>
782 <em class="parameter"><code>FILE</code></em>
783 [<span class="optional"><em class="parameter"><code>DB</code></em></span>]
785 This command restores a persistent database that was
786 previously backed up using backupdb. By default the data will
787 be restored back into the same database as it was created
788 from. By specifying dbname you can restore the data into a
790 </p></div><div class="refsect2"><a name="idm495"></a><h3>setdbreadonly <em class="parameter"><code>DB</code></em></h3><p>
791 This command will enable the read-only record support for a
792 database. This is an experimental feature to improve
793 performance for contended records primarily in locking.tdb and
794 brlock.tdb. When enabling this feature you must set it on all
795 nodes in the cluster.
796 </p></div><div class="refsect2"><a name="idm499"></a><h3>setdbsticky <em class="parameter"><code>DB</code></em></h3><p>
797 This command will enable the sticky record support for the
798 specified database. This is an experimental feature to
799 improve performance for contended records primarily in
800 locking.tdb and brlock.tdb. When enabling this feature you
801 must set it on all nodes in the cluster.
802 </p></div></div><div class="refsect1"><a name="idm503"></a><h2>INTERNAL COMMANDS</h2><p>
803 Internal commands are used by CTDB's scripts and are not
804 required for managing a CTDB cluster. Their parameters and
805 behaviour are subject to change.
806 </p><div class="refsect2"><a name="idm506"></a><h3>gettickles <em class="parameter"><code>IPADDR</code></em></h3><p>
807 Show TCP connections that are registered with CTDB to be
808 "tickled" if there is a failover.
809 </p></div><div class="refsect2"><a name="idm510"></a><h3>gratarp <em class="parameter"><code>IPADDR</code></em> <em class="parameter"><code>INTERFACE</code></em></h3><p>
810 Send out a gratuitous ARP for the specified interface through
811 the specified interface. This command is mainly used by the
813 </p></div><div class="refsect2"><a name="idm515"></a><h3>
814 pdelete <em class="parameter"><code>DB</code></em> <em class="parameter"><code>KEY</code></em>
817 </p></div><div class="refsect2"><a name="idm520"></a><h3>
818 pfetch <em class="parameter"><code>DB</code></em> <em class="parameter"><code>KEY</code></em>
820 Print the value associated with KEY in DB.
821 </p></div><div class="refsect2"><a name="idm525"></a><h3>
823 <em class="parameter"><code>DB</code></em>
824 <em class="parameter"><code>KEY</code></em>
825 <em class="parameter"><code>FILE</code></em>
827 Store KEY in DB with contents of FILE as the associated value.
828 </p></div><div class="refsect2"><a name="idm531"></a><h3>
830 <em class="parameter"><code>DB</code></em>
831 [<span class="optional"><em class="parameter"><code>FILE</code></em></span>]
833 Read a list of key-value pairs, one per line from FILE, and
834 store them in DB using a single transaction. An empty value
835 is equivalent to deleting the given key.
837 The key and value should be separated by spaces or tabs. Each
838 key/value should be a printable string enclosed in
840 </p></div><div class="refsect2"><a name="idm538"></a><h3>runstate [setup|first_recovery|startup|running]</h3><p>
841 Print the runstate of the specified node. Runstates are used
842 to serialise important state transitions in CTDB, particularly
845 If one or more optional runstate arguments are specified then
846 the node must be in one of these runstates for the command to
848 </p><div class="refsect3"><a name="idm542"></a><h4>Example</h4><pre class="screen">
851 </pre></div></div><div class="refsect2"><a name="idm545"></a><h3>setifacelink <em class="parameter"><code>IFACE</code></em> up|down</h3><p>
852 Set the internal state of network interface IFACE. This is
853 typically used in the <code class="filename">10.interface</code> script
854 in the "monitor" event.
856 Example: ctdb setifacelink eth0 up
857 </p></div><div class="refsect2"><a name="idm551"></a><h3>tickle</h3><p>
858 Read a list of TCP connections, one per line, from standard
859 input and send a TCP tickle to the source host for each
860 connection. A connection is specified as:
861 </p><pre class="synopsis">
862 <em class="parameter"><code>SRC-IPADDR</code></em>:<em class="parameter"><code>SRC-PORT</code></em> <em class="parameter"><code>DST-IPADDR</code></em>:<em class="parameter"><code>DST-PORT</code></em>
864 A single connection can be specified on the command-line
865 rather than on standard input.
867 A TCP tickle is a TCP ACK packet with an invalid sequence and
868 acknowledge number and will when received by the source host
869 result in it sending an immediate correct ACK back to the
872 TCP tickles are useful to "tickle" clients after a IP failover has
873 occurred since this will make the client immediately recognize the
874 TCP connection has been disrupted and that the client will need
875 to reestablish. This greatly speeds up the time it takes for a client
876 to detect and reestablish after an IP failover in the ctdb cluster.
877 </p></div><div class="refsect2"><a name="idm562"></a><h3>version</h3><p>
878 Display the CTDB version.
879 </p></div></div><div class="refsect1"><a name="idm565"></a><h2>DEBUGGING COMMANDS</h2><p>
880 These commands are primarily used for CTDB development and testing and
881 should not be used for normal administration.
882 </p><div class="refsect2"><a name="idm568"></a><h3>OPTIONS</h3><div class="variablelist"><dl class="variablelist"><dt><span class="term">--print-emptyrecords</span></dt><dd><p>
883 This enables printing of empty records when dumping databases
884 with the catdb, cattbd and dumpdbbackup commands. Records with
885 empty data segment are considered deleted by ctdb and cleaned
886 by the vacuuming mechanism, so this switch can come in handy for
887 debugging the vacuuming behaviour.
888 </p></dd><dt><span class="term">--print-datasize</span></dt><dd><p>
889 This lets database dumps (catdb, cattdb, dumpdbbackup) print the
890 size of the record data instead of dumping the data contents.
891 </p></dd><dt><span class="term">--print-lmaster</span></dt><dd><p>
892 This lets catdb print the lmaster for each record.
893 </p></dd><dt><span class="term">--print-hash</span></dt><dd><p>
894 This lets database dumps (catdb, cattdb, dumpdbbackup) print the
895 hash for each record.
896 </p></dd><dt><span class="term">--print-recordflags</span></dt><dd><p>
897 This lets catdb and dumpdbbackup print the
898 record flags for each record. Note that cattdb always
900 </p></dd></dl></div></div><div class="refsect2"><a name="idm591"></a><h3>process-exists <em class="parameter"><code>PID</code></em> <em class="parameter"><code>[SRVID]</code></em></h3><p>
901 This command checks if a specific process exists on the CTDB
902 host. This is mainly used by Samba to check if remote instances
903 of samba are still running or not. When the optional SRVID
904 argument is specified, the command check if a specific process
905 exists on the CTDB host and has registered for specified SRVID.
906 </p></div><div class="refsect2"><a name="idm596"></a><h3>getdbstatus <em class="parameter"><code>DB</code></em></h3><p>
907 This command displays more details about a database.
908 </p><div class="refsect3"><a name="idm600"></a><h4>Example</h4><pre class="screen">
909 # ctdb getdbstatus test.tdb.0
912 path: /usr/local/var/lib/ctdb/test.tdb.0
916 # ctdb getdbstatus registry.tdb # with a corrupted TDB
919 path: /usr/local/var/lib/ctdb/persistent/registry.tdb.0
921 HEALTH: NO-HEALTHY-NODES - ERROR - Backup of corrupted TDB in '/usr/local/var/lib/ctdb/persistent/registry.tdb.0.corrupted.20091208091949.0Z'
922 </pre></div></div><div class="refsect2"><a name="idm603"></a><h3>catdb <em class="parameter"><code>DB</code></em></h3><p>
923 Print a dump of the clustered TDB database DB.
924 </p></div><div class="refsect2"><a name="idm607"></a><h3>cattdb <em class="parameter"><code>DB</code></em></h3><p>
925 Print a dump of the contents of the local TDB database DB.
926 </p></div><div class="refsect2"><a name="idm611"></a><h3>dumpdbbackup <em class="parameter"><code>FILE</code></em></h3><p>
927 Print a dump of the contents from database backup FILE,
928 similar to <span class="command"><strong>catdb</strong></span>.
929 </p></div><div class="refsect2"><a name="idm616"></a><h3>wipedb <em class="parameter"><code>DB</code></em></h3><p>
930 Remove all contents of database DB.
931 </p></div><div class="refsect2"><a name="idm620"></a><h3>recover</h3><p>
932 This command will trigger the recovery daemon to do a cluster
934 </p></div><div class="refsect2"><a name="idm623"></a><h3>ipreallocate, sync</h3><p>
935 This command will force the recovery master to perform a full ip reallocation process and redistribute all ip addresses. This is useful to "reset" the allocations back to its default state if they have been changed using the "moveip" command. While a "recover" will also perform this reallocation, a recovery is much more hevyweight since it will also rebuild all the databases.
936 </p></div><div class="refsect2"><a name="idm626"></a><h3>attach <em class="parameter"><code>DBNAME</code></em> [persistent|replicated]</h3><p>
937 Create a new CTDB database called DBNAME and attach to it on
939 </p></div><div class="refsect2"><a name="idm630"></a><h3>detach <em class="parameter"><code>DB-LIST</code></em></h3><p>
940 Detach specified non-persistent database(s) from the cluster. This
941 command will disconnect specified database(s) on all nodes in
942 the cluster. This command should only be used when none of the
943 specified database(s) are in use.
945 All nodes should be active and tunable AllowClientDBAccess should
946 be disabled on all nodes before detaching databases.
947 </p></div><div class="refsect2"><a name="idm635"></a><h3>dumpmemory</h3><p>
948 This is a debugging command. This command will make the ctdb
949 daemon to write a fill memory allocation map to standard output.
950 </p></div><div class="refsect2"><a name="idm638"></a><h3>rddumpmemory</h3><p>
951 This is a debugging command. This command will dump the talloc memory
952 allocation tree for the recovery daemon to standard output.
953 </p></div><div class="refsect2"><a name="idm641"></a><h3>ban <em class="parameter"><code>BANTIME</code></em></h3><p>
954 Administratively ban a node for BANTIME seconds. The node
955 will be unbanned after BANTIME seconds have elapsed.
957 A banned node does not participate in the cluster. It does
958 not host any records for the clustered TDB and does not host
959 any public IP addresses.
961 Nodes are automatically banned if they misbehave. For
962 example, a node may be banned if it causes too many cluster
965 To administratively exclude a node from a cluster use the
966 <span class="command"><strong>stop</strong></span> command.
967 </p></div><div class="refsect2"><a name="idm649"></a><h3>unban</h3><p>
968 This command is used to unban a node that has either been
969 administratively banned using the ban command or has been
970 automatically banned.
971 </p></div></div><div class="refsect1"><a name="idm652"></a><h2>SEE ALSO</h2><p>
972 <span class="citerefentry"><span class="refentrytitle">ctdbd</span>(1)</span>,
974 <span class="citerefentry"><span class="refentrytitle">onnode</span>(1)</span>,
976 <span class="citerefentry"><span class="refentrytitle">ctdb</span>(7)</span>,
978 <span class="citerefentry"><span class="refentrytitle">ctdb-statistics</span>(7)</span>,
980 <span class="citerefentry"><span class="refentrytitle">ctdb-tunables</span>(7)</span>,
982 <a class="ulink" href="http://ctdb.samba.org/" target="_top">http://ctdb.samba.org/</a>
983 </p></div></div></body></html>