ctdbd.conf — CTDB daemon configuration file
This file contains CTDB configuration variables that are affect
the operation of CTDB. The default location of this file is
/usr/local/etc/ctdb/ctdbd.conf
.
This file is a shell script (see sh(1)) but is usually limited to simple variable assignments and shell-style comments.
CTDB configuration variables are grouped into several categories below.
Variables defined in this document can also be set in a
distribution-specific configuration file such as
/etc/sysconfig/ctdb
(Red Hat) or
/etc/default/ctdb
(Debian). However, these
files should be reserved for variables used by the initscript.
A historical alternative is
/usr/local/etc/ctdb/sysconfig/ctdb
- this is
deprecated.
Some options must be available to the initscript so they need to
be set in the distribution-specific initscript configuration,
such as /etc/sysconfig/ctdb
or
/etc/default/ctdb
.
FILENAME
FILENAME is the name of the file used to contain the process ID (PID) of the main CTDB daemon when it is running. This is passed from the initscript to ctdbd_wrapper(1).
Default is /usr/local/var/run/ctdb/ctdbd.pid
.
Corresponds to --pidfile
.
These options may be used in the initscripts, daemon and scripts.
DIRECTORY
DIRECTORY containing CTDB scripts and configuration files.
Variables in this section are processed by ctdbd_wrapper(1) and are converted into command-line arguments to ctdbd(1). Correspondence with ctdbd(1) options is shown for each variable. The the documentation for the relevant options for more details.
Many of these variables are also used by event scripts.
Defaults to yes. Corresponds to --no-lmaster
.
Defaults to yes. Corresponds to
--no-recmaster
.
DIRECTORY
Defaults to /usr/local/var/lib/ctdb
.
Apart from a DIRECTORY, this can take a special value of
the form
tmpfs
[:OPTIONS
].
OPTIONS is a comma-separated list of any permissible
options to the tmpfs filesystem. The only pre-specified
default is mode=700
, which can
overridden by specifying mode
in
OPTIONS. It probably makes sense to specify a maximum
size
.
Corresponds to --dbdir
.
DIRECTORY
Defaults to /usr/local/var/lib/ctdb/persistent
.
Corresponds to --dbdir-persistent
.
DIRECTORY
Defaults to /usr/local/var/lib/ctdb/state
.
Corresponds to --dbdir-state
.
DEBUGLEVEL
Default is NOTICE. Corresponds to -d
or
--debug
.
DIRECTORY
Default is CTDB_BASE
/events.d, so usually
/usr/local/etc/ctdb/events.d
. Corresponds to
--event-script-dir
.
STRING
STRING specifies where ctdbd will write its log. The
default is
file:/usr/local/var/log/log.ctdb
.
Corresponds to --logging
.
Valid values are:
FILENAME
FILENAME where ctdbd will write its log. This is usually
/usr/local/var/log/log.ctdb
.
METHOD
]CTDB will log to syslog. By default this will use the syslog(3) API.
If METHOD is specified then it specifies an extension that causes logging to be done in a non-blocking fashion. This can be useful under heavy loads that might cause the syslog daemon to dequeue messages too slowly, which would otherwise cause CTDB to block when logging. METHOD must be one of:
CTDB will log to syslog via
/dev/log
in non-blocking
mode.
CTDB will log to syslog via UDP to localhost:514. The syslog daemon must be configured to listen on (at least) localhost:514. Most implementations will log the messages against hostname "localhost" - this is a limit of the implementation for compatibility with more syslog daemon implementations.
As with "udp" but messages are sent in RFC5424 format. This method will log the correct hostname but is not as widely implemented in syslog daemons.
FILENAME
Default is CTDB_BASE
/nodes, so usually
/usr/local/etc/ctdb/nodes
. Corresponds to
--nlist
.
Defaults to no. Corresponds to --nosetsched
.
Usually CTDB runs with real-time priority. If you are running CTDB on a platform that does not support real-time priority, you can set this.
FILENAME
No default, usually
/usr/local/etc/ctdb/notify.sh
. Corresponds to
--notification-script
.
NUM
Default 0. Corresponds to
--max-persistent-check-errors
.
IPADDR
IPADDR is the private IP address that ctdbd will bind to.
Corresponds to --listen
.
By default ctdbd will select the first address from the nodes list that in can bind to. See also CTDB_NODES.
This option is only required when automatic address detection can not be used. This can be the case when running multiple ctdbd daemons/nodes on the same physical host (usually for testing), using InfiniBand for the private network or on Linux when sysctl net.ipv4.ip_nonlocal_bind=1.
FILENAME
No default, usually
/usr/local/etc/ctdb/public_addresses
.
Corresponds to --public-addresses
.
INTERFACE
No default. Corresponds to
--public-interface
.
LOCK
LOCK specifies the cluster-wide mutex used to detect and prevent a partitioned cluster (or "split brain").
No default, but the default configuration file specifies
/some/place/on/shared/storage
, which
should be change to a useful value. Corresponds to
--reclock
.
For information about the recovery lock please see the RECOVERY LOCK section in ctdb(7).
DEBUGLEVEL
Defaults to ERR. Corresponds to
--script-log-level
.
FILENAME
Defaults to /usr/local/var/run/ctdb/ctdbd.socket
.
Corresponds to --socket
.
If you change this then you probably want to set this in
root's environment (perhaps in a file in
/etc/profile.d
) so that you can use
the ctdb(1) command in a
straightforward manner.
Default is no. Corresponds to
--start-as-disabled
.
Default is no. Corresponds to
--start-as-stopped
.
Defaults to tcp. Corresponds to
--transport
.
While the following variables do not translate into daemon options they are used by ctdbd_wrapper(1) when starting and stopping ctdbd(1).
NUM
NUM is the number of seconds to wait for ctdbd(1) to shut down gracefully before giving up and killing it.
Defaults is 30.
NUM
NUM is the number of seconds to wait for ctdbd(1) complete early initialisation up to a point where it is unlikely to abort. If ctdbd doesn't complete the "setup" event before this timeout then it is killed.
Defaults is 10.
NAT gateway is used to configure fallback routing for nodes when they do not host any public IP addresses. For example, it allows unhealthy nodes to reliably communicate with external infrastructure. One node in a NAT gateway group will be designated as the NAT gateway master node and other (slave) nodes will be configured with fallback routes via the NAT gateway master node. For more information, see the NAT GATEWAY section in ctdb(7).
IPADDR
IPADDR is an alternate network gateway to use on the NAT gateway master node. If set, a fallback default route is added via this network gateway.
No default. Setting this variable is optional - if not set that no route is created on the NAT gateway master node.
FILENAME
FILENAME contains the list of nodes that belong to the same NAT gateway group.
File format:
IPADDR
[slave-only]
IPADDR is the private IP address of each node in the NAT gateway group.
If "slave-only" is specified then the corresponding node
can not be the NAT gateway master node. In this case
CTDB_NATGW_PUBLIC_IFACE
and
CTDB_NATGW_PUBLIC_IP
are optional and
unused.
No default, usually
/usr/local/etc/ctdb/natgw_nodes
when enabled.
IPADDR/MASK
IPADDR/MASK is the private sub-network that is internally routed via the NAT gateway master node. This is usually the private network that is used for node addresses.
No default.
IFACE
IFACE is the network interface on which the CTDB_NATGW_PUBLIC_IP will be configured.
No default.
IPADDR/MASK
IPADDR/MASK indicates the IP address that is used for outgoing traffic (originating from CTDB_NATGW_PRIVATE_NETWORK) on the NAT gateway master node. This must not be a configured public IP address.
No default.
IPADDR/MASK[@GATEWAY]
...Each IPADDR/MASK identifies a network or host to which NATGW should create a fallback route, instead of creating a single default route. This can be used when there is already a default route, via an interface that can not reach required infrastructure, that overrides the NAT gateway default route.
If GATEWAY is specified then the corresponding route on
the NATGW master node will be via GATEWAY. Such routes
are created even if
CTDB_NATGW_DEFAULT_GATEWAY
is not
specified. If GATEWAY is not specified for some
networks then routes are only created on the NATGW
master node for those networks if
CTDB_NATGW_DEFAULT_GATEWAY
is
specified.
This should be used with care to avoid causing traffic to unnecessarily double-hop through the NAT gateway master, even when a node is hosting public IP addresses. Each specified network or host should probably have a corresponding automatically created link route or static route to avoid this.
No default.
CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24 CTDB_NATGW_PUBLIC_IFACE=eth0
A variation that ensures that infrastructure (ADS, DNS, ...) directly attached to the public network (10.0.0.0/24) is always reachable would look like this:
CTDB_NATGW_NODES=/usr/local/etc/ctdb/natgw_nodes CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24 CTDB_NATGW_PUBLIC_IP=10.0.0.227/24 CTDB_NATGW_PUBLIC_IFACE=eth0 CTDB_NATGW_STATIC_ROUTES=10.0.0.0/24
Note that CTDB_NATGW_DEFAULT_GATEWAY
is
not specified.
A node running CTDB may be a component of a complex network
topology. In particular, public addresses may be spread
across several different networks (or VLANs) and it may not be
possible to route packets from these public addresses via the
system's default route. Therefore, CTDB has support for
policy routing via the 13.per_ip_routing
eventscript. This allows routing to be specified for packets
sourced from each public address. The routes are added and
removed as CTDB moves public addresses between nodes.
For more information, see the POLICY ROUTING section in ctdb(7).
FILENAME
FILENAME contains elements for constructing the desired routes for each source address.
The special FILENAME value
__auto_link_local__
indicates that no
configuration file is provided and that CTDB should
generate reasonable link-local routes for each public IP
address.
File format:
IPADDR
DEST-IPADDR/MASK
[GATEWAY-IPADDR
]
No default, usually
/usr/local/etc/ctdb/policy_routing
when enabled.
NUM
NUM sets the priority (or preference) for the routing rules that are added by CTDB.
This should be (strictly) greater than 0 and (strictly) less than 32766. A priority of 100 is recommended, unless this conflicts with a priority already in use on the system. See ip(8), for more details.
LOW-NUM
,
CTDB_PER_IP_ROUTING_TABLE_ID_HIGH=HIGH-NUM
CTDB determines a unique routing table number to use for the routing related to each public address. LOW-NUM and HIGH-NUM indicate the minimum and maximum routing table numbers that are used.
ip(8) uses some reserved routing table numbers below 255. Therefore, CTDB_PER_IP_ROUTING_TABLE_ID_LOW should be (strictly) greater than 255.
CTDB uses the standard file
/etc/iproute2/rt_tables
to maintain
a mapping between the routing table numbers and labels.
The label for a public address
ADDR
will look like
ctdb.addr
. This means that
the associated rules and routes are easy to read (and
manipulate).
No default, usually 1000 and 9000.
For a general description see the LVS section in ctdb(7).
FILENAME
FILENAME contains the list of nodes that belong to the same LVS group.
File format:
IPADDR
[slave-only]
IPADDR is the private IP address of each node in the LVS group.
If "slave-only" is specified then the corresponding node
can not be the LVS master node. In this case
CTDB_LVS_PUBLIC_IFACE
and
CTDB_LVS_PUBLIC_IP
are optional and
unused.
No default, usually
/usr/local/etc/ctdb/lvs_nodes
when enabled.
INTERFACE
INTERFACE is the network interface that clients will use
to connection to CTDB_LVS_PUBLIC_IP
.
This is optional for slave-only nodes.
No default.
IPADDR
CTDB_LVS_PUBLIC_IP is the LVS public address. No default.
Whether one or more offline interfaces should cause a monitor event to fail if there are other interfaces that are up. If this is "yes" and a node has some interfaces that are down then ctdb status will display the node as "PARTIALLYONLINE".
Note that CTDB_PARTIALLY_ONLINE_INTERFACES=yes is incompatible with NAT gateway, since NAT gateway relies on the interface configured by CTDB_NATGW_PUBLIC_IFACE to be up.
Default is "no".
CTDB can be configured to manage and/or monitor various NAS (and other) services via its eventscripts.
In the simplest case CTDB will manage a service. This means the service will be started and stopped along with CTDB, CTDB will monitor the service and CTDB will do any required reconfiguration of the service when public IP addresses are failed over.
Should CTDB manage Samba?
Default is no.
Should CTDB manage Winbind?
Default is no.
PORT-LIST
When monitoring Samba, check TCP ports in space-separated PORT-LIST.
Default is to monitor ports that Samba is configured to listen on.
As part of monitoring, should CTDB skip the check for the existence of each directory configured as share in Samba. This may be desirable if there is a large number of shares.
Default is no.
SERVICE
Distribution specific SERVICE for managing nmbd.
Default is distribution-dependant.
SERVICE
Distribution specific SERVICE for managing smbd.
Default is distribution-dependant.
SERVICE
Distribution specific SERVICE for managing winbindd.
Default is "winbind".
This includes parameters for the kernel NFS server.
Alternative NFS subsystems (such as NFS-Ganesha)
can be integrated using CTDB_NFS_CALLOUT
.
Should CTDB manage NFS?
Default is no.
COMMAND
COMMAND specifies the path to a callout to handle interactions with the configured NFS system, including startup, shutdown, monitoring.
Default is the included nfs-linux-kernel-callout.
DIRECTORY
Specifies the path to a DIRECTORY containing files that describe how to monitor the responsiveness of NFS RPC services. See the README file for this directory for an explanation of the contents of these "check" files.
CTDB_NFS_CHECKS_DIR can be used to point to different sets of checks for different NFS servers.
One way of using this is to have it point to, say,
/usr/local/etc/ctdb/nfs-checks-enabled.d
and populate it with symbolic links to the desired check
files. This avoids duplication and is upgrade-safe.
Default is
/usr/local/etc/ctdb/nfs-checks.d
,
which contains NFS RPC checks suitable for Linux kernel
NFS.
As part of monitoring, should CTDB skip the check for the existence of each directory exported via NFS. This may be desirable if there is a large number of exports.
Default is no.
IPADDR
|HOSTNAME
IPADDR or HOSTNAME indicates the address that rpcinfo should connect to when doing rpcinfo check on IPv4 RPC service during monitoring. Optimally this would be "localhost". However, this can add some performance overheads.
Default is "127.0.0.1".
IPADDR
|HOSTNAME
IPADDR or HOSTNAME indicates the address that rpcinfo should connect to when doing rpcinfo check on IPv6 RPC service during monitoring. Optimally this would be "localhost6" (or similar). However, this can add some performance overheads.
Default is "::1".
TYPE
The type of filesystem used for a clustered NFS' shared state. No default.
DIR
The directory where a clustered NFS' shared state will be located. No default.
CTDB can manage the Apache web server.
Should CTDB manage the Apache web server?
Default is no.
CTDB has support to manage the popular anti-virus daemon ClamAV.
Should CTDB manage ClamAV?
Default is no.
FILENAME
FILENAME is the socket to monitor ClamAV.
No default.
CTDB has support for managing the Linux iSCSI tgtd service.
Should CTDB manage iSCSI tgtd?
Default is no.
DIRECTORY
DIRECTORY on shared storage containing scripts to start tgtd for each public IP address.
No default.
CTDB can monitor multipath devices to ensure that active paths are available.
20.multipathd |
This eventscript is not enabled by default. Use ctdb enablescript to enable it.
MP-DEVICE-LIST
MP-DEVICE-LIST is a list of multipath devices for CTDB to monitor?
No default.
CTDB can manage the vsftpd FTP server.
Should CTDB manage the vsftpd FTP server?
Default is no.
CTDB can experience seemingly random (performance and other) issues if system resources become too constrained. Options in this section can be enabled to allow certain system resources to be checked. They allows warnings to be logged and nodes to be marked unhealthy when system resource usage reaches the configured thresholds.
Some checks are enabled by default. It is recommended that these checks remain enabled or are augmented by extra checks. There is no supported way of completely disabling the checks.
FS-LIMIT-LIST
FS-LIMIT-LIST is a space-separated list of
FILESYSTEM
:WARN_LIMIT
[:UNHEALTHY_LIMIT
]
triples indicating that warnings should be logged if the
space used on FILESYSTEM reaches WARN_LIMIT%. If usage
reaches UNHEALTHY_LIMIT then the node should be flagged
unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
left blank, meaning that check will be omitted.
Default is to warn for each filesystem containing a
database directory (CTDB_DBDIR
,
CTDB_DBDIR_PERSISTENT
,
CTDB_DBDIR_STATE
) with a threshold of
90%.
MEM-LIMITS
MEM-LIMITS takes the form
WARN_LIMIT
[:UNHEALTHY_LIMIT
]
indicating that warnings should be logged if memory
usage reaches WARN_LIMIT%. If usage reaches
UNHEALTHY_LIMIT then the node should be flagged
unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
left blank, meaning that check will be omitted.
Default is 80, so warnings will be logged when memory usage reaches 80%.
SWAP-LIMITS
SWAP-LIMITS takes the form
WARN_LIMIT
[:UNHEALTHY_LIMIT
]
indicating that warnings should be logged if
swap usage reaches WARN_LIMIT%. If usage reaches
UNHEALTHY_LIMIT then the node should be flagged
unhealthy. Either WARN_LIMIT or UNHEALTHY_LIMIT may be
left blank, meaning that check will be omitted.
Default is 25, so warnings will be logged when swap usage reaches 25%.
CTDB tunables (see ctdbd-tunables(7)) can be set from the configuration file. They are set as follows:
CTDB_SET_TUNABLE
=VALUE
For example:
CTDB_SET_MonitorInterval=20
Variable in this section are for debugging and testing CTDB. They should not generally be needed.
FILENAME
FILENAME is a script to run to log debug information when an event script times out.
Default is
.
CTDB_BASE
/debug-hung-script.sh
FILENAME
FILENAME specifies where log messages should go when debugging hung eventscripts. This is a testing option. See also CTDB_DEBUG_HUNG_SCRIPT.
No default. Messages go to stdout/stderr and are logged to the same place as other CTDB log messages.
REGEXP
REGEXP specifies interesting processes for which stack traces should be logged when debugging hung eventscripts and those processes are matched in pstree output. REGEXP is an extended regexp so choices are separated by pipes ('|'). However, REGEXP should not contain parentheses. See also CTDB_DEBUG_HUNG_SCRIPT.
Default is "exportfs|rpcinfo".
FILENAME
FILENAME is a script to run to log debug information when an CTDB fails to freeze databases during recovery.
No default, usually
.
CTDB_BASE
/debug_locks.sh
DIRECTORY
DIRECTORY containing system configuration files. This is used to provide alternate configuration when testing and should not need to be changed from the default.
Default is /etc
.
This is the init style used by the Linux distribution (or other operating system) being used. This is usually determined dynamically by checking the system. This variable is used by the initscript to determine which init system primitives to use. It is also used by some eventscripts to choose the name of initscripts for certain services, since these can vary between distributions.
No fixed default.
If this option needs to be changed from the calculated
default for the initscript to function properly, then it
must be set in the distribution-specific initscript
configuration, such as
/etc/sysconfig/ctdb
NUM
NUM is the maximum number of volatile TDB database backups to be kept (for each database) when a corrupt database is found during startup. Volatile TDBs are zeroed during startup so backups are needed to debug any corruption that occurs before a restart.
Default is 10.
NUM
NUM is the maximum number of open files.
There is no default.
FILENAME
FILENAME is a script fragment to be sourced by the
functions
that is sourced by scripts.
On example use would be to override function definitions
in unit tests. As a sanity check, this file must be
executable for it to be used.
No default.
Whether CTDB should simulate timing out monitor events.
This uses the 99.timeout
eventscript.
Default is no.
Whether CTDB core files should be suppressed.
Default is no.
COMMAND
If "yes", this causes
ctdbd(1) to be run under
valgrind(1) with logs going to
/usr/local/var/log/ctdb_valgrind
. If neither
"yes" nor "no" then the value is assumed to be a COMMAND
(e.g. a valgrind variation, a
gdb(1) command) that is
used in place of the default valgrind
command. In either case, the --valgrind
option is passed to ctdbd.
Default is no.
DIRECTORY
DIRECTORY containing CTDB files that are modified at runtime.
Defaults to /usr/local/var/lib/ctdb
.