TakTuk - a tool for large scale remote execution deployment
taktuk [-hinpsvPW] [-c command] [-d limit] [-f name] [-g n] [-m name [-[ args ... -]]] [-o name=specification] [-t timeout] [-w number] [-C separators] [-F name] [-L name] [-O separators] [-R name=number ] [-T command] [commands ... ]
TakTuk is a tool for broadcasting the remote execution of one ore more command to a set of one or more distant machines. TakTuk combines local parallelization (using concurrent deployment processes) and work distribution (using an adaptive work-stealing algorithm) to achieve both scalability and efficiency.
TakTuk is especially suited to interactive tasks involving several distant machines and parallel remote executions. This is the case of clusters administration and parallel program debugging.
TakTuk also provides a basic communication layer to programs it executes.
This communication layer uses the communication infrastructure set up by
TakTuk during its deployment. It is available both for the Perl and the C
langages and is described in taktukcomm(3).
In TakTuk, options are parsed in the order given on the comand line. This is
important as some of the options change the behavior of following ones. The
defaults settings of TakTuk can be obtained by using the --print-defaults
option. The following options are given in alphabetical
order (no other logic).
defines the connector command used to contact the following remote machines.
turns dynamic mode on (work stealing) for the deployment of all the following remote machines specifications. Uses limit as a maximal arity (0 = no limit). A negative value for limit turns dynamic mode off. Warning, currently it is a bad idea to use several -d options on the same command line.
name is the name of a file that contains remote machines names (equivalent to several -m opions).
sets to n the maximal interval between timeouts checks (usually checks are made more often: at each message received a timeouts check is made).
prints a short description of TakTuk on the screen.
forces TakTuk interactive mode even after some batch commands given on the command line.
sets the login name for the following hosts. This assumes that the connector
command accepts -l option (this is the case for ssh, the default).
name is the name of a remote machine on which TakTuk has to be deployed. This option can be followed by any number of arguments enclosed between -[ and -] which will be transmitted as arguments of the remote TakTuk that will execute on name.
Disables TakTuk logical numbering and the variables TAKTUK_COUNT and
TAKTUK_RANK are not defined in executed commands. This has the advantage of
removing the global synchronization occuring at the end of the deployment and
making the deployment more efficient.
WARNING: use this option only before any remote node specification (-m or
-f) otherwise you might get serious synchronization issues in TakTuk.
Using TakTuk point-to-point communication along with this option will
fail and produce TakTuk warnings.
set an output template specification for one of output streams designed by the given name. When giving only a name (without specification) this disables the stream. Available streams include :
connector (errors from the connector command)
error, output (error and output of remotely executed commands)
status (commands exit status)
taktuk (internal messages, warnings and errors)
A specification is a Perl expression that should evaluate to the string eventually displayed. Within a specification, some variables might be used depending on the concerned stream:
$command (not for taktuk)
The command line execution that generated output.
$count
The total number of TakTuk instances.
$eof
An empty string or a newline depending on what character ended command output.
$filename (taktuk only)
The file in which a TakTuk diagnostic function has been called.
$host
The name of the host executing the current TakTuk instance
$level (taktuk only)
The level of the diagnostic function called.
$level_name (taktuk only)
The name associated with the level of the diagnostic function called.
$line
The raw data outputed by a command or passed to a diagnostic function, any trailing newline removed (see $eof).
$line_number (taktuk only)
The line number at which a TakTuk diagnostic function has been called.
$package (taktuk only)
The package in which a TakTuk diagnostic function has been called.
$peer (connector only)
The remote machine for which a connector command outputed error messages.
$pid
The pid of the command line executed (that produced output).
$rank
The logical number of TakTuk instance in which output is collected.
$type
The name of the output stream (connector, error, output, ...).
$user_scalar
A scalar, global to the package, initially initialized to undef, that the user is free to use for any purpose.
At the end of the day, the specification is evaluated for each line of the
concerned stream and the result printed on the root node.
Note that the newline has to be added explicitely as \n in the template if
needed. This only has an effect on the remote hosts following this option.
WARNING: take care of your specification, if the Perl syntax is not correct lots of awfull compilation error messages will be displayed and TakTuk execution will fail.
propagate the TakTuk executable through connectors and thus eliminates the need for a TakTuk installation on remote machines. Advice : use it before any peer specification option.
sets the timeout for connectors (0 = no timeout). Notice that this option
override the timeout of the connector command (ssh for instance). When the
delay expires, the connector command is simply killed.
prints the TakTuk version.
sets initial window size to number (= pipeline width).
Replace the default set of characters that separate successive TakTuk commands.
name is the name of a file that contains additional options for TakTuk. The parsing of the file is done when encountering this option.
changes the name of localhost as viewed by TakTuk.
Replace the default set of characters that separate successive words when
reading options and commands from a file (using the -F switch).
prints the defaults settings used by TakTuk (environment variables taken into account).
redirect a given stream output to a given file descriptor number (located at
the root node). See -o option for more details about streams output.
com is the name of the TakTuk command (default is $0, the name used to
launch TakTuk. This is the name used once connected to initiate the remote
TakTuk engine. It will be used on the following remote machines. Note that
the effect of this switch is void when using the -s option.
turns off the dynamic adjustment of window size (no effect, dynamic window mode not implemented).
These option are not useful for most users. They are used either internally by TakTuk itself or for development purposes.
prints the content of package packagename extracted from the taktuk code in
execution.
current TakTuk instance is not the root node. This is an internal switch used by spawned TakTuk instances. This is usually not useful for most users.
sets debugging level (1..4) for scope. The scope might be a TakTuk package name or default and the lowest the level the more verbose the output.
After the options parsing, TakTuk expects some commands either on the
remaining of the command line (batch mode) or on the standard input
(interactive mode). These commands are actions to be performed by TakTuk
using the logical network infrastructure set up during the deployment. By
default, commands might be separated by ; or newlines. For all the commands,
any non ambiguous prefix can be used instead of their full name.
In interactive mode, TakTuk has support for readline (history, command
line editing) if installed on your system.
When TakTuk commands accept arguments, they should be enclosed into matching delimiters (indicated by * below). In other words, * might be replaced either by any character or by a pair of matching braces, brackets or parenthesis. Note that you can still use the braces kind you choose inside the arguments line as long as they are well imbricated.
Taktuk understands the following commands:
Sends the execution of command to all the peers belonging to the given set. For more details about the set specification, see section SET SPECIFICATION.
Broadcasts the execution of command on all the remote peers (not including the node initiating the broadcast).
Closes the inputs descriptor of all executing commands on the local machine.
Spreads the execution of command on all the children of the node initiating the downcast (not including itself).
Executes the command line on the local machine but has no effect on the root
node. Inputs/outputs of the execution are multiplexed and fowarded to the
TakTuk root.
Sends the content of a file (which must be local to the node executing the file_input) as input to each command in execution on the local machine.
WARNING: this command is not atomic. If you manage somehow to initiate a file_input command from two different TakTuk instances, data will probably be interleaved. In this case you should synchronize the two instances. This is not required when spreading files only from the root node.
Sends the remaining of the line as input to each command in execution on the local machine.
Sends the remaining of the line with an additional newline as input to each command in execution on the local machine.
Prints the current TakTuk deployment tree. Numbers in parenthesis match the
peer rank in the logical TakTuk numbering and the peer ready state. If the
deployment is not complete, the printed tree will display connecting ...
leaves.
Forces the given command to wait for the completion of deployment, nodes
numbering and previous commands before executing. This is usefull for commands
like broadcast which does not wait for nodes numbering or print_tree
which does not wait for the completion of the deployment.
Forks a perl interpreter on the local node just as if the command exec perl
arguments have been used. The difference is that this interpreter is
previously fetched with the taktuk package that contains point-to-point
communication routines (taktuk::send and taktuk::recv, see
taktukcomm(3)).
WARNING: due to the limitations of the parser that analyses the arguments of
this command, you have to give arguments (even if empty) and to use '--' if you
give any option to the perl interpreter (and even if you give it only options).
Quit the TakTuk engine and shut down the logical communication network established during the deployment.
Some of TakTuk defaults settings can be changed locally on some host using environment variables. These settings are completely local and are not propagated to other hosts that take part of the deployment. They are overridden by command line options.
To change some default setting use the variable TAKTUK_NAME where NAME is
the name of the according long option in upper case and with dashes replaced by
underscores. For option taking complex value (such as --debug) just add an
underscore and the field you want to change in upper case at the end of the
name.
Using taktuk --print-defaults will give you examples of names used to change
default settings.
Note that defining in the environment a default setting not used by TakTuk
will have no effect.
Taktuk sets the following environment variables for all the commands it executes :
A file descriptor, used internally by TakTuk.
The total number of successfully deployed TakTuk instances
Local node hostname as given to TakTuk (on the command line)
List of pids (separated by spaces) of commands executed by the local TakTuk instance.
The logical rank of the local instance
The TakTuk command line and the taktuk::send routine accept a set
specification as destination host(s). A set specification is made of interval
specifications separated by slashes. An interval specification is made of a
single number or two numbers separated by a dash. Of course the two numbers
specifying an interval must be given in increasing order.
The remote peers included in a set specification are all the peer which logical number belong to at least one interval of the set. Here are some exemples of set specifications : 1
the peer numbered 1 2-7
the peers numbered 2,3,4,5,6 and 7 2-4/1/10
the peers numbered 1,2,3,4 an 10
The following examples illustrate the basic use of TakTuk on a few machines and the use of developper options. Notice that TakTuk is designed to scale to much more peers than the number involved in these examples.
the simplest way to use TakTuk is to make it selfpropagate with option
-s. In this case, the basic remote execution of hostname on the host
toto.nowhere.com can be written :
taktuk -s -m toto.nowhere.com broadcast exec [hostname]
In this example, -s asks TakTuk to propagate its own code on remote
hosts. It can be removed by installing the taktuk executable on
toto.nowhere.com. By the following we will assume that TakTuk is
installed on all the remote hosts.
The -m toto.nowhere.com describe the set of remote hosts to be contacted by
TakTuk and broadcast exec [hostname] is a command that will be executed
by the TakTuk interpreter.
This example can be written in many other ways. In interactive mode, the same execution might become:
taktuk -m toto.nowhere.com
here TakTuk is blocked waiting for commands from stdin. Thus, we just have to type:
broadcast exec {hostname}
Ctrl-D
here you can notice that parameters to the exec TakTuk command (as all
commands parameters) can be enclosed in any reasonable pair of delimiters.
We might also write the list of hosts involved in the command in
a file machine that contains:
toto.nowhere.com
and the TakTuk command becomes:
taktuk -f machine broadcast exec -hostname-
We could also use another file options that contains:
-f machine
and use it as the options line given to TakTuk:
taktuk -F options broadcast exec \(hostname\)
Finally, everything could be stored in a last file command_line that
contains:
-f machine broadcast exec =hostname=
and the following command achieve the same result:
taktuk -F command_line
All of these variants have the same effect: they execute hostname on toto.nowhere.com and the output of the program is forwarded to the localhost. In this case:
toto.nowhere.com: hostname: somepid: output > toto.nowhere.com
notice that braces for command parameters can contain other braces as long as the imbrication is well balanced:
taktuk -m localhost broadcast exec '[if [ $RANDOM -gt 10000 ];then echo greater;else echo lower;fi]'
In this example, quotes are necessary to prevent the shell from interpreting
the $ character. In this case the variable will be interpolated only on
remote hosts. This same example can also be expressed using shortcuts and
intercative mode:
taktuk -m localhost
then type:
b e [if [ $RANDOM -gt 10000 ];then echo greater;else echo lower;fi]
Ctrl-D
but be careful of shell arguments parsing and -m option arguments which
supersede command braces:
taktuk -m localhost -[ exec [cat -] -] broadcast input [hello]
in this last case, the first -] which should belong to the cat command
will be interpreted as closing the -[. To avoid this, you can use quotes:
taktuk -m localhost -[ exec '[cat -]' -] broadcast input [hello]
or other kind of braces:
taktuk -m localhost -[ exec {cat -} -] broadcast input [hello]
usually, if you want to be safe, you can quote all commands parameters. Notice that parameters should not be quoted in interactive mode as input lines are not interpreted by the shell.
Usually, TakTuk deploys itself using a relatively flat tree because its
default window size is quite large (10 simultaneous ongoing connections).
Using a smaller window size will result in a deeper tree although it also
depends on the local load of the deployment nodes. You can use the
TakTuk print_tree command to print the tree constructed by TakTuk.
Notice that it is usually a bad idea to use a too large window as it results in too much local load and bad distribution of work (something like 10 is often sufficient).
You can also force TakTuk to use more specific topologies. For instance, to
execute echo $$ using a flat-tree as deployment topology, just disable
work-stealing in TakTuk:
taktuk -d -1 -m host1 -m host2 -m host3 broadcast exec '[echo $$]'
and to use a chain-like topology, either encode the topology in arguments structure:
taktuk -m host1 -[ -m host2 -[ -m host3 -] -] broadcast exec '[echo $$]'
or limits the arity of the dynamic tree to 1:
taktuk -d 1 -m host1 -m host2 -m host3 broadcast exec '[echo $$]'
Finally, the default will use a dynamicly constructed topology:
taktuk -d 0 -m host1 -m host2 -m host3 broadcast exec '[echo $$]'
On a lightweight grid, because of locality issues (ldap cache, network topology, ...) it seems interesting to separate the deployment of each subcluster. To do this, it is possible to choose one node in each subcluster that will be deployed first and from which the rest of the cluster will be deployed:
taktuk -m node1.cluster1 -[ -m node2.cluster1 -m node3.cluster1 -m node4.cluster1 -] -m node1.cluster2 -[ -m node2.cluster2 -m node3.cluster2 -m node4.cluster2 -] broadcast exec [hostname]
This command has the effect of deploying TakTuk on two clusters (cluster 1
and 2) made of four nodes (node 1 to 4) using node1 of each cluster to deploy
the other nodes in the same cluster. Finally, once the deployment is complete,
it executes the command hostname on all these nodes.
each distinct host can be given its own command using arguments:
./taktuk -m host1 -[ exec [hostname] -] -m host2 -[ exec [id] -] -m host3 -[ exec '[echo $TAKTUK_RANK; ls]' -] quit
but this could also be given using set specification (in this case logical number are used for hosts):
./taktuk -m host1 -m host3 -m host8 1 exec [hostname], 2 exec [id], 3 exec '[echo $TAKTUK_RANK; ls]'
or in interactive mode:
./taktuk -m host1 -m host3 -m host8
1 exec [hostname]
2 exec [id]
3 exec [echo $TAKTUK_RANK; ls]
Ctrl-D
spreading and executing a perl script named essai.pl on three hosts, knowing
that taktuk and essai.pl are only present on the root node:
taktuk -s -m host1 -m host2 -m host3
broadcast exec [perl -- -]
broadcast file_input [essai.pl]
broadcast close
copying a file named message.txt to the /tmp directory of each remote
host:
taktuk -s -m host1 -m host2 -m host3
broadcast exec [cat - >/tmp/message.txt]
broadcast file_input [message.txt]
broadcast close
be careful to shell interpretation when typing everything directly on the command line (each command must be a single argument):
taktuk -s -m host1 -m host2 -m host3 broadcast exec '[cat - \>/tmp/message.txt]'\;broadcast file_input [message.txt]
establishing a point-to-point communication. Assume the file
communication.pl contains the following Perl script:
if ($ENV{'TAKTUK_RANK'} == 1)
{
print "I'm process 1\n";
if ($ENV{'TAKTUK_COUNT'} > 1)
{
taktuk::send(to=>2, body=>"Hello world");
}
}
elsif ($ENV{'TAKTUK_RANK'} == 2)
{
print "I'm process 2\n";
my ($to, $from, $message) = taktuk::recvZ<>();
print "Process $to received $message from $from\n";
}
then the execution of the following command:
taktuk -m localhost -m localhost broadcast taktuk_perl []\;broadcast file_input [communication.pl]\;broadcast close
would produce an output similar to:
Astaroth.local: taktuk_perl: 3523: output > I'm process 2
Astaroth.local: taktuk_perl: 3523: output > Process 2 received Hello world from 1
Astaroth.local: taktuk_perl: 3523: status > 0
Astaroth.local: taktuk_perl: 3524: output > I'm process 1
Astaroth.local: taktuk_perl: 3524: status > 0
if the file communication.pl was placed in the login directory of the user,
this could have also been executed by the more simple:
taktuk -m localhost -m localhost broadcast taktuk_perl [communication.pl]
making each site echo its rank without command status information:
taktuk -o status -m host1 -m host2 broadcast exec '[echo $TAKTUK_RANK]'
or removing the prompt before each line of output from commands:
taktuk -o output='"$line\n"' -m host1 -m host2 broadcast exec '[echo $TAKTUK_RANK]'
or even changing the prompt to make it display only the stream type:
taktuk -o default='"$type > $line\n"' -m host1 -m host2 broadcast exec '[echo $TAKTUK_RANK]'
and it also possible to redirect the status to file descriptor 2 only for the second host:
taktuk -m host1 -R status=2 -m host2 broadcast exec '[echo $TAKTUK_RANK]'
and so on...
debugging in TakTuk is made using debug, warning and error functions of the package diagnostic. These routines, depending on the debugging level of the package in which they are called, produce an output that is propagated up to the root node and printed on screen.
By default the debugging level of packages is set to 2 (everything is printed out except debug messages). It might be changed for each package using the -D option. For instance the following code executes true on toto.nowhere.com and prints out every bit of internal messaging:
taktuk -D default=1 -m toto.nowhere.com broadcast exec [true]
but one could have executed the same command keeping only messages from the scheduler package:
taktuk -D scheduler=1 -m toto.nowhere.com broadcast exec [true]
or ensuring an execution exempted of any warning or error messages:
taktuk -D default=4 -m toto.nowhere.com broadcast exec [true]
the internal message server used in TakTuk for the management of logical network construction, commands execution and I/O forwarding can be exposed using the -r option:
taktuk -r
Notice that in this mode the behavior of TakTuk can seem very cryptic. This is not intended for ordinary users.
The development of TakTuk is still in progress, so there are propably numbers of bugs. For now, the following characteristics (some of them are not really bugs) have been identified :
this is not a bug: if the Time::HiRes Perl Core module is not installed on your system (although this is not likely to happen except on very old systems), then TakTuk timeouts do not work at all (they just never expire).
currently TakTuk do not quit (even if asked to do so) if some local commands do not terminate upon the closing of their standard input. Interrupting TakTuk actually terminate the engine but do not kill these commands. We still have to decide if this is the appropriate behavior.
broadcasts and multicast (set specifications) are not synchronized the same way
in TakTuk. Thus, broadcast commands given after multicast commands might be
executed before these last ones. Use the synchronize command to avoid this.
implementation of the synchronize command do not allow a proper
synchronization of the print_tree command. This might change in the future
but is not in the priority list.
this is not really a bug : on some clusters, the connections are VERY slow when doing a large deployment (over 50 nodes) with self propagation. This result in a total time for the deployment of 50 nodes that can take several minutes (up to half an hours in some cases). I don't know what's the exact cause (ldap ? local load ? writes block ?) but it can be fixed by adding connection timeouts to connectors.
this is not really a bug : when a machine is highly loaded, fork may fail. If this happen to a local command, a distributed application may deadlock, because the taktuk instances numbering is correct but some instance didn't fork the local command. The simplest fix is to use timeouts on taktuk::recv().
taktukcomm(3)
The original concept of TakTuk has been proposed by Cyrille Martin in his PhD thesis. People involved in this work include Jacques Briat, Olivier Richard, Thierry Gautier and Guillaume Huard.
The author of the perl version and current maintainer of the package is Guillaume Huard.
TakTuk is provided under the terms of the GNU General Public License version 2 or later.