This section assumes a knowledge of the basic concepts in section Overview of `TOP-C/C++'.
Every `TOP-C' application must include a `topc.h' header, open
with TOPC_init()
, call TOPC_master_slave()
one or more
times, and then close with TOPC_finalize()
.
#include <topc.h>
Required at head of any file using TOPC library calls.
TOPC_master_slave()
;
Recommended to place this as first executable statement in
main()
. It will strip off extra `TOP-C' and
communication layer arguments such as --TOPC_verbose
,
which are added by `TOP-C'.
See section Command Line Parameters in TOP-C Applications, for extended syntax.
TOPC_master_slave()
, each invoking different callback functions,
between TOPC_init()
and TOPC_finalize()
.
A task input or task output is simply a buffer of bytes, specified by
TOPC_MSG()
.
GenerateTaskInput()
and DoTask()
.
Specifies arbitrary user data structure. In DoTask()
,
buf must not reference a buffer on the stack; Declare
buffer static
to avoid this. (This requirement is relaxed in the experimental
version.) `TOP-C' will copy buf to `TOP-C' space.
It remains the responsibility of the application to free or
reuse the space pointed to by buf. If you create
message buffers dynamically, using malloc
, the following example
shows how to easily free the buffer before creating a new one.
Note that the example declares buf
to be static.
TOPC_BUF GenerateTaskInput() { static void *buf = NULL; if ( buf != NULL ) { free(buf); mydata = NULL; } ... [ Compute buf_size for new message ] ... buf = malloc( buf_size ); ... [ Add new message data to buf ] ... return TOPC_MSG(buf, buf_size); }
EXAMPLE:
TOPC_BUF convert_string_to_msg( char *mystring ) { if (mystring == NULL) return TOPC_MSG(NULL,0); else return TOPC_MSG(mystring, strlen(mystring)+1); }
TOPC_master_slave()
The application writer must define the following four callback
functions (although the last can be NULL
). The first two
functions return a TOPC_BUF
, which is produced by TOPC_MSG()
.
TOPC_MSG(buf, buf_size)
.
It should return NOTASK
, when there are no more tasks, and it
should be prepared to return NOTASK
again if invoked again.
GenerateTaskInput()
;
returns a data structure specified by
TOPC_MSG(buf, buf_size)
.
buf must be a static or global user buffer.
DoTask()
;
returns an ACTION that determines what happens to the task next.
The terminology result refers to an `(input, output)' pair.
An easy way to write CheckTaskResult()
appears in the
example for the utility TOPC_is_up_to_date()
.
See the section section TOP-C Utilities, for more details.
DoTask()
, and the original task returned by
GenerateTaskInput()
;
called only if CheckTaskResult()
returned UPDATE
;
useful for updating non-shared, global variables in all processes;
The pointer argument, update_shared_data, may be NULL
if
an application never requests an UPDATE
action.
In a shared memory environment, only the master calls
UpdateSharedData()
. See the section section Optimizing TOP-C Code for the Shared Memory Model, for more details.
CheckTaskResult()
The actions returned by CheckTaskResult()
are:
UpdateSharedData( void *task)
(see below)
also updates bookkeeping for sake of TOPC_is_up_to_date()
(see section TOP-C Utilities)
DoTask()
on original task input again, and on
original slave; useful if shared data has changed since original
invocation of DoTask()
; see TOPC_is_up_to_date()
,
below. See section Strategies for Greater Concurrency,
for slave strategies to efficiently process a REDO
action.
CONTINUATION()
is a parametrized action that may be returned.
It's like REDO
, but if the result of
CONTINUATION( next_input )
is returned by
CheckTaskResult()
,
then DoTask( next_input )
is called on the original slave.
useful if only the master can decide whether task is complete.
Note that any pending calls to UpdateSharedData()
will have
occurred on the slave before the new call to DoTask()
.
Hence, this allows an extended conversation between master and
slave, in which the slave continues to receive updates of the
shared data before each new input from the master.
Note also that even though a CONTINUATION
action returns
to the original slave, any previous pointers to input buffers
(and pointers to output buffers from intervening UPDATE
actions) will no longer be valid. Data from previous buffers
should have been copied into global variables.
In the case of the shared memory model, those global variables must be
thread-private. (see section Thread-Private Global Variables)
`TOP-C' also defines some utilities.
CheckTaskResult()
has not returned the result UPDATE
(invoking
UpdateSharedData()
)
between the time when GenerateTaskInput()
was originally
called on the
current task, and the time when the corresponding
CheckTaskResult()
was called.
Typical usage:
TOPC_ACTION CheckTaskResult( void *input, void *output ) { if (input == NULL) return NO_ACTION; else if (! TOPC_is_up_to_date()) return NO_ACTION; else return REDO; }
TOPC_rank() == 0
.
TOPC_num_slaves() + 1
.
/* NOT CURRENTLY SUPPORTED */
returns unique id, as a C int, for the last process from
which a message was received. This is trivial, when called on
a slave, but it is useful on the master.
TOPC_MSG()
: Memory Allocation of TOP-C Message Buffers
Recall the syntax for creating a message buffer of type TOPC_BUF
:
TOPC_MSG(buf, buf_size)
. The two callback functions
GenerateTaskInput()
and DoTask()
both return such a
message buffer. In the case of
GenerateTaskInput()
, `TOP-C' saves a copy of the buffer,
which becomes an input argument to CheckTaskResult()
and to UpdateSharedData
on the master..
Hence, if buf points to a temporarily allocated buffer,
it is the responsibility of the `TOP-C' callback function to free the
buffer only after the callback function has returned.
This seeming contradiction can be easily handled by the following code
reproduced from section The Main TOP-C Library Calls.
TOPC_BUF GenerateTaskInput() { static void *buf = NULL; if ( buf != NULL ) { free(buf); mydata = NULL; } ... [ Compute buf_size for new message ] ... buf = malloc( buf_size ); ... [ Add new message data to buf ] ... return TOPC_MSG(buf, buf_size); }
Note that buf
is allocated as a static local
variable. Currently, `TOP-C' restricts the buf of
TOPC_MSG(buf, buf_size)
to point to a buffer that is in
the heap (not on the stack). Hence, buf must not point to
non-static local data. This restriction may be relaxed in a future
version of `TOP-C'.
Sometimes your shared data may be on the stack, instead of in a global
variable.
In order to make this local data accessible to a `TOP-C' callback
function, such as UpdateSharedData, you will need to create a global
variable pointing to your local shared data. The example below makes
the local shared data in the array data[]
in the function
foo()
available to the callback function
UpdateSharedData()
.
int *global_copy_of_data; void UpdateSharedData( int *input, void *output) { global_copy_of_data[*input] = output; } int foo() { int data[SIZE]; global_copy_of_data = data; TOPC_master_slave(...); }
Of course, `C++' provides a more natural way to handle this by
placing global_copy_of_data
, foo
and
UpdateSharedData
in a single class.
If you use a distributed memory model and the buffer pointed to by
input
includes fields with their own pointers, the application
must first follow all pointers and copy into a new buffer all data
referenced directly or indirectly by input
. The new buffer can
then be passed to TOPC_MSG()
. This copying process is called
marshalling.
If following all pointers is a burden, then one can
load the application on the master and slaves at a common absolute
address, and insure that all pointer references have been initialized
before the first call to TOPC_master_slave()
. In `gcc',
one specifies an absolute load address with code such as:
gcc -Wl,-Tdata -Wl,-Thex_addr ...
These flags are for the data segment. If the pointers indirectly reference data on the stack, you may have to similarly specify stack absolute addresses. Choosing a good hex_addr for all machines may be a matter of trial and error. In a test run, print out the absolute addresses of some pointer variables near the beginning of your data memory.
Specifying an absolute load address has many risks, such as if the master and slaves use different versions of the operating system, the compiler, other software, or different hardware configurations. Hence, this technique is recommended only as a last resort.
IMPORTANT:
`TOP-C' sets alarm()
before waiting to receive
message from master. By default,
if the master does not reply in one hour, then the slave receives
SIGALRM
and dies.
This is to prevent runaway processes in dist. memory version when master dies
without killing all slaves. section Long Jobs and Courtesy to Others,
in order to change this default.
If your applications also uses SIGALRM
, then run your
application with --TOPC_slave_timeout=0
and `TOP-C'
will not use SIGALRM
.
GenerateTaskInput()
and DoTask()
This memory is managed by `TOP-C'.
The slave process attempts to set current directory to same as master
inside TOPC_init()
and produces a warning if unsuccessful.
When a task buffer is copied into `TOP-C' space, it becomes word-aligned. If the buffer was originally not word-aligned, but some field in the buffer was word-aligned, the internal field will no longer be word-aligned. On some architectures, casting a non-word-aligned field to `int' or certain other types will cause a bus error.
The `TOP-C' programmer's model changes slightly for shared
memory. With careful design, one can use the same application source
code both for distributed memory and shared memory architectures.
Processes are replaced by threads. UpdateSharedData()
is
executed only by the master thread, and not by any slave thread. As
with distributed memory, TOPC_MSG()
buffers are copied to
`TOP-C' space (shallow copy), and it is the responsibility of the
application to free any application buffers that it may have created.
Furthermore, since the master and slaves share memory, `TOP-C'
creates the slaves only during the first call to master_slave. If a
slave needs to initialize any private data (see
TOPC_private_global
, below), then this can be done by the slave
the first time that it gains control through DoTask()
.
Two issues arise in porting a distributed memory `TOP-C' application to shared memory.
DoTask()
must not read
shared data while UpdateSharedData()
(on the master)
simultaneously writes to the shared data.
Most `TOP-C' applications for the distributed memory model will run unchanged in the shared memory model. In some cases, one must add additional `TOP-C' code to handle these additional issues. In all cases, one can easily retain compatibility with the distributed memory model.
In shared memory, `TOP-C' uses a classical single-writer,
multiple-reader strategy with writer-preferred for lock requests.
By default, DoTask()
acts as the critical section of the
readers (the slave threads) and UpdateSharedData()
acts as the
critical section of the writer (the master thread).
`TOP-C' sets a read lock around all of DoTask() and a write
lock around all of UpdateSharedData().
As always in the `TOP-C' model,
it is an error if an application writes to shared data outside
of UpdateSharedData()
. Note that GenerateTaskInput()
and CheckTaskResult()
can safely read the shared data without
a lock in this case, since these routines and UpdateSharedData()
are all invoked only by the master thread.
As always in the `TOP-C' model,
it is an error if an application writes to shared data outside
of UpdateSharedData()
. Note that GenerateTaskInput()
and CheckTaskResult()
can safely read the shared data without
a lock in this case, since these routines and UpdateSharedData()
are all invoked only by the master thread.
The default behavior implies that DoTask()
and
UpdateSharedData()
never run simultaneously. Optionally, one
can achieve greater concurrency through a finer level of granularity
by declaring to `TOP-C' which sections of code read or write
shared data.
The number 0 refers to page 0 of shared data. `TOP-C' currently supports only a single common page of shared data, but future versions will support multiple pages. In the future, two threads will be able to simultaneously hold write locks if they are for different pages.
The following alternatives to TOPC_ATOMIC_READ()
and TOPC_ATOMIC_WRITE()
are provided for greater flexibility.
TOPC_ATOMIC_READ
and TOPC_ATOMIC_WRITE
.
In the distributed memory model of `TOP-C', all of the above invocations for atomic reading and writing are ignored, thus retaining full compatibility between the shared and distributed memory models.
The only variables that are thread-private by default in
shared memory are those on the stack (non-static, local variables). All
other variables exist as a single copy, shared by all threads.
`TOP-C' provides primitives to declare a single global variable,
for which each thread has a private, global copy. `TOP-C' allows
the application programmer to declare the type of that variable. If
more than one variable is desired, this can be emulated by declaring a
struct
.
`TOP-C' provides for a single, global, thread-private variable using the
primitives below.
TOPC_private_global_t
. It may be
used like any C variable, and each thread has its own private
copy that will not be shared.
typedef
if TOPC_private_global
is used.
If you need more than one global, thread-private variable, define
TOPC_private_global_t
as a struct, and use each
field as a separate thread-private variable.
EXAMPLE:
/* The pre-defined thread-private TOP_C variable, TOPC_private_global, * will have the type: struct {int my_rank; int rnd;} */ typedef struct {int my_rank; int rnd;} TOPC_private_global_t; /* Re-define TOPC_private_global to more meaningful name for application */ #define mystruct TOPC_private_global void foo() { mystruct.my_rank = TOPC_rank(); mystruct.rnd = rand(); printf("Slave %d random number: %d\n", mystruct.my_rank, mystruct.rnd); } void bar() { foo(); if (mystruct.my_rank != TOPC_rank()) printf("ERROR\n"); printf("Slave %d random number: %d\n", mystruct.my_rank, mystruct.rnd); }
The shared memory model, like any `SMP' code,
allows the master and slaves to
communicate through global variables, which are shared by default.
It is recommended not to use this feature, and instead to maintain
communication through TOPC_MSG()
, for ease of code maintenance,
and to maintain portability with the other `TOP-C' models
(distributed memory and sequential). If you do use your own global shared
variables between master and slaves, be sure to declare it
volatile
.
volatile int myvar;
ANSI C requires this qualifier if a variable may be changed while a given thread is keeping an older value in its register, and your program may not run correctly without the qualifier.
Note that `SMP' involves certain performance issues that do not arise
in other modes. If you find a lack of performance, please read
section Improving Performance. Also, note that the vendor-supplied
compiler, cc
, is often recommended over gcc
for
`SMP', due to specialized vendor-specific architectural issues.
`TOP-C' also provides a sequential memory model. That model is useful for first debugging an application in a sequential context, and then re-compiling it with one of the parallel `TOP-C' libraries for production use. The application code for the sequential library is usually both source and object compatible with the application code for a parallel library. The sequential library emulates an application with a single `TOP-C' library.
The sequential memory model emulates an application in which
DoTask() is executed in the context of the single slave
process/thread, and all other code is executed in the context of the
master process/thread. This affects the values returned by
TOPC_is_master()
and TOPC_rank()
. In particular,
conditional code for execution on the master will work
correctly in the sequential memory model, but the following conditional
code for execution on the slave will probably not work correctly.
int main( int argc, char *argv[] ) { TOPC_init( &argc, &argv ); if ( TOPC_is_master() ) ...; /* is executed in sequential model */ else ...; /* is never executed in sequential model */ TOPC_master_slave( ..., ..., ..., ...); TOPC_finalize(); }
Go to the first, previous, next, last section, table of contents.