Go to the first, previous, next, last section, table of contents.


Writing `TOP-C' Applications

The Main TOP-C Library Calls

This section assumes a knowledge of the basic concepts in section Overview of `TOP-C/C++'.

Every `TOP-C' application must include a `topc.h' header, open with TOPC_init(), call TOPC_master_slave() one or more times, and then close with TOPC_finalize().

#include <topc.h>
      Required at head of any file using TOPC library calls.

Function: void TOPC_init ( int *argc, char ***argv )
Required before first occurrence of TOPC_master_slave(); Recommended to place this as first executable statement in main(). It will strip off extra `TOP-C' and communication layer arguments such as --TOPC_verbose, which are added by `TOP-C'. See section Command Line Parameters in TOP-C Applications, for extended syntax.
Function: void TOPC_finalize ( void )
Placed after last `TOP-C' command.
Function: void TOPC_master_slave
( TOPC_BUF (*generate_task_input)(),
TOPC_BUF (*do_task)(void *input),
TOPC_ACTION (*check_task_result)(void *input, void *output),
void (*update_shared_data)(void *input, void *output)
)
Primary call, passed four application callbacks to `TOP-C'. One can have multiple calls to TOPC_master_slave(), each invoking different callback functions, between TOPC_init() and TOPC_finalize().

A task input or task output is simply a buffer of bytes, specified by TOPC_MSG().

Function: TOPC_BUF TOPC_MSG ( void *buf, int buf_size )
Must be returned by GenerateTaskInput() and DoTask(). Specifies arbitrary user data structure. In DoTask(), buf must not reference a buffer on the stack; Declare buffer static to avoid this. (This requirement is relaxed in the experimental version.) `TOP-C' will copy buf to `TOP-C' space. It remains the responsibility of the application to free or reuse the space pointed to by buf. If you create message buffers dynamically, using malloc, the following example shows how to easily free the buffer before creating a new one. Note that the example declares buf to be static.
        TOPC_BUF GenerateTaskInput() {
          static void *buf = NULL; 
          if ( buf != NULL ) { free(buf); mydata = NULL; }
          ... [ Compute buf_size for new message ] ...
          buf = malloc( buf_size );
          ... [ Add new message data to buf ] ...
          return TOPC_MSG(buf, buf_size);
        }

EXAMPLE:

    TOPC_BUF convert_string_to_msg( char *mystring ) {
        if (mystring == NULL) return TOPC_MSG(NULL,0);
        else return TOPC_MSG(mystring, strlen(mystring)+1);
    }

Callback Functions for TOPC_master_slave()

The application writer must define the following four callback functions (although the last can be NULL). The first two functions return a TOPC_BUF, which is produced by TOPC_MSG().

Function: TOPC_BUF GenerateTaskInput ( void )
acts on master; returns a data structure specified by TOPC_MSG(buf, buf_size). It should return NOTASK, when there are no more tasks, and it should be prepared to return NOTASK again if invoked again.
Function: TOPC_BUF DoTask ( void *input )
acts on slave; operates on the result of GenerateTaskInput(); returns a data structure specified by TOPC_MSG(buf, buf_size). buf must be a static or global user buffer.
Function: TOPC_ACTION CheckTaskResult ( void *input, void *output)
acts on master; operates on the result of DoTask(); returns an ACTION that determines what happens to the task next. The terminology result refers to an `(input, output)' pair. An easy way to write CheckTaskResult() appears in the example for the utility TOPC_is_up_to_date(). See the section section TOP-C Utilities, for more details.
Function: void UpdateSharedData ( void *input, void *output )
acts on master and all slaves; operates on the result of DoTask(), and the original task returned by GenerateTaskInput(); called only if CheckTaskResult() returned UPDATE; useful for updating non-shared, global variables in all processes; The pointer argument, update_shared_data, may be NULL if an application never requests an UPDATE action. In a shared memory environment, only the master calls UpdateSharedData(). See the section section Optimizing TOP-C Code for the Shared Memory Model, for more details.

Actions Returned by CheckTaskResult()

The actions returned by CheckTaskResult() are:

Action: TOPC_ACTION NO_ACTION
C constant, causing no further action for task
Action: TOPC_ACTION UPDATE
C constant, invoking UpdateSharedData( void *task) (see below) also updates bookkeeping for sake of TOPC_is_up_to_date() (see section TOP-C Utilities)
Action: TOPC_ACTION REDO
Invoke DoTask() on original task input again, and on original slave; useful if shared data has changed since original invocation of DoTask(); see TOPC_is_up_to_date(), below. See section Strategies for Greater Concurrency, for slave strategies to efficiently process a REDO action.
Action: TOPC_ACTION CONTINUATION ( void *next_input )
CONTINUATION() is a parametrized action that may be returned. It's like REDO, but if the result of CONTINUATION( next_input ) is returned by CheckTaskResult(), then DoTask( next_input ) is called on the original slave. useful if only the master can decide whether task is complete. Note that any pending calls to UpdateSharedData() will have occurred on the slave before the new call to DoTask(). Hence, this allows an extended conversation between master and slave, in which the slave continues to receive updates of the shared data before each new input from the master. Note also that even though a CONTINUATION action returns to the original slave, any previous pointers to input buffers (and pointers to output buffers from intervening UPDATE actions) will no longer be valid. Data from previous buffers should have been copied into global variables. In the case of the shared memory model, those global variables must be thread-private. (see section Thread-Private Global Variables)

TOP-C Utilities

`TOP-C' also defines some utilities.

Function: TOPC_BOOL TOPC_is_up_to_date ( void )
returns TRUE or FALSE (1 or 0); returns TRUE if and only if CheckTaskResult() has not returned the result UPDATE (invoking UpdateSharedData()) between the time when GenerateTaskInput() was originally called on the current task, and the time when the corresponding CheckTaskResult() was called. Typical usage:
          TOPC_ACTION CheckTaskResult( void *input, void *output )
          { if (input == NULL) return NO_ACTION;
            else if (! TOPC_is_up_to_date()) return NO_ACTION;
            else return REDO;
          }

Function: int TOPC_rank ( void )
Unique ID of process or thread. Master always has rank 0. Slaves have contiguous ranks, beginning at 1.
Function: TOPC_BOOL TOPC_is_master ( void )
Returns boolean, 0 or 1, depending on if this is master. Equivalent to TOPC_rank() == 0.
Function: int TOPC_num_slaves ( void )
Total number of slaves.
Function: int TOPC_node_count ( void )
Total number of processes or threads. Equivalent to TOPC_num_slaves() + 1.
Function: int get_last_source ( void )
/* NOT CURRENTLY SUPPORTED */ returns unique id, as a C int, for the last process from which a message was received. This is trivial, when called on a slave, but it is useful on the master.

Miscellaneous Issues in Writing TOP-C Applications

TOPC_MSG(): Memory Allocation of TOP-C Message Buffers

Recall the syntax for creating a message buffer of type TOPC_BUF: TOPC_MSG(buf, buf_size). The two callback functions GenerateTaskInput() and DoTask() both return such a message buffer. In the case of GenerateTaskInput(), `TOP-C' saves a copy of the buffer, which becomes an input argument to CheckTaskResult() and to UpdateSharedData on the master.. Hence, if buf points to a temporarily allocated buffer, it is the responsibility of the `TOP-C' callback function to free the buffer only after the callback function has returned. This seeming contradiction can be easily handled by the following code reproduced from section The Main TOP-C Library Calls.

    TOPC_BUF GenerateTaskInput() {
      static void *buf = NULL;
      if ( buf != NULL ) { free(buf); mydata = NULL; }
      ... [ Compute buf_size for new message ] ...
      buf = malloc( buf_size );
      ... [ Add new message data to buf ] ...
      return TOPC_MSG(buf, buf_size);
    }

Note that buf is allocated as a static local variable. Currently, `TOP-C' restricts the buf of TOPC_MSG(buf, buf_size) to point to a buffer that is in the heap (not on the stack). Hence, buf must not point to non-static local data. This restriction may be relaxed in a future version of `TOP-C'.

Local Shared Data

Sometimes your shared data may be on the stack, instead of in a global variable. In order to make this local data accessible to a `TOP-C' callback function, such as UpdateSharedData, you will need to create a global variable pointing to your local shared data. The example below makes the local shared data in the array data[] in the function foo() available to the callback function UpdateSharedData().

  int *global_copy_of_data;
  void UpdateSharedData( int *input, void *output) {
    global_copy_of_data[*input] = output;
  }
  int foo() {
    int data[SIZE];
    global_copy_of_data = data;
    TOPC_master_slave(...);
  }

Of course, `C++' provides a more natural way to handle this by placing global_copy_of_data, foo and UpdateSharedData in a single class.

Marshalling Indirect Pointers into `TOP-C' Task Buffers

If you use a distributed memory model and the buffer pointed to by input includes fields with their own pointers, the application must first follow all pointers and copy into a new buffer all data referenced directly or indirectly by input. The new buffer can then be passed to TOPC_MSG(). This copying process is called marshalling.

If following all pointers is a burden, then one can load the application on the master and slaves at a common absolute address, and insure that all pointer references have been initialized before the first call to TOPC_master_slave(). In `gcc', one specifies an absolute load address with code such as:

  gcc -Wl,-Tdata -Wl,-Thex_addr ...

These flags are for the data segment. If the pointers indirectly reference data on the stack, you may have to similarly specify stack absolute addresses. Choosing a good hex_addr for all machines may be a matter of trial and error. In a test run, print out the absolute addresses of some pointer variables near the beginning of your data memory.

Specifying an absolute load address has many risks, such as if the master and slaves use different versions of the operating system, the compiler, other software, or different hardware configurations. Hence, this technique is recommended only as a last resort.

Caveats

IMPORTANT: `TOP-C' sets alarm() before waiting to receive message from master. By default, if the master does not reply in one hour, then the slave receives SIGALRM and dies. This is to prevent runaway processes in dist. memory version when master dies without killing all slaves. section Long Jobs and Courtesy to Others, in order to change this default. If your applications also uses SIGALRM, then run your application with --TOPC_slave_timeout=0 and `TOP-C' will not use SIGALRM.

GenerateTaskInput() and DoTask() This memory is managed by `TOP-C'.

The slave process attempts to set current directory to same as master inside TOPC_init() and produces a warning if unsuccessful.

When a task buffer is copied into `TOP-C' space, it becomes word-aligned. If the buffer was originally not word-aligned, but some field in the buffer was word-aligned, the internal field will no longer be word-aligned. On some architectures, casting a non-word-aligned field to `int' or certain other types will cause a bus error.

Optimizing TOP-C Code for the Shared Memory Model

The `TOP-C' programmer's model changes slightly for shared memory. With careful design, one can use the same application source code both for distributed memory and shared memory architectures. Processes are replaced by threads. UpdateSharedData() is executed only by the master thread, and not by any slave thread. As with distributed memory, TOPC_MSG() buffers are copied to `TOP-C' space (shallow copy), and it is the responsibility of the application to free any application buffers that it may have created. Furthermore, since the master and slaves share memory, `TOP-C' creates the slaves only during the first call to master_slave. If a slave needs to initialize any private data (see TOPC_private_global, below), then this can be done by the slave the first time that it gains control through DoTask().

Two issues arise in porting a distributed memory `TOP-C' application to shared memory.

  1. reader-write synchronization: DoTask() must not read shared data while UpdateSharedData() (on the master) simultaneously writes to the shared data.
  2. creating thread-private (unshared) global variables:

Most `TOP-C' applications for the distributed memory model will run unchanged in the shared memory model. In some cases, one must add additional `TOP-C' code to handle these additional issues. In all cases, one can easily retain compatibility with the distributed memory model.

Reader-Writer Synchronization

In shared memory, `TOP-C' uses a classical single-writer, multiple-reader strategy with writer-preferred for lock requests. By default, DoTask() acts as the critical section of the readers (the slave threads) and UpdateSharedData() acts as the critical section of the writer (the master thread). `TOP-C' sets a read lock around all of DoTask() and a write lock around all of UpdateSharedData().

As always in the `TOP-C' model, it is an error if an application writes to shared data outside of UpdateSharedData(). Note that GenerateTaskInput() and CheckTaskResult() can safely read the shared data without a lock in this case, since these routines and UpdateSharedData() are all invoked only by the master thread.

As always in the `TOP-C' model, it is an error if an application writes to shared data outside of UpdateSharedData(). Note that GenerateTaskInput() and CheckTaskResult() can safely read the shared data without a lock in this case, since these routines and UpdateSharedData() are all invoked only by the master thread.

The default behavior implies that DoTask() and UpdateSharedData() never run simultaneously. Optionally, one can achieve greater concurrency through a finer level of granularity by declaring to `TOP-C' which sections of code read or write shared data.

Function: void TOPC_ATOMIC_READ ( 0 ) { ... C code ... }
Function: void TOPC_ATOMIC_WRITE ( 0 ) { ... C code ... }
This sets a global read or write lock in effect during the time that C code is being executed. If a thread holds a write lock, no thread may hold a read lock. If no thread holds a write lock, arbitrarily many threads hold a read lock. If a thread requests a write lock, no additional read locks will be granted until after the write lock has been granted.

The number 0 refers to page 0 of shared data. `TOP-C' currently supports only a single common page of shared data, but future versions will support multiple pages. In the future, two threads will be able to simultaneously hold write locks if they are for different pages.

The following alternatives to TOPC_ATOMIC_READ() and TOPC_ATOMIC_WRITE() are provided for greater flexibility.

Function: void TOPC_BEGIN_ATOMIC_READ ( 0 )
Function: void TOPC_END_ATOMIC_READ ( 0 )
Function: void TOPC_BEGIN_ATOMIC_WRITE ( 0 )
Function: void TOPC_END_ATOMIC_WRITE ( 0 )
The usage is the same as for TOPC_ATOMIC_READ and TOPC_ATOMIC_WRITE.

In the distributed memory model of `TOP-C', all of the above invocations for atomic reading and writing are ignored, thus retaining full compatibility between the shared and distributed memory models.

Thread-Private Global Variables

The only variables that are thread-private by default in shared memory are those on the stack (non-static, local variables). All other variables exist as a single copy, shared by all threads. `TOP-C' provides primitives to declare a single global variable, for which each thread has a private, global copy. `TOP-C' allows the application programmer to declare the type of that variable. If more than one variable is desired, this can be emulated by declaring a struct. `TOP-C' provides for a single, global, thread-private variable using the primitives below.

Variable: TOPC_private_global
A pre-defined thread-private variable of type, TOPC_private_global_t. It may be used like any C variable, and each thread has its own private copy that will not be shared.
Type: TOPC_private_global_t
Initially, undefined. User must define this type using typedef if TOPC_private_global is used.

If you need more than one global, thread-private variable, define TOPC_private_global_t as a struct, and use each field as a separate thread-private variable.

EXAMPLE:

/* The pre-defined thread-private TOP_C variable, TOPC_private_global,
 *   will have the type:  struct {int my_rank; int rnd;}
 */
typedef struct {int my_rank; int rnd;} TOPC_private_global_t;
/* Re-define TOPC_private_global to more meaningful name for application */
#define mystruct TOPC_private_global
void foo() {
  mystruct.my_rank = TOPC_rank();
  mystruct.rnd = rand();
  printf("Slave %d random number:  %d\n", mystruct.my_rank, mystruct.rnd);
}
void bar() {
foo();
  if (mystruct.my_rank != TOPC_rank()) printf("ERROR\n");
  printf("Slave %d random number:  %d\n", mystruct.my_rank, mystruct.rnd);
}

The shared memory model, like any `SMP' code, allows the master and slaves to communicate through global variables, which are shared by default. It is recommended not to use this feature, and instead to maintain communication through TOPC_MSG(), for ease of code maintenance, and to maintain portability with the other `TOP-C' models (distributed memory and sequential). If you do use your own global shared variables between master and slaves, be sure to declare it volatile.

  volatile int myvar;

ANSI C requires this qualifier if a variable may be changed while a given thread is keeping an older value in its register, and your program may not run correctly without the qualifier.

SMP Performance

Note that `SMP' involves certain performance issues that do not arise in other modes. If you find a lack of performance, please read section Improving Performance. Also, note that the vendor-supplied compiler, cc, is often recommended over gcc for `SMP', due to specialized vendor-specific architectural issues.

Optimizing TOP-C Code for the Sequential Memory Model

`TOP-C' also provides a sequential memory model. That model is useful for first debugging an application in a sequential context, and then re-compiling it with one of the parallel `TOP-C' libraries for production use. The application code for the sequential library is usually both source and object compatible with the application code for a parallel library. The sequential library emulates an application with a single `TOP-C' library.

The sequential memory model emulates an application in which DoTask() is executed in the context of the single slave process/thread, and all other code is executed in the context of the master process/thread. This affects the values returned by TOPC_is_master() and TOPC_rank(). In particular, conditional code for execution on the master will work correctly in the sequential memory model, but the following conditional code for execution on the slave will probably not work correctly.

int main( int argc, char *argv[] ) {
  TOPC_init( &argc, &argv );
  if ( TOPC_is_master() )
    ...;  /* is executed in sequential model */
  else
    ...;  /* is never executed in sequential model */
  TOPC_master_slave( ..., ..., ..., ...);
  TOPC_finalize();
}


Go to the first, previous, next, last section, table of contents.