This file is intended as a "getting-started" guide to communications
in Karma. It should be used in conjuction with the library reference
documentation, which documents all the functions available for
communicating.


Purpose:
--------

Communications are useful when writing applications which are
comprised of many modules (processes). Using communications, one can
even write applications to perform parallel processing on a
heterogeneous network.


Architecture:
-------------

The communications support in Karma is built on a layered approach. In
fact, one of the original motivations for the layered approach to the
Karma library was the need to provide communications at a high level
of abstraction for most needs, but not to preclude access to more
basic communications when required. Also, in order to leave the future
of the communications open-ended, a layered approach, with increasing
levels of abstraction, was also seen to be useful.
The various packages in the Karma library which provide communications
support are listed below, grouped by layer.

  Level 1
  =======

The lowest level (most primitive) communications routines are in the
<r_> package in the Karma library. These routines provide a very low
level interface to Unix sockets (on Unix systems). Routines exist for
allocating ports and making connections to ports.
These routines are *not* recommended for use, as they are inherently
non-portable (outside of Unix, say) in their interactions, and too
primitive.
The <r_> package also provides other miscellaneous functions to
manipulate the process environment, determining the hostname of a
machine, computing port numbers, etc. These routines provide some OS
independent functions, and hence are preferred over the vendor
supplied routines.
The <r_> package is a key package in providing OS abstraction.

  Level 2
  =======

The basic Input/Output class in Karma is the "channel". This is
provided by the <ch_> package. A channel is a (possibly) full duplex
buffered stream of data. These are similar to the standard C library
FILE * streams, but fix a number of deficiencies in FILE streams, such
as:

  Separate read and write buffers
  Recovery from signals
  Query routines (methods)

A channel may be associated with a file (disc file, named FIFO, tty,
etc.), a connection to another process, a memory object, and others.
A single interface for reading and writing is presented by the <ch_>
package.
A powerful extension for Channels is the ability to push a stack of
filter functions (termed "converters") onto a Channel. Examples of
useful converters include encryption and compression filters. These
converters do not change the interface to Channels, hence their effect
is transparent to the application.

  Level 3
  =======

When a process uses communication channels, it must have some way of
responding to incoming data and events. This is called "channel
management". In Unix, there are two ways of doing this. One way is to
install signal handlers. Unfortunately, becauses most code is not
re-entrant, there are inherent dangers in using signals. The other
alternative is to using a polling loop to watch for communication
events. The <chm_> package provides the means (when running under a
plain Unix environment) to register event callback functions to
process communications events. The <chm_> package also provides the
chm_poll routine which does the actual polling for events. On Unix
systems, this can effectively place a process in hibernation until an
event occurs.
When running under a different environment, say using the Xt or XView
toolkits, the <chx_> and <notify_chm_> packages provide the
callback registration functions, respectively. These packages do not
provide a polling routine, as the toolkit choosen supplies this
function.

The <pio_> package provides a simple means to transfer binary data
between a unified "network" format and local (host) format. These
routines write to/ read from channels. These routines are handy for
simple communications.

  Level 4
  =======

The real power in the communications support in Karma comes from the
<conn_> package. This is the recommended interface for all user-level
communications (and most library communications too).
The <conn_> package is initialised with the <conn_register_managers>
routine. This tells the <conn_> package whether to use the plain Unix
channel management, or the Xt or XView based channel management.
The <conn_> package provides support for modules (processes) to
connect to each other, using agreed upon "protocols". The <conn_>
package does not enforce the definition of a "protocol", it just
provides a means for modules to differentiate between the various
connections they may have. It is the responsibility of the code
implementing the protocol to determine the format of data which may
pass down a connection of a particular protocol.

Each module (client) that wishes to make connections with a particular
protocol to another module (server) must register the callback
routines which will process events on connections of this protocol. If
the client does not register support for this protocol, it cannot make
connections with that protocol.

Each module (server) which wishes to receive connections of a
particular protocol from other modules (clients) must register the
callback routines which will process events on connections of this
protocol. If the server does not register support for this protocol,
it cannot accept connections with that protocol.

Each module may have both client and server support for zero, one or
more protocols. Also, each module may be operating both as a client
and a server for different connections. The difference between a
client and server (as far as the <conn_> package is concerned) is that
a client initiates a connection while the server accepts (or rejects)
the connection. See the section on security below which details how
access may be restricted.

In order to transfer data over a Connection, one must call the
<conn_get_channel> routine to obtain the underlying Channel, since all
I/O is based on the Channel.

The <conn_> package, in conjunction with the <ch_> package provides
complete abstraction of communications. Higher level library code and
user code has no knowledge of the underlying transport mechanisms
used. This allows the library to choose the most efficient transport
available.
It is worth pointing out that the communications abstraction does not
come at the cost of speed or efficiency, both in terms of raw
throughput and turnaround times.

The <arln_> package allows the reading (with blocking: waiting for
input) from the standard input of lines, while still polling for other
events. This package uses the <chm_> package for channel management,
hence is not applicable to Xt or XView toolkit based modules.

  Level 5
  =======

Another fundamental part of the Karma library is the Karma Data
Structure. This flexible, heirarchical data structure can be simply
transferred between modules by using the <dsxfr_> package. This
package defines the "multi_array" protocol, and provides simple
routines to read or write an entire data structure from/ to another
modules (or disc file).
Part of the definition of the "multi_array" protocol is that data
always flows from the client to the server, never the other way around.

Another package in the Karma library is the <kcmap_> package. This
package defines an opaque dynamic pseudocolourmap class. This provides
a graphics system independent means to modify colourmaps. In addition,
it provides for dynamic changes to colourmaps to be shared between
modules by using the <conn_> package to define the "colourmap_indices"
and "full_colourmap" protocols. This allows processes attached to the
same graphics display and separate graphics displays (ie. separate X
displays) to dynamically share colourmaps.

The <iedit_> package defines an opaque image edit instruction class.
This is designed for simple image painting operations. The <iedit_>
package also uses the <conn_> package to define the "2D_edit" protocol
which allows changes to an edit list to be propagated to co-operating
modules. Using this package, it is trivial to implement a "shared
paintboard" application for several users, where each user sees what
every other user is painting, in real time.

  Level 6
  =======

The Karma library provides a simplified interface to the general Karma
Data Structure in the <iarray_> package. This package defines the
"Intelligent Array" class.
As with the <dsxfr_> package, the <iarray_> package provides similar
routines to transfer data between modules and to/from disc files.


Tutorial:
---------

In a simple module, the following sequence should be followed if
communications are required:

1)  Register the channel management package applicable
2)  Register any client or server protocols
3)  Optionally allocate a server port
4)  Enter the polling loop

In the case where a module only wishes to transmit data, and is not
interested in input (or closure) events on connections, it can omit
the stage where the polling loop is entered.

Below are some examples:

  Example 1
  =========

This simple example is a module which generates some data (a 2
dimensional Intelligent Array) and then wishes to transmit this to a
module (name "receiver") on another machine (with name
"server.regional.edu.mars").


/*----------------------------------------------------------*/
/*  Communications sample program: multi_array transmitter  */
/*----------------------------------------------------------*/

#include <karma_iarray.h>
#include <karma_dsxfr.h>
#include <karma_conn.h>
#include <karma_chm.h>
#include <karma_r.h>

main ()
{
    /*  Declare variables  */
    int i, j;
    iarray a;

    /*  Initialise communications package  */
    conn_register_managers (chm_manage, chm_unmanage, ( void (*) () ) NULL);
    /*  Register "multi_array" client protocol support  */
    dsxfr_register_connection_limits (-1, 1);
    /*  Attempt connection to module  */
    if (conn_attempt_connection ("server.regional.edu.mars",
				 r_get_def_port ("receiver", NULL),
				 "multi_array")
	!= TRUE)
    {
	(void) fprintf (stderr, "Error connecting\n");
	exit (1);
    }
    /*  Create a 10x10 integer 2D iarray  */
    a = iarray_create_2D (10, 10, K_INT);

    /*  Fill  a  with data  */
    for (i = 0; i < 10; i++)
    {
	for (j = 0; j < 10; j++)
	{
	    I2 (a, i, j) = i * j;
	}
    }

    /*  Send  a  to module  */
    (void) iarray_write (a, "connections");

    /*  Deallocate array  */
    iarray_dealloc (a);
}

  Example 2
  =========

This simple example is a module which receives some data (an N-dimensional
Intelligent Array) and writes it to a disc file "out.kf".


/*----------------------------------------------------------*/
/*  Communications sample program: multi_array receiver     */
/*----------------------------------------------------------*/

#include <karma_iarray.h>
#include <karma_dsxfr.h>
#include <karma_conn.h>
#include <karma_chm.h>
#include <karma_r.h>

void read_func (/* first_time_data */);

main ()
{
    unsigned int port_number;

    /*  Initialise communications package  */
    conn_register_managers (chm_manage, chm_unmanage, ( void (*) () ) NULL);
    /*  Register "multi_array" server protocol support  */
    dsxfr_register_connection_limits (1, -1);
    /*  Register callback for new data  */
    dsxfr_register_read_func (read_func);
    /*  Allocate a port  */
    port_number = r_get_def_port ("receiver", NULL);
    if (conn_become_server (&port_number, 0) != TRUE)
    {
	(void) fprintf (stderr, "Could not become a server\n");
	exit (1);
    }

    /*  Enter polling loop (forever)  */
    while (TRUE) chm_poll (-1);
}

void read_func (first_time_data)
/*  This routine will be called whenever new data arrives on a "multi_array"
    connection.
    Note that by the time this routine is called, the data structure has
    already been read and cached in the library.
    If data appears on a connection for the first time, the value of
    first_time_data  will be TRUE. Any subsqeuent data that appears on a
    connection will not set this flag.
    The routine returns nothing.
*/
flag first_time_data;
{
    iarray a;

    /*  Read (non-blocking) from connection  */
    if ( ( a = iarray_read_nD ("connection", TRUE, NULL, 0, (char **) NULL,
			       NULL, K_CH_MMAP_NEVER) )
	== NULL )
    {
	(void) fprintf (stderr, "Error getting Intelligent Array\n");
	exit (1);
    }
    /*  Write "out.kf"  */
    (void) iarray_write (a, "out");
    /*  Exit module now that data has been read  */
    exit (0);
}

  Example 3
  =========

This simple example is a module which reads lines from the standard
input and transmits the lines to all modules connected with the
"experimental" protocol.
The module will attempt a connection to a module (name "receiver") on
another machine (with name "server.regional.edu.mars").


/*-----------------------------------------------------------*/
/*  Communications sample program: experimental transmitter  */
/*-----------------------------------------------------------*/

#include <karma_arln.h>
#include <karma_conn.h>
#include <karma_chm.h>
#include <karma_r.h>

main ()
{
    /*  Declare variables  */
    Connection conn;
    Channel channel;
    unsigned int num_conn;
    unsigned int count;
    char buffer[256];

    /*  Initialise communications package  */
    conn_register_managers (chm_manage, chm_unmanage, ( void (*) () ) NULL);
    /*  Register "experimental" client protocol support  */
    /*  No callbacks registered because client will never receive data  */
    conn_register_client_protocol ("experimental", 0, 1, ( flag (*) () ) NULL,
				   ( flag (*) () ) NULL, ( flag (*) () ) NULL,
				   ( void (*) () ) NULL);
    /*  Attempt connection to module  */
    if (conn_attempt_connection ("server.regional.edu.mars",
				 r_get_def_port ("receiver", NULL),
				 "experimental")
	!= TRUE)
    {
	(void) fprintf (stderr, "Error connecting\n");
	exit (1);
    }

    /*  Loop waiting for input and transmit each line  */
    while ( arln (buffer, 256, "Hurry up> ") )
    {
	/*  Have line of data: transmit to all "experimental" servers  */
	/*  Get number of connections made (should be 1 in this example)  */
	num_conn = conn_get_num_client_connections ("experimental");
	for (count = 0; count < num_conn; ++count)
	{
	    /*  Get a particular connection  */
	    if ( ( conn = conn_get_client_connection ("experimental", count) )
		== NULL )
	    {
		(void) fprintf (stderr,
			    "Error getting \"experimental\" connection: %u\n",
				count);
		exit (1);
	    }
	    /*  Get the channel  */
	    channel = conn_get_channel (conn);
	    /*  Write and flush  */
	    (void) ch_puts (channel, buffer, TRUE);
	    (void) ch_flush (channel);
	}
    }
}

  Example 4
  =========

This simple example is a module which reads lines from modules connected with
the "experimental" protocol and displays the lines.


/*--------------------------------------------------------*/
/*  Communications sample program: experimental receiver  */
/*--------------------------------------------------------*/

#include <karma_arln.h>
#include <karma_conn.h>
#include <karma_chm.h>
#include <karma_r.h>

flag read_func (/* connection, info */);

main ()
{
    /*  Initialise communications package  */
    conn_register_managers (chm_manage, chm_unmanage, ( void (*) () ) NULL);
    /*  Register "experimental" server protocol support  */
    conn_register_server_protocol ("experimental", 0, 1, ( flag (*) () ) NULL,
				   read_func, ( void (*) () ) NULL);
    /*  Allocate a port  */
    port_number = r_get_def_port ("receiver", NULL);
    if (conn_become_server (&port_number, 0) != TRUE)
    {
	(void) fprintf (stderr, "Could not become a server\n");
	exit (1);
    }

    /*  Wait forever for events  */
    while (TRUE) chm_poll (-1);
}

flag read_func (connection, info)
/*  This routine will read in data from the connection given by  connection
    and will write any appropriate information to the pointer pointed to by
    info  .
    The routine returns TRUE on successful reading,
    else it returns FALSE (indicating the connection should be closed).
    Note that the  close_func  will be called if this routine returns FALSE
*/
Connection connection;
void **info;
{
    Channel channel;
    char buffer[256];

    /*  Get channel  */
    channel = conn_get_channel (connection);
    if (ch_gets (channel, buffer, 256) != TRUE)
    {
	(void) fprintf (stderr, "Error reading line\n");
	return (FALSE);
    }
    (void) fprintf (stderr, "Incoming line: \"%s\"\n", buffer);
    return (TRUE);
}


Connection Management:
----------------------

When developing a complex application comprised of many communicating
modules, the establishment of connections (and even the starting of
modules) can become rather difficult to manage, with connection
attempts (calls to <conn_attempt_connection>) in many different
modules. To streamline this, there exists the Connection Management
shell, an interpreter which reads a script file. The script file
specifies which hosts modules will run on, which modules should be
run and the connections between all the modules.
This script allows the application designer to centralise the startup
of modules and connections: making the application somewhat
self-documenting.
Modules started locally have the same working directory as the script,
whereas modules started on other hosts have an undefined working
directory.
Modules started with the Connection Management Shell have their output
redirected to logfiles in  /tmp

See the file: $KARMABASE/cm_script/example  for information on the
syntax of the Connection Management Shell.


Security (authentication and encryption):
-----------------------------------------

  Authentication
  ==============
In a networked environment, any client module is able to make a
connection to a server module. On the Internet, this means every
machine with Internet access can potentially connect to your server
module. In many (most) cases, this is not a concern since your server
module will only be running a short time. However, there may be
circumstances which require a more prudent approach. The Karma
communications infrastructure provides multiple levels of
authentication verification to support your needs.

  Encryption
  ==========

Distinct from authentication is the issue of privacy. Since the Karma
communications infrastructure utilises the network transport
facilities provided by the operating system, the level of privacy is
the same as that provided by the network layer. On a local area
network (LAN), you can expect that every machine on that network is
capable of spying on your data as it passes over the network. For most
people, this is probably not a serious concern since their data files
are also transferred over this network. Of greater concern is the
possibility of information to be captured as it passes between two
machines, each on a different LAN. If you are transferring data
between two machines across the Internet, you really have no idea of
who might be spying on your data. Karma provides strong encryption to
assure your privacy.

  Purpose
  =======

The Karma communcations security support is aimed at:

  - Individual users who want to secure their communications

  - Application developers who want to write secure, network-aware
    applications without worrying about the implemtation details

  - Commercial service providers on the Internet who wish to ensure
    their own security and that of their customers.

  Details
  =======

All connections maintained by the <conn_> package may be both
authenticated and encrypted. The configuration of this is centralised
in your ~/.KARMAauthority file (this file should bar all access except
for the owner). You may set a uniform authentication/encryption
requirement for all Karma protocols, or you may selectively protect
some protocols. If a protocol is not specified in your authority file,
the default authentication/encryption is used.  The format of this
file is a follows (one line per protocol):

  protocol_name  security_type  [extra information]

The <protocol_name> may be any protocol name (a specific module need
not support a protocol). The name "RAW_PROTOCOL" designates the raw
Karma protocol used by the <conn_> package prior to negotiating a
specific application protocol.

<security_type> may be any of the following:

  key-only        IDEA used to verify authentication. Data unencrypted
  IDEA            IDEA used to verify authentication and data encrypted
  PGPuserID-IDEA  PGP used for session key transfer, then use IDEA mode
  drop-encryption drop data encryption for a specific protocol

For both the "key-only" and "IDEA" modes, the security type field must
be followed by a 24 byte IDEA session key and CFB initialisation
vector. This data must be in ASCII hexadecimal format. Both these
modes require the secure transfer of the authority file.

The "PGPuserID-IDEA" mode is the most secure: the server module uses
PGP (Philip R. Zimmermann's Pretty Good Privacy) to encrypt a random
IDEA session key which is sent the the client module. The client
module uses PGPdaemon (part of the PGPsendmail/Unix Suite available
from: ftp.atnf.csiro.au:pub/people/rgooch) to decode this session key.
The security type field must be followed by the authorised PGP userID.
You must have PGP and PGPsendmail/Unix 1.4 or later installed.

The "drop-encryption" mode allows you to drop encryption for a
specified protocol. This is useful when you have specified an
encryption mode for all protocols (using "RAW_PROTOCOL") and you are
concerned about the performance degradation of encrypting data.

Note that in all cases, encryption is automatically dropped after
protocol authentication if the connection is local (i.e. a connection
to the same machine).

  Restrictions
  ============

See the Karma security guide on information regarding restrictions.