Channels: the basic Input/Output Mechanism

   

Purpose

The basic Input/Output class in Karma is the ``channel''. This is provided by the ch package. A channel is a (possibly) full duplex buffered stream of data.

A channel may be associated with a file (disc file, named FIFO, tty, etc.), a connection to another process, a memory object, and others. A single interface for reading and writing is presented by the ch package.

Advantages over FILE * streams

Channels are similar to the standard C library FILE * streams, but fix a number of serious deficiencies in FILE * streams, such as:

• separate read and write buffers which give you proper full-duplex operation. FILE * streams require painful rewind and flush operations whenever you wish to change between reading and writing over bi-directional connections. Excessive flushing reduces the efficiency of FILE * streams to the point where you may as well not use them at all
• recovery from signals rather than failure. This means the application does not have to clean up after a signal is delivered which interrupted an I/O transfer. The library automatically restarts the remaining transfer

Besides these changes, a number of powerful enhancements are available with Channels:

• query routines (methods) to determine what type of channel subclass a particular channel object is
• simple syntax for making a full-duplex connection to a Unix or TCP/IP socket
• support for automatic decompression of disc-based channels
• the ability to push a stack of filter functions (termed ``converters'') onto a Channel. Examples of useful converters include encryption and compression filters. These converters do not change the interface to Channels, hence their effect is transparent to the application
• support for memory-mapping disc files. This means that the entire file contents are mapped into the process virtual memory space. This is especially useful if you wish to ``read'' a large data file into memory: with a suitable file format like the Karma format the data file can be mapped without any need to read the contents: the array data is directly available in the mapped region. Not having to read the file contents can save an enormous amount of time of you only need to access a small part of the data. Conventional techniques for accessing small parts of a file have the disadvantage of requiring the application explicitely read in the part of the data it requires: this makes the application more complex. Memory mapping places that burden on the operating system
• memory-based Channels which do not require any file storage. This is useful when it is easier to write some data into a ``temporary file'' and read it back later, but you would rather avoid the performance penalty of creating and manipulating a disc-based channel. Memory channels consume virtual memory space and are far more efficient than disc channels unless they take a significant proportion of physical memory (RAM), in which case their benefit is marginal
• various optimised convenience routines to drain, copy and pad to/from channels and between channels, read and write with byte swapping
• virtual channels such as data sinks (similar to /dev/null but more efficient)

Using Channels

The interface to channel objects is designed to be similar to the standard C library FILE * streams interface, however there a few subtle differences:

• the ch_read and ch_write functions take a buffer of bytes rather than a buffer of elements and similarly require only one size parameter rather than two. Some FILE * stream implementations may get confused when partial elements are written (due to interrupted system calls, EndOfFile or other failure), with no way to report to the application how many bytes were actually transferred. In addition, with Channel objects it makes no sense to split a single concept into two parameters
• there is no equivalent to the rewind function, as the ch_seek function provides the desired functionality

Tutorials

Below are a few simple examples which show how to use Channels.

Simple ASCII File Reader

This example shows a simple ASCII file reader for a 2-dimensional data format. The first line contains the horizontal and vertical lengths. The remaining lines contain the array data.

#include <stdio.h>
#include <errno.h>
#include <karma.h>
#include <karma_ch.h>
#include <karma_m.h>

int main (int argc, char **argv)
{
    Channel channel;
    int xlen, ylen, x, y;
    float *array;
    extern char *sys_errlist[];
    static char usage_string[] = "readfile infile";

    if (argc != 2)
    {
        fprintf (stderr, "Usage:\t%s\n", usage_string);
        exit (RV_BAD_PARAM);
    }
    /*  Open the input file (this is a simple 2D format)  */
    if ( ( channel = ch_open_file (argv[1], "r") ) == NULL )
    {
        fprintf (stderr, "Error opening file: \"%s\"\t%s\n",
                 argv[1], sys_errlist[errno]);
        exit (RV_CANNOT_OPEN);
    }
    /*  Read the array size  */
    ch_scanf (channel, "%d %d", &xlen, &ylen);
    /*  Create 2D float array  */
    if ( ( array = (float *) m_alloc (sizeof *array * xlen * ylen) ) == NULL )
    {
        fprintf (stderr, "Error allocating image\n");
        exit (RV_MEM_ERROR);
    }
    /*  Now read data values in from the input file and write to the array  */
    for (y = 0; y < naxis2; ++y) for (x = 0; x < naxis1; ++x)
    {
        ch_scanf (channel, "%e", array + x + y * xlen);
    }
    ch_close (channel);
    /*  Now do something with the array. Since the point of this example is to
        show how to use channels, let's not bother with being really useful  */
    return (RV_OK);
}   /*  End Function main  */

If you wanted to add support for automatic uncompression of the input file you would only need to change the call to ch_open_file to:

channel = ch_open_file (argv[1], "rz");

If the file has the extension .gz then the input file is automatically uncompressed. Note that you must have the <gzip> (GNU zip) programme in your PATH.

Loading a simple Memory-Mapped Binary File

This example shows how to ``load'' a simple 2-dimensional data file which is stored in binary format. The first two words contain the size of the array and the rest of the file contains the binary single precision floating point data. The first and last values in the array are multiplied by 2.0 and ``written'' back. Note that this file format is machine-dependant: if you were to read the same file on a big-endian and little-endian machines, the results would be quite different. So one would not use such a file format in real life. The point of this example is to show how to use memory-mapping to avoid slow reading.

#include <stdio.h>
#include <errno.h>
#include <karma.h>
#include <karma_ch.h>

int main (int argc, char **argv)
{
    Channel channel;
    char *ptr;
    int xlen, ylen;
    float *array;
    extern char *sys_errlist[];
    static char usage_string[] = "modfile infile";

    if (argc != 2)
    {
        fprintf (stderr, "Usage:\t%s\n", usage_string);
        exit (RV_BAD_PARAM);
    }
    /*  Map the input file (this is a simple 2D format)  */
    if ( ( channel = ch_map_disc (argv[1], K_CH_MAP_ALWAYS, TRUE, TRUE) )
         == NULL )
    {
        fprintf (stderr, "Error opening file: \"%s\"\t%s\n",
                 argv[1], sys_errlist[errno]);
        exit (RV_CANNOT_OPEN);
    }
    /*  Get the mapped file  */
    ptr = ch_get_mmap_addr (channel);
    /*  Get the array size  */
    xlen = *(int *) ptr;
    ptr += sizeof xlen;
    ylen = *(int *) ptr;
    ptr += sizeof ylen;
    array = (float *) ptr;
    /*  Now multiply the first and last elements in the array by 2.0  */
    array[0] *= 2.0;
    array[xlen - 1 + (ylen - 1) * xlen] *= 2.0;
    /*  Close the channel: this also causes the modified data to be written  */
    ch_close (channel);
    return (RV_OK);
}   /*  End Function main  */

Notice how the call to ch_map_disc specifies that the file should be mapped so the memory region is writable and changes made to the region are written back to disc. An application that did not want to modify the file would instead ask for a read-only mapping (an example application would be one that displays the first and last values in the array).