Karma is a general purpose programmer's toolkit and contains KarmaLib (the structured library and API) and a large number of modules (applications) to perform many standard tasks. This manual describes the library and gives programming examples.
This document is written for Karma version 1.6.26 , which is probably my ``experimental'' version. Most of this manual will still be relevant to the previously released binary-only (or ``beta'') version, since binary releases come every few weeks or so. Full public releases come once or twice a year, so this document may talk about several new things not available in the last public release of Karma.
Click here to get back to the Karma home page.
The library is useful in a wide range of applications, from process/instrumentation control to advanced visualisation. Some of the major features available are:
The philosophy of Karma was developed after an afternoon discussion with Patrick Jordan (then with the CSIRO Division of Radiophysics) in April 1992. His increasing fustrations with the networking support I had added to the SIP package (originally written by Donald McLean at CSIRO Radiophysics) led to my descision to abandon SIP and start afresh and develop a new package. The prime goal was to develop a philosophy which would make navigation of the library easy, with a structure that lent itself to expansion. Later that year while on holiday with some overseas relatives I had some time to sit down with pencil and paper (no computer access) and sketch out my prospective library. Upon my return (to my computer), I spent two fast and furious months coding. Before spring ended I had the core of my library implemented and could start using it.
The next major push forward came in the second quarter of the next year (once again while overseas). Until that time the library focussed mainly on interprocess communications and data structure support. There was only minimal graphics support in the library: each graphics application had to do a fair bit of work to manipulate the window system. Not surprisingly, there were not many graphics applications. In keeping with Patrick's general philosophy of programming (try to con me into doing the hard bits), he pushed for a graphics package which would understand the Karma data structures. Thus was born the graphics library.
A little more than a year later in mid-1994 (overseas again) came the next push: the move towards Xt for the user-interface toolkit. Until then, I was using the XView toolkit, and duplicating hundreds of lines code to interface XView and the graphics library. Over the course of the previous year Patrick had already convinced me that Xt was the way to go, and I agreed with him, except for the timing. It was always ``yeah, someday''. He had already contributed various widgets (a fileselector and a colourmap editor, plus other goodies) to a widget library I was maintaining for him (again in accordance with his general philosophy). After several fustrating and wasted days trying to port XView to another platform, in time for a demonstration, I spat the dummy and wrote my first Xt widget: a canvas widget. When I went back to give another round of demonstrations a few weeks later, I had ported my volume rendering tool to Xt and the platform in question.
Over the course of the next two years bigger and better widgets were developed. December 1994 saw the creation of the ImageDisplay widget, virtually an application in itself, the biggest widget written so far. In relative terms, 1995 was rather quiet, with most work being in improving existing library packages and enhancing the few visualisation applications.
In the second quarter of 1996 the next push came, except that here it was solidly in the applications area. A visiting astronomer (Vincent McIntyre, then of the University of Wollongong) asked me how hard it would be to develop an interactive ``Position-Velocity slicer'' tool. We decided to collaborate, and after a few days apiece, we had our new application. The popularity of this tool resulted in a flurry of requests from astronomers for other tools. The next six months saw the development of several new tools. The efforts of the previous years in developing the library were really starting to pay of.
Perhaps the most important aspect to the Karma design is that it should be easy to determine what package would suit your needs. This is important because a flexible library is layered, with more powerful packages built on top of simpler packages. Otherwise it is possible to quickly become lost amongst the various packages all appearing to provide the same basic functionality (i.e. communications support).
Another important point is that the lower-level (simpler) packages should still be available to the application programmer, as it is impossible for the library maintainer to forecast everybody's needs. The programmer has the ``right'' to delve into the guts of the library.
One of the key design goals of Karma is to ensure the library is simple to navigate. It should be clear to a programmer seeking a certain functionality from the library where to look. This is coupled with the realisation that functional concepts (such as communications support or data structure definitions) may be repeated in the library when complex functional systems are implemented by using more primitive systems. This is because most programming libraries are implemented with some sort of layering of functions. The experience with SIP was that it became increasingly difficult for a programmer not intimately familiar with the library to determine which function was more appropriate to use. Often, a programmer would choose a function that provided a more primitive facility rather than another which was more powerful.
When the programmer is faced with several functional systems which appear somewhat similar, it should be immediately obvious which system provides the greatest flexibility, power and ease of use. The Karma solution employs a strict layering with defined levels inside the structure.
With few exceptions, the highest level packages are recommended for use, as these provide the greatest flexibility, ease of use and are also more highly optimised. This last point may seem to be the reverse of common-sense, as one might expect a simple package to be more efficient than a complex package. In fact, because the higher level package has a better knowledge of the data being dealt with, there is more scope for making ``assumptions'' (perfectly safe because the package knows when it is safe to do so), caching intermediate results and avoiding duplication of effort.
To implement this design goal, the structure as set out below was developed.
An example of how this works is the core communications infrastructure, which spans four levels. These four levels are:
A programmer using the communications facilities of the Karma library would naturally choose to use the conn package, since it is the highest level package providing generic communications support.
The include file needed for each package is based on the common prefix. For example: karma_conn.h
The facilities available in the library are described in the following chapters. They are briefly described here:
This ranges from convenience wrappers around standard operating system facilities (such was the dir package which provides a portable and more powerful means of scanning directories), to runtime error detection for allocated memory blocks (with the m package). Other packages provide simplified signal handling (the s package) and default error display (the a package).
The Karma communications support includes process management and a powerful connection package. Using a Connection Management tool, the applications developer can launch and connect a number of modules (processes) on a network with ease. A simple scripting language is provided which allows development of network applications launched from the command line.
The communications support in Karma forms one of the major components to the library. Full authentication and encryption support is included, making the development of secure, network-aware applications trivial.
Packages which provide high-level functionality (such as Intelligent Arrays and various sections of the graphics library) are able to call upon the communications facilities. Because a universal interface is provided, packages need only register their specific protocols, leaving the initiation of connections to another agent (such as a connection management tool). Packages are notified when connections are attempted, succeed, have incoming data and when they close.
A generic callback mechanism to dispatch events and a portable mechanism to deliver regularly scheduled events at a variety of priorities. The latter has most application in realtime process and instrumentation control systems.
The display system both provides an abstract interface to the underlying graphics system (i.e. the X window system), and also provides much higher level functionality than many graphics libraries. As well as supporting simple geometric primitives and text display, a powerful and flexible image display system is included. This allows the direct mapping of application data structures (ie. 2-D and 3-D arrays) to display windows (canvases). These images may be animated at high speed (such as in a movie tool). The complex machinery required to handle window resize and refresh events, as well as other events (i.e. mouse events) is built into the display system. Other facilities such as graphics overlay lists (which are easily networked and shared amongst processes), image editing (a simple painting mechanism, also network shareable), contour images and axes display are also supplied.
The Karma widget library does not follow the strict layered approach of the rest of the library. This is not to say that the widget library is not layered, simply that the strict dependency rules are not enforced (though usually followed). More relevant to the application programmer is the lack of package names for widgets, and hence it is not quite so obvious which widget amongst a group of similar widgets is the most appropriate/most powerful. The chapter on the widget library provides a quick overview which should help you navigate.
Karma provides a highly extensible, recursive, heirarchical data structure with extensive library support. This data structure may be accessed in a very portable, generic fashion by applications and may also be accessed directly when speed is critical. By supplying powerful library routines, data structures may be accessed in an abstract way, removing the need for the applications programmer to write code to deal with complex data structures while also allowing users to pass increasingly complex data structures to modules without the need for recompiliation. The applications programmer is left to concentrate on processing simple sub-structures (typically 1, 2 or 3 dimensional arrays) without incurring any processing overhead. Tiling of multi dimensional arrays is supported transparently, and, using special indexing techniques (also transparent to the application), incur no performance overhead.
The library also supplies routines to allow the programmer to transfer data structures as objects to and from named objects (disc files and network connections) without requiring the programmer to pay any attention to data formats, byte swapping machines, word sizes, etc. All data transfers are highly optimised to take advantage of the features of any particular platform and operating system.
In addition, data structures may be automatically memory mapped from disc into the process address space, yielding enormous peformance increases when reading large data structures from disc, and also saving swap space.
This powerful data format supports not only the simple two and three-dimensional datasets used in the visualisation tools, but also the more complex data structures require to store and distribute other information such as graphics overlay lists.
As powerful as the Karma data structure may be, real data come in (too many) different formats. The library provides a package for reading foreign data formats such as FITS and Miriad n-dimensional arrays. Common image formats such as PPM and Sun Rasterfile are also supported.
The basic idea of ``Intelligent Arrays'' was conceived by Patrick Jordan of the CSIRO Division of Radiophysics. The following requirements were subsequently defined after discussions between Jordan and myself:
Jordan's first implementation of Intelligent Arrays was limited to two
dimensions and used the conventional indexing scheme used with
multi-dimensional dynamically allocated arrays in C:
where for a two-dimensional array m is the number of columns in the
array, j is the row (vertical) position and i is the column
(horizontal) position. This has the disadvantage of requiring a
multiplication for every array access, which incurrs a considerable
performance penalty on some systems.
The implementation of Intelligent Arrays I developed for Karma uses
address offset arrays, one for each dimension of the
array. Thus, for a two-dimensional array the following C code is used
for accessing data elements (although more elegant access macros are
provided for aesthetic reasons):
where x is an array of offsets for the horizontal dimension and y
is an array of offsets for the vertical dimension. The values in these
arrays may be calculated thus:
This scheme is much faster than the conventional indexing scheme and
has the remarkable feature that non-contiguous arrays may be
implemented by simply changing the index arrays. This is
possible when the array mapping to memory is separable for each
dimension. Examples of non-contiguous arrays where this is true are
tiled or toroidal arrays. In both these cases, conventional techniques
used to access array elements can slow down access several
times. Using address offset arrays there is no performance
penalty.
This section attempts to present a simple roadmap which you would use to start writing a new application. If you are new to Karma and want to know which parts of this manual you need to read (and more importantly, which parts you can ignore), this section should help. A variety of simple application types are described, and which parts of the library you are likely to use.
This tool can load an image (stored as a 2-dimensional array) and display it. The tool includes a file browser, a magnifying window, a colourmap editor, zoom controls and much more, all in less than 500 lines of code. The Image Display Tool chapter is devoted to this tool.
In order to display anything, you need to use the graphics library. The chapter on image display will describe the relevant parts of the graphics library. To bind your application together into a graphical user interface (GUI) you will need to use the widgets library, so the chapter on widgets will be helpful here. Finally, the chapter on Intelligent Arrays should be read, as Intelligent Arrays provide a simple interface to loading data.
This is much the same as the previously described tool, except that you also need to use other parts of the graphics library (see the chapter on image editing). A cursory knowledge of Karma communictions would also be helpful (see the chapter on interprocess communications).
This requires knowledge of Karma communications (see the chapter on interprocess communications) and event management (see the chapter on event generation and dispatch). You will also find some complete examples in the chapter on advanced examples.
A simple programme which reads in a floating-point image, multiplies all the values by 2.0 and writes a new file, could be written in a matter of minutes. To support a variety of input file formats, the chapter on foreign data explains how to read and write various data formats. The chapter on Intelligent Arrays describes how to manipulate array data. You will find a worked example in the chapter on Foreign Data. This example even shows you how you would manipulate a FITS-style header (Karma calls these ``attachments'').