While the Karma data format is a very fast and compact way of storing data, for some reason not everyone uses this format. Hence a means of reading in data in other formats and converting to Karma format is required. The foreign package provides routines which will read and write many different formats, including:
The foreign package also provides routines to write a Karma data structure to a file in a foreign data format.
Combined with the powerful ``Intelligent Array'' (iarray) package in the Karma library, easy access to foreign data is possible.
The documentation on ``Intelligent Arrays'' is highly recommended.
Some sample code extracts are shown below which demonstrate how to read data in foreign formats.
This routine will read a file of any supported format (including Karma format) and returns a 2-dimensional Intelligent Array.
iarray read_2d_array (CONST char *filename)
/* [PURPOSE] This routine will read a file containing a 2-dimensional
array and will return an Intelligent Array.
<filename> The name of the file to read.
[RETURNS] An Intelligent Array on success, else NULL.
*/
{
iarray arr;
multi_array *multi_desc;
if ( ( multi_desc = foreign_guess_and_read (filename, K_CH_MAP_LOCAL,
FALSE, NULL,
FA_GUESS_READ_END) ) == NULL )
{
fprintf (stderr, "Error reading file: \"%s\"\n", filename);
return (NULL);
}
if ( ( arr = iarray_get_from_multi_array (multi_desc, NULL, 2,
(char **) NULL, NULL) )
== NULL )
{
fprintf (stderr, "Error extracting 2-D Intelligent Array\n");
return (NULL);
}
return (arr);
}
This programme will read in an image of any supported format (including Karma format) and doubles all the data values. If the FITS-style ``DATAMIN'' and ``DATAMAX'' keywords are present, the value of these are doubled as well. The new image is then written out in Karma format. The way you would run this programme is as follows:
doubler <infile> <outfile>
/*----------------------------------------------------------*/
/* Image Processing sample programme */
/*----------------------------------------------------------*/
#include <stdio.h>
#include <karma.h>
#include <karma_foreign.h>
#include <karma_iarray.h>
int main (int argc, char **argv)
{
iarray in, out;
unsigned int xlen, ylen, x, y;
double value[2];
/* Read argv[1] for input file */
if ( !foreign_read_and_setup (argv[1], K_CH_MAP_LOCAL,
FALSE, NULL, TRUE, 2, K_FLOAT, TRUE,
&in, NULL, NULL, TRUE, NULL) ) exit (1);
/* Create output array */
if ( ( out = iarray_create_from_template (in, K_FLOAT, TRUE, TRUE, TRUE) )
== NULL ) exit (2);
/* Double data values */
xlen = iarray_dim_length (in, 1);
ylen = iarray_dim_length (in, 0);
for (y = 0; y < ylen; ++y) for (x = 0; x < xlen; ++x)
{
F2 (out, y, x) = F2 (in, y, x) * 2.0;
}
/* Update the FITS-style "DATAMIN" and "DATAMAX" keywords if they exist */
if ( iarray_get_named_value (in, "DATAMIN", NULL, value) )
{
iarray_put_double (out, "DATAMIN", value[0] * 2.0);
}
if ( iarray_get_named_value (in, "DATAMAX", NULL, value) )
{
iarray_put_double (out, "DATAMAX", value[0] * 2.0);
}
/* Write the result to the file in argv[2] */
iarray_write (out, argv[2]);
} /* End Function main */
This section shows you how you would implement you own custom data filter which would provide automatic support for new data formats. See the section on datafilters in the Karma User Manual for a description on how to configure a data filter. The data filters must write either Karma, FITS or PNM format. This section deals with writing Karma format (it is assumed that if you want to write FITS or PNM format you will use something else).
There are two ways you can write a data filter. The first way requires you to allocate a Karma data structure of the required size and fill in the array with data from the input file. You then write the entire Karma data structure to the standard output. This approach is the easiest to implement, but has the disadvantage that the data filter programme will require as much virtual memory as the input data file. The second approach is to write a ``streaming'' data filter, which writes the Karma descriptors first (descriptors are similar to headers), and then reads in blocks of input data and writes out blocks of data in Karma format. This approach is a little harder to implement, but it requires very little virtual memory (swap), irrespective of how big your datafile is.
Note that for all types of data filter, only the following values are permitted for blank data:
The use of IEEE not-a-number (NaN) values is not permitted.
This section shows you how to code a data filter which reads in all the input data into virtual memory and then writes it out again. Below is the source code. The filter supports an extremely simple file format which has two ASCII lines describing each axis followed by data values.
#include <stdio.h>
#include <errno.h>
#include <karma.h>
#include <karma_iarray.h>
#include <karma_dsrw.h>
#include <karma_ch.h>
int main (int argc, char **argv)
{
iarray a;
int naxis1, naxis2, x, y;
float value;
double crval1, crval2, crpix1, crpix2, cdelt1, cdelt2;
char ctype1[STRING_LENGTH], ctype2[STRING_LENGTH];
uaddr dim_lengths[2];
FILE *fp;
extern char *sys_errlist[];
static char usage_string[] = "filter1 infile";
if (argc != 2)
{
fprintf (stderr, "Usage:\t%s\n", usage_string);
exit (RV_BAD_PARAM);
}
/* Open the input file (this is a simple 2D format) */
if ( ( fp = fopen (argv[1], "r") ) == NULL )
{
fprintf (stderr, "Error opening file: \"%s\"\t%s\n",
argv[1], sys_errlist[errno]);
exit (RV_CANNOT_OPEN);
}
/* Read the horizontal axis information. Format is:
naxis1 crval1 crpix1 cdelt1 ctype1 */
fscanf (fp, "%d %le %le %le %s",
&naxis1, &crval1, &crpix1, &cdelt1, ctype1);
/* Read the vertical axis information. Format is:
naxis2 crval2 crpix2 cdelt2 ctype2 */
fscanf (fp, "%d %le %le %le %s",
&naxis2, &crval2, &crpix2, &cdelt2, ctype2);
/* Create 2D float Intelligent Array, preferably in SHM */
dim_lengths[0] = naxis2;
dim_lengths[1] = naxis1;
if ( ( a = iarray_create_shm (K_FLOAT, 2, NULL, dim_lengths, NULL, FALSE) )
== NULL )
{
fprintf (stderr, "Error allocating image\n");
exit (RV_MEM_ERROR);
}
/* Change dimension names */
iarray_set_dim_name (a, 0, ctype2, TRUE);
iarray_set_dim_name (a, 1, ctype1, TRUE);
/* Attach FITS-style keywords */
iarray_put_named_string (a, "CTYPE1", ctype1);
iarray_put_double (a, "CRVAL1", crval1);
iarray_put_double (a, "CRPIX1", crpix1);
iarray_put_double (a, "CDELT1", cdelt1);
iarray_put_named_string (a, "CTYPE2", ctype2);
iarray_put_double (a, "CRVAL2", crval2);
iarray_put_double (a, "CRPIX2", crpix2);
iarray_put_double (a, "CDELT2", cdelt2);
/* Now read data values in from the input file and write to the array */
for (y = 0; y < naxis2; ++y) for (x = 0; x < naxis1; ++x)
{
fscanf (fp, "%e", &value);
F2 (a, y, x) = value;
}
fclose (fp);
/* Create the special standard output channel */
ch_open_stdout ();
/* Write out the Karma array */
dsrw_write_multi (ch_stdout, a->multi_desc);
iarray_dealloc (a);
return (RV_OK);
} /* End Function main */
The programme opens up the input file, grabs the ``header'' information and constructs an Intelligent Array from this. The Intelligent Array will be placed into a shared memory segment if possible. The header information is attached to the array and then the input data is read in an element at a time and placed into the array. Finally the array is written to the standard output and the programme returns the status code RV_OK indicating that the conversion was successful. If the array was allocated in shared memory and the standard output is piped into a Karma programme, the transfer time to the programme is effectively zero, because no data has to be transferred. Both programmes will share the same copy of the data. When both programmes terminate the shared memory segment is deleted.
You may also need to refer to the chapter on Intelligent Arrays.
An example input file is given below. It contains a 3*4 array.
3 1.0 2.0 3.0 x-axis 4 5.0 6.0 7.0 y-axis 1 2 3 4 5 6 7 8 9 10 11 12
This section shows you how to code a data filter which reads the input in blocks and writes out each block as it goes. Below is the source code. The file format supported is the same as the one in the example above. This is the kind of filter you should write, because it has the best performance and lowest resource utilisation under all circumstances.
#include <stdio.h>
#include <errno.h>
#include <karma.h>
#include <karma_dsrw.h>
#include <karma_ds.h>
#include <karma_ch.h>
#include <karma_im.h>
STATIC_FUNCTION (flag read_func,
(void *buffer, uaddr num_values, void *info) );
int main (int argc, char **argv)
{
int naxis1, naxis2;
unsigned int data_type = K_FLOAT;
double crval1, crval2, crpix1, crpix2, cdelt1, cdelt2;
multi_array *multi_desc;
array_desc *arr_desc;
char ctype1[STRING_LENGTH], ctype2[STRING_LENGTH];
uaddr dim_lengths[2];
char *dim_names[2];
double dval[2];
FILE *fp;
extern char *sys_errlist[];
static char *bunit = "JY/BEAM";
static char usage_string[] = "filter1 infile";
if (argc != 2)
{
fprintf (stderr, "Usage:\t%s\n", usage_string);
exit (RV_BAD_PARAM);
}
/* A few initialisation so that we get nice history */
im_register_module_name ("filter1");
im_register_module_version_date ("1.0");
im_register_lib_version (KARMA_VERSION);
/* Open the input file (this is a simple 2D format) */
if ( ( fp = fopen (argv[1], "r") ) == NULL )
{
fprintf (stderr, "Error opening file: \"%s\"\t%s\n",
argv[1], sys_errlist[errno]);
exit (RV_CANNOT_OPEN);
}
/* Read the horizontal axis information. Format is:
naxis1 crval1 crpix1 cdelt1 ctype1 */
fscanf (fp, "%d %le %le %le %s",
&naxis1, &crval1, &crpix1, &cdelt1, ctype1);
/* Read the vertical axis information. Format is:
naxis2 crval2 crpix2 cdelt2 ctype2 */
fscanf (fp, "%d %le %le %le %s",
&naxis2, &crval2, &crpix2, &cdelt2, ctype2);
/* Create a low-level Karma data structure descriptor without allocating
space for the array data yet */
dim_lengths[0] = naxis2;
dim_lengths[1] = naxis1;
dim_names[0] = ctype2;
dim_names[1] = ctype1;
if ( ( arr_desc = ds_easy_alloc_array_desc (2, dim_lengths, NULL, NULL,
NULL,(CONST char **) dim_names,
1, &data_type,
(CONST char **) &bunit) )
== NULL )
{
fprintf (stderr, "Error allocating descriptors\n");
exit (RV_MEM_ERROR);
}
/* Create the special standard output channel */
ch_open_stdout ();
if ( ( multi_desc = dsrw_filter_setup (ch_stdout, arr_desc,
argv[1], "stdout") ) == NULL )
{
exit (RV_MEM_ERROR);
}
/* Attach FITS-style keywords */
dval[1] = 0.0;
ds_put_unique_named_string (multi_desc->headers[0], multi_desc->data,
"CTYPE1", ctype1, FALSE);
dval[0] = crval1;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CRVAL1", K_DOUBLE, dval, FALSE);
dval[0] = crpix1;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CRPIX1", K_DOUBLE, dval, FALSE);
dval[0] = cdelt1;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CDELT1", K_DOUBLE, dval, FALSE);
ds_put_unique_named_string (multi_desc->headers[0], multi_desc->data,
"CTYPE2", ctype2, FALSE);
dval[0] = crval2;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CRVAL2", K_DOUBLE, dval, FALSE);
dval[0] = crpix2;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CRPIX2", K_DOUBLE, dval, FALSE);
dval[0] = cdelt2;
ds_put_unique_named_value (multi_desc->headers[0], multi_desc->data,
"CDELT2", K_DOUBLE, dval, FALSE);
if ( !dsrw_filter_process (ch_stdout, multi_desc, read_func, fp) )
exit (RV_SYS_ERROR);
ds_dealloc_multi (multi_desc);
return (RV_OK);
} /* End Function main */
static flag read_func (void *buffer, uaddr num_values, void *info)
/* [SUMMARY] Read data and write to a buffer.
<buffer> The buffer to write to.
<num_values> The number of values to read.
<info> The arbitrary function information pointer.
[RETURNS] TRUE on success, else FALSE.
*/
{
float *ptr = (float *) buffer;
FILE *fp = (FILE *) info;
while (num_values-- > 0) fscanf (fp, "%e", ptr++);
return (TRUE);
} /* End Function read_func */
The programme opens up the input file, grabs the ``header''
information and constructs a low-level Karma data structure from
this. The
ds_easy_alloc_array_desc function is
used to create a two-dimensional array descriptor. A special utility
function for data filters dsrw_filter_setup is then
called to create the data structure. The data structure created is
specially constructed and cannot be used by the ordinary functions in
the dsrw package. The header information is then attached to
the structure. To actually perform the filtering process the
dsrw_filter_process function is called, which will
call your read_func function many times, each time reading some
data and then writing out the converted data in Karma format. Finally
the programme returns the status code RV_OK indicating that the
conversion was successful.
This filter will also attempt to use shared memory to transfer the data to the Karma programme, in which case the transfer time is effictively zero. If shared memory is not available, the filter falls back to a streaming mode which consumes very little virtual memory.
As you will note, the streaming data filter is slightly more complicated than the VM data filter, but it is much more efficient in its use of virtual memory. You should be aware that the programme which is receiving the data (i.e. your visualisation tool) will keep a copy of the data in virtual memory, so if you only implement a VM data filter, your virtual memory requirements will double, unless your system has shared memory support. Transferring shared memory arrays is by far the fastest mode of operation.