Introduction

SDS is a library of calls, written in C but  with  interfaces  to
Fortran  and  C++,  whose  primary job is to describe arbitrarily
structured data.  As well as describing existing  collections  of
data, the SDS interface provides ways of organising existing data
and of requesting allocated memory for data which only exists  as
description.

At the moment [and perhaps forever] SDS can describe data  struc-
tures  that it cannot help generate: in particular data with size
and type information embedded in them which are generated by spe-
cial purpose hardware readout systems.

                              Terms

An Sds is a managed collection of objects and their descriptions.
The  objects may exist purely as descriptions or as arrays of one
or more instantiations, and their storage locations may  be  dif-
ferent.  Datasets  exist in two forms: a 'prototyping' stage when
descriptions of objects are being added to the dataset: these ob-
jects  may or may not have memory allocated for them; and an 'as-
sembled' stage when all memory that should be allocated has been.
When  assembled,  the  dataset is "closed" and objects may not be
added or removed, but full access is given to manipulate the data
themselves;  in  the  proto state the data cannot be modified but
the of the dataset may be.

Here, the term refers to a object,  which may range from  'primi-
tive' types such as integer, float and  character to complex data
types such as C structures and structures with embedded size  and
type information. Examples of objects are:

      int i;

      int iarray[5];

      struct foo {
        int    a[20];
        char   adesc[32];
        double asize;
      } bar[20];

      struct Module {
        int   Type;
        int   DataSize;
        short Data[DataSize];
      } Module[NumberOfModules];

This last case shows an array of objects with embedded  size  in-
formation.  Using a simple naming convention described more fully
below, SDS can apply such descriptions to real data  and  present
the user with the size information and object addresses.

Data objects managed by a dataset may  be  stored  with  the  SDS
headers  or  may  be marked 'disjoint': that is, their storage is
not with the headers. In the latter case, SDS may be informed  of
the storage location by the programmer (having retrieved this in-
formation from, for instance, a database) or the knowledge may be
explicitly stored with the header so that loading or attaching to
the object may be done automatically.


                      Sds uses a number of
: that is, opaque values acting as pointers  into  system  tables
which describe each of the components; they are analagous to file
descriptors.

The allows access to type information containing :

- A type code, indicating if the described  data  is  'primitive'
(eg SDS_INT, SDS_DOUBLE), 'primitive' with special interpretation
(eg SDS_UNIX_TIME, which is in fact SDS_LONG but  can  be  inter-
preted in a special fashion), or compound - referring to a struc-
ture. The typeH is the type code for such a compound  data  type,
and accesses a hierarchy of type codes which finally resolve into
primitives.

- For each type code, a multiplicity.

- For each type code, a definition name.

Each object is associated with a typeH, and  subobjects  used  to
describe compuond objects may also need type handles.  Thus, when
Sds describes the C structure:

    struct Fred {
      int aint;
      float bfloat[20];
    } Albert[3];

the typeH will access the information:

      -  one integer, name aint
      -  twenty floats, name bfloat

Both of these names are referred  to  here  as  names.  The  name
'Fred'  is  also  a definition name, but in Sds it is replaced by
the typeH. By contrast, the name 'Albert' is an  name,  which  is
not  part  of  the  type  description  and is accordingly managed
through the (see below).

To formalise data structures which cannot be easily described  by
C,  in  particular  when size and type information is embedded in
the data themselves, naming conventions  are  used.  A  structure
described by Sds (in pseudo-code) as :

    struct Fred {
      float scale_factor;
      short data_count;
      char data[MULTIPLICITY_ONE];
      int checksum;
    } Albert[MULTIPLICITY_TWO];

can be scanned by Sds to pick up the multiplicity of  the  'data'
arrays in each element of the 'Albert' array from embedded infor-
mation. Here, the strategy is to look for an array called  'data'
with  undetermined  length  after  an  [integer,short,byte] field
called  'data_count'  is  encountered.  Sds  can  then  scan  the
described  data  and, if required, store pointer and multiplicity
arrays with the header for subsequent fast access. Typically, the
total storage size will determine the value of Albert's multipli-
city.

The (storeH) allows access to information detailing where an  ob-
ject or subobject is stored:

- The storage name, for example a  filename  or  the  name  of  a
shared memory partition.

- the name of the host which controls the storage

- an indicator of the storage type  -  SDS_FILE,  SDS_SHARED_MEM,
SDS_DATABASE etc.

The sdsH allows access to dataset information: number of objects,
name and timestamp of dataset and object information such as mul-
tiplicities, structures and addresses; the  sdsP  refers  to  da-
tasets  whose  contents  -  number  and  type of objects - may be
changed. Many queries and manipulations may be asked of proto-sds
datasets,  but they will not respond to queries about the placing
of objects they manage: this is to provide protection against ac-
quiring addresses that may change due to Sds internals or copying
of objects. Such protection will be given if  decent  programming
practice  is  followed  but  inappropriate  use of pointers whose
validity is not guaranteed will cause problems. In  general,  the
transformation  from  a proto dataset to an object-accessible da-
taset should be followed by calls re-establishing  the  addresses
of any objects used.

The recordH  and  objectH  allow  access  to  object  information
(alignment, size, name, multiplicity, instantiation names and in-
directly their associated typeH and storeH).

The objectI points to the position of an object within a dataset:
user  objects start at number 1 [object 0 is the dataset directo-
ry, and should not be directly accessed by the user].

In general, sds calls will return a  long integer; a return of zero indicates an
error or warning which may be investigated with  the  error  han-
dling routines detailed below. All handles are invalid if 0.

NOTE: In what follows, 'Level 1' calls  are  those  of  immediate
usefullness  for  basic use of SDS, whereas Level 2 are of secon-
dary interest but still intended for the user. Internal calls are
not described.


                int call_succeeded = sds_init();
Initialise the Sds system tables. Required. Has  effect  only  on
the first call.


int call_succeeded = sds_reinit_enable();

Where a system may require a second initialisation, for  instance
a    realtime    system    after    partial    failure,   calling
sds_reinit_enable() will allow a subsequent call to sds_init() to
take  effect. Note that only the first subsequent sds_init() will
have effect; after that the normal behaviour returns.

int call_succeeded = sds_fprint_def(FILE *stream, objectH);

Print the structural definition of an object (ie names, types and
structure levels) to the named output stream.

int call_succeeded = sds_fprint_object(FILE *stream, objectH);

Print the data contents of an object to the named output  stream.
An attempt is made to give reasonable format.

int version = sds_version();

    Returns a floating point version number of the form 2.3



                    sdsH = sds_open(storeH)

Gets access to an existing dataset. You may now  query  it  about
what  is there and where it is, but you do not yet have access to
the data....

sdsH = sds_attach(storeH, int access_mechanism);

Attach to a dataset; ie you are  getting  access  to  the  actual
stuff,  through  mechanisms  such as local shared memory, mapping
disk, or reflective memory links.

sdsP = sds_create(char *name)

Gets  a  new   dataset   prototype   ready   for   filling   with
records/objects.  The  name given is the dataset name, which must
be unique for the datasets currently accessed by a process.

objectI = sds_add_oh(sdsP, objectH);

Add an object to a dataset prototype; returns the object_index of
the  object  within  the  dataset (Note that an object may be re-
gistered in more than one dataset, in which  case  their  indices
are in general different.)

int call_succeeded = sds_delete_oh(sdsP, objectH);

Delete an object from a dataset prototype.

sdsH = sds_make_public(sdsP, storeH);

This call creates the Sds described by the input  prototype;  the
dataset  is  now  in  principio  accessible  by  other  processes
(although system permissions may regulate this).  The  call  thus
implies  that some copying will be done: at least the Sds headers
and any objects declared in memory must  be  copied  to  the  new
storage  area. Data objects marked 'disjoint' will not be copied.
The sdsH returned now allows access to  all  data  pointers,  but
further manipulation of the dataset structure is illegal.

sdsH = sds_make_private(sdsP);

This call is similar to sds_make_public() in that a transform  is
made  between  a  proto dataset whose structure may be changed to
one where data may be manipulated but structure remains constant.
In  this case however objects in process memory remain there, and
proto objects without allocated memory will have process  storage
created.

sdsP = sds_protoize(sdsH);

Moves a dataset to the prototype state so that the dataset  stuc-
ture  may  be  changed.  Attempts to obtain object addresses from
sdsP or sdsH are now invalid. It cannot be assumed  that  objects
whose addresses were found from the generating sdsH before proti-
sation (yuk) will remain in these locations after further dataset
manipulation.

sdsH = sds_duplicate(sdsH, storeH);

Duplicates the dataset sdsH to new storage.

sdsH = sds_deep_duplicate(sdsH, storeH);

Duplicates the dataset sdsH to new storage, including  all  'dis-
joint' objects.

int call_succeeded = sds_discard_all(sdsH);

Discard all system descriptive information about a dataset.

int call_succeeded = sds_destroy_all(sdsH);

Discard all system descriptive information about  a  dataset  and
the  objects  in it, having first free'd any Sds-system allocated
memory in the objects.


            typeH = sds_make_th(struct typelist *tl);


           storeH = sds_make_sh(struct storeage *st);


     objectH = sds_new_object(th,int number,char *names[]);
int call_succeeded =
   sds_oh_instantiations(objectH,         storeH[],          char
*instatiation_names[]);
int call_succeeded =
   sds_oh_struct_storage(objectH, storeH[], char *name = NULL);

oh = sds_copy(storeH, objectI, pointer = NULL);

Having open'd the dataset, you may now *copy* some or all of  the
objects  (including architecture conversion if necessary).  Space
will be allocated unless 'pointer' has a non-null value, in which
case data will be loaded starting at the pointer position: caveat
programmer.

objectH   =    sds_ohfromindex(sdsH,    objectI);    objectH    =
sds_ohfromname(sdsH, name);

Find the appropriate object handle from the index or name  of  an
object within a dataset.

objectH = sds_ohlikename(sdsH, part_name,int occurrence);

Find the occurence'th instance of  an  object  within  a  dataset
whose name contains 'part_name' as a substring.

void *ohptr = sds_ohptr(objectH);

Find the pointer to the data acessed through objectH.
Note: this call will give an error if the objectH  was  found  by
querying  a  proto-dataset: default behaviour is to print out the
error stack and stop.

char *ohname = sds_ohname(objectH);

Find the name of the data acessed through objectH.

int multiplicity = sds_ohmult(objectH);

Find the multiplicity of the data acessed through  objectH  -  ie
how many of each element - the array size.

int sizeof = sds_ohsizeof(objectH);

Find the size in bytes of one element of  an  object.   The  size
could    be    undetermined;    sds_error    is   then   set   to
SDS_SIZE_UNDETERMINED and the error level set to SDS_WARNING; the
size returned is 0.

int sizeof = sds_thsizeof(th);

Find the size in bytes of the  data  structure  accessed  by  the
type_handle.  The  size  could be undetermined; sds_error is then
set  to  SDS_SIZE_UNDETERMINED  and  the  error  level   set   to
SDS_WARNING; the size returned is 0.

int call_succeeded = sds_ohtstamp(int tstamp);

Sets the time stamp of an object.

int tstamp = sds_get_ohtstamp(objectH);

Returns the time stamp of an object.

int call_succeeded = sds_discard(objectH);

Discard all system descriptive information about an object.

int call_succeeded = sds_destroy(objectH);

Free  any  Sds-system  allocated  memory  in  the  object,   then
sds_discard() all system information.


        int descriptor_handle = sds_desc_handle(objectH);

Get a handle to refer to subsequent analysis calls on an object.

There are two main types of object analysis: linear and hierarch-
ical.   In  each case repeated calls to the analysis routines re-
turn information about the analysed objects. In linear  analysis,
a  sequence  of statements about the primitive elements that make
up the object is made - eg  10  floats,  23  int,  120  char,  10
floats,  23  int,  120 char......  This is effected by sequential
calls to

int level = sds_linear(descH, struct level_description **ld.sp);

In hierarchical elements, a series of index numbers  refering  to
an array of descriptor structures is returned from

int level = sds_hierarch(descH, struct level_description **ld);

each of which give: The start address of the field
Its name
Its type_code
The number of elements in the array (or 1).
Its size in bytes.

When no more fields remain for analysis, 0 is  returned  and  the
warning SDS_DONE_ANALYSIS is set.

int call_succeeded = sds_close_dh(descH);

Clean up all system stuff created for an analysis. Use  of  descH
before it is reallocated with sds_desc_handle() will cause an er-
ror; however  sds_close_dh()  is  called  automatically  when  an
analysis sequence is run completion. Re-closing an already closed
descH has no effect.

int   level   =    sds_find_level(descH,    field_name,    struct
level_description **ld);

Makes calls to sds_hierarch() until a match is made to the  field
named field_name.


             recordH = sds_begin_record(char *name);

sds_record_entry(recordH, handle, int  number,  void  *ptr,  char
*name);

sds_begin_sub_record(rh, name)

sds_end_sub_record(rh)

sds_end_and_declare(recordH, sdsH);

sds_destroy_record_def(recordH, int destroy_object);

sds_print_record_def(recordH);



             void sds_stop_if_error(char *comment);

Really crude. If there has been an Sds error, it prints  out  the
error stack and does exit(1); otherwise it is quiet.

int level = sds_push_error(int errcode, int errlevel, char  *err-
string);

Push a new error onto the stack. We assume that errstring is zero
terminated,  and  check to see if it is NULL. Returns the 'level'
of the error in the stack. Filename and line information is added
by the preprocessor from the defines __LINE__ and __FILE__.

  int error_code = sds_last_error();

Returns the last sds_error, which is 0 if no problems found.

  int error_code = sds_last_warning();

Returns the last sds_error at SDS_WARNING level, which is 0 if no
problems found.

  void sds_perror(char *comment);

Print out the top level comment, any filled parts  of  the  error
stack, and the last operating system error if there was one.



void sds_output_proginfo(int truefalse);

Default 0 == Do not print out line and file information.

void sds_exit_on_fatal(int truefalse);

Default 1 == An SDS_FATAL error causes exit(truefalse & 0xff);

void sds_output_errors(int level);

Default 0 == Do not output errors as they are registered; this is
when  you  want  to  wait  for  a  final catastrophe and then use
sds_perror(), or when you may be able to take corrective  action;
in  most  programs sds_output_errors(SDS_FATAL) will be appropri-
ate.

sds_clear_errors();

Start with a clean slate.

int error_code = sds_pop_error(int errlevel, char *errstring);

void printoutprog(int i);

Print out program info for the i'th entry on the stack

void printout(int i);

Print out the i'th entry on the stack

Currently, error levels are:

SDS_WARNING
SDS_ERROR
SDS_FATAL