Adding Parsers

Each parser in Source-Navigator is capable of understanding source files written in a specific programming language. In the Source-Navigator suite, parsers are stand-alone executables which adhere to a consistent command line interface. This interface allows Source-Navigator to control the parser's behavior through command line switches that the parser is expected to observe.

The Source-Navigator Software Development Kit (SDK) provides a C-based application programming interface (API) which enables parsers to insert information into a project database.

All files listed in SDK-Related Files can be found in the .../share/sdk directory.

Parsers may be implemented in any programming language. Naturally, the task of writing a parser is significantly simpler when using compiler generation tools such as GNU flex. All of the examples provided with Source-Navigator are written using GNU flex.

A sample parser for the Elf language (an embedded SQL-like language) is provided with the SDK. To experiment with this parser, change to the .../share/sdk/parsers directory and type

make test 

This will compile the parser and then parse a number of Elf source files, placing items of interest into a project database.

The Parser Toolbox Library

A library, implemented above the API, simplifies the task of writing new parsers. This library, known as the parser toolbox, allows programmers to focus on the issues of parsing source files in their chosen language and then storing the relevant information in the database.

The toolbox provides a number of C functions that can help you write your parser with less effort. These functions can be grouped as follows:

• functions to maintain line and column counts in your source files. Whenever the parser encounters an interesting symbol in the source program, it must know where in the source file the symbol occurred.

  That is, the function sn_advance_line() will increment a line counter maintained internally by the toolbox library. This function would be called along with other actions that the parser might perform when it encounters a newline character in the source text.

• a function to determine the name of the source file currently being parsed.

• miscellaneous text processing functions for counting the number of lines and columns consumed by a given block of text and so on.

• an entry point, called sn_main, which manages all of the details of interfacing with Source-Navigator and parsing each source file. In most circumstances, it is sufficient for the parser to call sn_main and allow this function to call back to your actual parsing function when required.

  This introduces a number of important concepts best explained with an example:

  int sn_main(int argc, char *argv[], char *lang_string,
      FILE **infile_ref, int(*parse)(), void(*reset)());
  char group[] = "java";
  int main(int argc, char *argv[])
  {
    return sn_main(argc, argv, group, &yyin, yylex, reset);
  }

  When calling sn_main, it is necessary to pass:

  • the argc and argv variables as passed into your main() function. This allows the library to access the command line options given by Source-Navigator.

  • a pointer to a string identifying the language. In the example above, the string is called group.

  • a pointer to a FILE * stream variable. You must pass the address of this variable as the library will manipulate the stream when opening new source files. If you are using GNU flex/bison, then pass a pointer to the global variable yyin.

  • A pointer to a function which takes no arguments and returns an int. This is a pointer to your actual parsing function. If you are using GNU flex/bison, then pass yylex or yyparse.

  • A pointer to a function which takes no arguments and returns void. This function is expected to perform any actions prior to processing the next source file. Typically this function might look like:

    void reset()
    {
      sn_reset_line(); /* reset line count */
      sn_reset_column(); /* reset column count */
    }

A detailed description of the functions available in the parser toolbox library can be found in .../share/sdk/include/snptools.h.

Unless the parser has to do something out of the ordinary, it should be possible to create a new parser by following these steps:

1. #include snptools.h in your program (or within the verbatim section of your lex specification).

2. Define a main function which calls sn_main with the appropriate arguments and returns the result of sn_main to the host environment.

3. Utilize the parser toolbox routines to simplify your work within the parser. For example, sn_message may be called to display messages in a dialog box that is shown to the user during the parsing process.

4. When your parser recognizes important language constructs such as comments, function declarations, or function invocations, call the appropriate sn_insert function to insert this information into the project database.

5. Link your final program with the sntools library. A typical command line for linking a parser can be found in the Makefile given in the .../share/sdk/parsers/examples/elf directory.

Project database calls

The toolbox library provides a number of functions for inserting information into the database that the parser will encounter in the source text. Each of these functions return an int with two possible return values:

sn_insert_symbol

int sn_insert_symbol(int id_type, char *classname, 
  char *identifier, char *filename, int start_lineno, 
  int startCol, int endLine, int endCol, unsigned long attrib,
  char *returnType, char *argTypes, char *argNames, 
  char *comment, int highStartLine, int highStartCol, 
  int highEndLine, int highEndCol);

sn_insert_symbol inserts a symbol into the project database.

type determines the type of the symbol. Possible values are:

Examples

The following example inserts a class definition:

sn_insert_symbol(SN_CLASS_DEF, NULL, classname,
  sn_current_file(), sn_line(), sn_column(), sn_line(),
  sn_column() + strlen(classname), 0L, NULL, NULL,
  NULL, NULL, sn_line(), sn_column(), sn_line(),
  sn_column() + strlen(classname));

The following example inserts a method definition:

sn_insert_symbol(SN_MBR_FUNC_DEF, classname, methodname,
  sn_current_file(), sn_line(), sn_column(), sn_line(),
  sn_column() + strlen(methodname), 0L, NULL, NULL, 
  NULL, NULL, sn_line(), sn_column(), sn_line(), 
  sn_column() + strlen(methodname));

For a C function with the following prototype:

int transform(struct coord * p, int x, int y, unsigned att);

the following example inserts the definition into the database:

sn_insert_symbol(SN_FUNC_DEF, NULL, "transform",
  sn_current_file(), sn_line(), sn_column(), sn_line(),
  sn_column + len, 0L, "int", "struct coord *, int, int,
  unsigned", "p,x,y,att", NULL, NULL, sn_line(), sn_column(),
  sn_line(), sn_column() + len);

sn_insert_xref

sn_insert_xref inserts cross-referencing information.

One of the most useful aspects of Source-Navigator is its cross-referencing capabilities. For instance, it is possible to see which functions are used by other functions or which functions modify a particular global variable.

Where possible, a parser should attempt to collect information of this nature and insert it into the project database. In a language with a flat namespace such as C, this can be achieved by noting the name of the current function within the parser. If a function invocation is encountered in the source text, then the cross-referencing can be inferred based on the current function's name and the function being called.

Cross-referencing information is added using:

int sn_insert_xref(int type, int scope_type, int scope_level,
  char *classname, char *funcname, char *argtypes, 
  char *refclass, char *refsymbol, char *ref_arg_types, 
  char *filename, int lineno, int acc);

Examples

The following example inserts cross-referencing information for a function that is called from another function.

sn_insert_xref(SN_REF_TO_FUNCTION, SN_FUNC_DEF,
  SN_REF_SCOPE_GLOBAL, NULL, currentFunction, NULL, NULL,
  calledFunction, NULL, sn_current_file(), sn_line(),
  SN_REF_PASS);

The following example inserts cross-referencing information for a function that is called from a member function called insert which belongs to a C++ class called Stack.

sn_insert_xref(SN_REF_TO_MBR_VAR, SN_MBR_FUNC_DEF,
  SN_REF_SCOPE_GLOBAL, "Stack", "insert", NULL, "Stack", 
  "i", NULL, sn_current_file(), sn_line(), SN_REF_READ);

sn_insert_comment

sn_insert_comment inserts comments into the project database.

When comments are encountered in the source text, the parser should call sn_insert_comment to add these comments to the project database. In some Source-Navigator projects, the user will choose to not include comments, but the parser should call this function regardless. The library function will decide whether or not to actually store the information in the database.

Comments are added to the database using:

int sn_insert_comment(char *classname, char *funcname, 
  char *filename, char *comment, int beg_line, int beg_col);

Integration with Source-Navigator

A completed parser can be integrated into Source-Navigator by following these steps:

1. Copy the parser executable into the .../bin directory.

2. Edit sn_prop.cfg in the directory .../share/etc. The Tcl procedure sn_add_parser is used to add parsers to the configuration. The following example shows how to include support for Java:

   sn_add_parser java -suffix {*.java} \
     -brow_cmd $odd_path(bindir)/jbrowser \
     -high_cmd $odd_path(bindir)/jbrowser \
     -high_switch "-h"

When Source-Navigator is restarted, support for the new language becomes available. When creating a project, the Parsers tab in the Project Preferences dialog shows the new language and its associated filename extensions.

This is all that is required to add new language support to Source-Navigator. If a project is created which contains files with any of the specified filename extensions, the new parser is invoked to process those files as a part of the overall parsing process.