     [file docstyle.texi not included]

Semantic Manual
***************

Semantic is a suite of Emacs libraries and utilities for parsing source code.  At its core is a lexical analyzer and two parser generators (bovinator and wisent) written in Emacs Lisp. Semantic provides a variety of tools for making use of the parser output, including user commands for code navigation and completion, as well as enhancements for imenu, speedbar, whichfunc, eldoc, hippie-expand, and several other parts of Emacs.

To send bug reports, or participate in discussions about semantic, use the mailing list cedet-semantic@sourceforge.net via the URL: http://lists.sourceforge.net/lists/listinfo/cedet-semantic

This manual documents the Semantic library and utilities.

Copyright © 1999--2005, 2007, 2009--2016 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,'' and with the Back-Cover Texts as in (a) below.  A copy of the license is included in the section entitled ``GNU Free Documentation License.''

     (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and modify this GNU manual.''

Introduction
************

This chapter gives an overview of Semantic and its goals.

Ordinarily, Emacs uses regular expressions (and syntax tables) to analyze source code for purposes such as syntax highlighting.  This approach, though simple and efficient, has its limitations: roughly speaking, it only ``guesses'' the meaning of each piece of source code in the context of the programming language, instead of rigorously ``understanding'' it.

Semantic provides a new infrastructure to analyze source code using parsers instead of regular expressions.  It contains two built-in parser generators (an LL generator named Bovine and an LALR generator named Wisent, both written in Emacs Lisp), and parsers for several common programming languages.  It can also make use of external parsers---programs such as GNU Global and GNU IDUtils.

Semantic provides a uniform, language-independent API for accessing the parser output.  This output can be used by other Emacs Lisp programs to implement ``syntax-aware'' behavior.  Semantic itself includes several such utilities, including user-level Emacs commands for navigating, searching, and completing source code.

The following diagram illustrates the structure of the Semantic package:

Please Note:
     The words in all-capital are those that Semantic itself provides. Others are current or future languages or applications that are not distributed along with Semantic.

                                                                  Applications
                                                                      and
                                                                   Utilities
                                                                     -------
                                                                    /       \
                    +---------------+    +--------+    +--------+
              C --->| C      PARSER |--->|        |    |        |
                    +---------------+    |        |    |        |
                    +---------------+    | COMMON |    | COMMON |<--- SPEEDBAR
           Java --->| JAVA   PARSER |--->| PARSE  |    |        |
                    +---------------+    | TREE   |    | PARSE  |<--- SEMANTICDB
                    +---------------+    | FORMAT |    | API    |
         Scheme --->| SCHEME PARSER |--->|        |    |        |<--- ecb
                    +---------------+    |        |    |        |
                    +---------------+    |        |    |        |
        Texinfo --->| TEXI.  PARSER |--->|        |    |        |
                    +---------------+    |        |    |        |

                         ...                ...           ...         ...

                    +---------------+    |        |    |        |
        Lang. Y --->| Y      Parser |--->|        |    |        |<--- app. ?
                    +---------------+    |        |    |        |
                    +---------------+    |        |    |        |<--- app. ?
        Lang. Z --->| Z      Parser |--->|        |    |        |
                    +---------------+    +--------+    +--------+

Semantic Components
===================

In this section, we provide a more detailed description of the major components of Semantic, and how they interact with one another.

The first step in parsing a source code file is to break it up into its fundamental components.  This step is called lexical analysis:

             syntax table, keywords list, and options
                              |
                              |
                              v
         input file  ---->  Lexer   ----> token stream

The output of the lexical analyzer is a list of tokens that make up the file.  The next step is the actual parsing, shown below:

                         parser tables
                              |
                              v
         token stream --->  Parser  ----> parse tree

The end result, the parse tree, is Semantic's internal representation of the language grammar.  Semantic provides an API for Emacs Lisp programs to access the parse tree.

Parsing large files can take several seconds or more.  By default, Semantic automatically caches parse trees by saving them in your .emacs.d directory.  When you revisit a previously-parsed file, the parse tree is automatically reloaded from this cache, to save time.  See SemanticDB.

Using Semantic
**************

     [file sem-user.texi not included]

Semantic Internals
******************

This chapter provides an overview of the internals of Semantic. This information is usually not needed by application developers or grammar developers; it is useful mostly for the hackers who would like to learn more about how Semantic works.

Parser code
===========

Semantic parsing code is spread across a range of files.

semantic.el
     The core infrastructure sets up buffers for parsing, and has all the core parsing routines.  Most parsing routines are overloadable, so the actual implementation may be somewhere else.

semantic-edit.el
     Incremental reparse based on user edits.

semantic-grammar.el
semantic-grammar.wy
     Parser for the different grammar languages, and a major mode for editing grammars in Emacs.

semantic-lex.el
     Infrastructure for implementing lexical analyzers.  Provides macros for creating individual analyzers for specific features, and a way to combine them together.

semantic-lex-spp.el
     Infrastructure for a lexical symbolic preprocessor.  This was written to implement the C preprocessor, but could be used for other lexical preprocessors.

bovine/bovine-grammar.el
bovine/bovine-grammar-macros.el
bovine/semantic-bovine.el
     The ``bovine'' grammar.  This is the first grammar mode written for Semantic and is useful for simple creating simple parsers.

wisent/wisent.el
wisent/bison-wisent.el
wisent/semantic-wisent.el
wisent/semantic-debug-grammar.el
     A port of bison to Emacs.  This infrastructure lets you create LALR based parsers for Semantic.

semantic-ast.el
     Manage Abstract Syntax Trees for parsers.

semantic-debug.el
     Infrastructure for debugging grammars.

semantic-util.el
     Various utilities for manipulating tags, such as describing the tag under point, adding labels, and the all important semantic-something-to-tag-table.

Tag handling
============

A tag represents an individual item found in a buffer, such as a function or variable.  Tag handling is handled in several source files.

semantic-tag.el
     Basic tag creation, queries, cloning, binding, and unbinding.

semantic-tag-write.el
     Write a tag or tag list to a stream.  These routines are used by semanticdb-file.el when saving a list of tags.

semantic-tag-file.el
     Files associated with tags.  Goto-tag, file for include, and file for a prototype.

semantic-tag-ls.el
     Language dependent features of a tag, such as parent calculation, slot protection, and other states like abstract, virtual, static, and leaf.

semantic-dep.el
     Include file handling.  Contains the include path concepts, and routines for looking up file names in the include path.

semantic-format.el
     Convert a tag into a nicely formatted and colored string.  Use semantic-test-all-format-tag-functions to test different output options.

semantic-find.el
     Find tags matching different conditions in a tag table. These routines are used by semanticdb-find.el once the database has been converted into a simpler tag table.

semantic-sort.el
     Sorting lists of tags in different ways.  Includes sorting a plain list of tags forward or backward.  Includes binning tags based on attributes (bucketize), and tag adoption for multiple references to the same thing.

semantic-doc.el
     Capture documentation comments from near a tag.

Semanticdb Internals
====================

Semanticdb complexity is certainly an issue.  It is a rather hairy problem to try and solve.

semanticdb.el
     Defines a database and a table base class.  You can instantiate these classes, and use them, but they are not persistent.

     This file also provides support for semanticdb-minor-mode, which automatically associates files with tables in databases so that tags are saved while a buffer is not in memory.

     The database and tables both also provide applicable cache information, and cache flushing system.  The semanticdb search routines use caches to save data structures that are complex to calculate.

     Lastly, it provides the concept of project root.  It is a system by which a file can be associated with the root of a project, so if you have a tree of directories and source files, it can find the root, and allow a tag-search to span all available databases in that directory hierarchy.

semanticdb-file.el
     Provides a subclass of the basic table so that it can be saved to disk.  Implements all the code needed to unbind/rebind tags to a buffer and writing them to a file.

semanticdb-el.el
     Implements a special kind of system database that uses Emacs internals to perform queries.

semanticdb-ebrowse.el
     Implements a system database that uses Ebrowse to parse files into a table that can be queried for tag names.  Successful tag hits during a find causes Semantic to pick up and parse the reference files to get the full details.

semanticdb-find.el
     Infrastructure for searching groups Semantic databases, and dealing with the search results format.

semanticdb-ref.el
     Tracks crossreferences.   Cross references are needed when buffer is reparsed, and must alert other tables that any dependent caches may need to be flushed.  References are in the form of include files.

Analyzer Internals
==================

The Semantic analyzer is a complex engine which has been broken down across several modules.  When the Semantic analyzer fails, start with semantic-analyze-debug-assist, then dive into some of these files.

semantic-analyze.el
     The core analyzer for defining the current context.  The current context is an object that contains references to aspects of the local context including the current prefix, and a tag list defining what the prefix means.

semantic-analyze-complete.el
     Provides semantic-analyze-possible-completions.

semantic-analyze-debug.el
     The analyzer debugger.  Useful when attempting to get everything configured.

semantic-analyze-fcn.el
     Various support functions needed by the analyzer.

semantic-ctxt.el
     Local context parser.  Contains overloadable functions used to move around through different scopes, get local variables, and collect the current prefix used when doing completion.

semantic-scope.el
     Calculate scope for a location in a buffer.  The scope includes local variables, and tag lists in scope for various reasons, such as C++ using statements.

semanticdb-typecache.el
     The typecache is part of semanticdb, but is used primarily by the analyzer to look up datatypes and complex names.  The typecache is bound across source files and builds a master lookup table for data type names.

semantic-ia.el
     Interactive Analyzer functions.  Simple routines that do completion or lookups based on the results from the Analyzer.  These routines are meant as examples for application writers, but are quite useful as they are.

semantic-ia-sb.el
     Speedbar support for the analyzer, displaying context info, and completion lists.

Tools
=====

These files contain various tools a user can use.

semantic-idle.el
     Idle scheduler for Semantic.  Manages reparsing buffers after edits, and large work tasks in idle time.  Includes modes for showing summary help and pop-up completion.

senator.el
     The Semantic navigator.  Provides many ways to move through a buffer based on the active tag table.

semantic-decorate.el
     A minor mode for decorating tags based on details from the parser. Includes overlines for functions, or coloring class fields based on protection.

semantic-decorate-include.el
     A decoration mode for include files, which assists users in setting up parsing for their includes.

semantic-complete.el
     Advanced completion prompts for reading tag names in the minibuffer, or inline in a buffer.

semantic-imenu.el
     Imenu support for using Semantic tags in imenu.

semantic-mru-bookmark.el
     Automatic bookmarking based on tags.  Jump to locations you've been before based on tag name.

semantic-sb.el
     Support for Semantic tag usage in Speedbar.

semantic-util-modes.el
     A bunch of small minor-modes that exposes aspects of the semantic parser state.  Includes semantic-stickyfunc-mode.

document.el
document-vars.el
     Create an update comments for tags.

semantic-adebug.el
     Extensions of data-debug.el for Semantic.

semantic-chart.el
     Draw some charts from stats generated from parsing.

semantic-elp.el
     Profiler for helping to optimize the Semantic analyzer.

Tests
=====

semantic-utest.el
     Basic testing of parsing and incremental parsing for most supported languages.

semantic-ia-utest.el
     Test the semantic analyzer's ability to provide smart completions.

semantic-utest-c.el
     Tests for the C parser's lexical pre-processor.

semantic-regtest.el
     Regression tests from the older Semantic 1.x API.

Glossary
********

BNF
     In semantic 1.4, a BNF file represented ``Bovine Normal Form'', the grammar file used for the 1.4 parser generator.  This was a play on Backus-Naur Form which proved too confusing.

bovinate
     A verb representing what happens when a bovine parser parses a file.

bovine lambda
     In a bovine, or LL parser, the bovine lambda is a function to execute when a specific set of match rules has succeeded in matching text from the buffer.

bovine parser
     A parser using the bovine parser generator.  It is an LL parser suitable for small simple languages.

context

LALR

lexer
     A program which converts text into a stream of tokens by analyzing them lexically.  Lexers will commonly create strings, symbols, keywords and punctuation, and strip whitespaces and comments.

LL

nonterminal
     A nonterminal symbol or simply a nonterminal stands for a class of syntactically equivalent groupings.  A nonterminal symbol name is used in writing grammar rules.

overloadable
     Some functions are defined via define-overload. These can be overloaded via ....

parser
     A program that converts tokens to tags.

tag
     A tag is a representation of some entity in a language file, such as a function, variable, or include statement.  In semantic, the word tag is used the same way it is used for the etags or ctags tools.

     A tag is usually bound to a buffer region via overlay, or it just specifies character locations in a file.

token
     A single atomic item returned from a lexer.  It represents some set of characters found in a buffer.

token stream
     The output of the lexer as well as the input to the parser.

wisent parser
     A parser using the wisent parser generator.  It is a port of bison to Emacs Lisp.  It is an LALR parser suitable for complex languages.

GNU Free Documentation License
******************************

     [file doclicense.texi not included]