XEmacs 21.4.19 (Constant Variable).

[chise/xemacs-chise.git.1] / info / internals.info-2

This is ../info/internals.info, produced by makeinfo version 4.8 from
internals/internals.texi.

INFO-DIR-SECTION XEmacs Editor
START-INFO-DIR-ENTRY
* Internals: (internals).       XEmacs Internals Manual.
END-INFO-DIR-ENTRY

   Copyright (C) 1992 - 1996 Ben Wing.  Copyright (C) 1996, 1997 Sun
Microsystems.  Copyright (C) 1994 - 1998, 2002, 2003 Free Software
Foundation.  Copyright (C) 1994, 1995 Board of Trustees, University of
Illinois.

   Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that this permission notice may be stated in a
translation approved by the Foundation.

   Permission is granted to copy and distribute modified versions of
this manual under the conditions for verbatim copying, provided also
that the section entitled "GNU General Public License" is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

   Permission is granted to copy and distribute translations of this
manual into another language, under the above conditions for modified
versions, except that the section entitled "GNU General Public License"
may be included in a translation approved by the Free Software
Foundation instead of in the original English.

\1f
File: internals.info,  Node: Catch and Throw,  Prev: Simple Special Forms,  Up: Evaluation; Stack Frames; Bindings

14.4 Catch and Throw
====================

     struct catchtag
     {
       Lisp_Object tag;
       Lisp_Object val;
       struct catchtag *next;
       struct gcpro *gcpro;
       jmp_buf jmp;
       struct backtrace *backlist;
       int lisp_eval_depth;
       int pdlcount;
     };

   `catch' is a Lisp function that places a catch around a body of
code.  A catch is a means of non-local exit from the code.  When a catch
is created, a tag is specified, and executing a `throw' to this tag
will exit from the body of code caught with this tag, and its value will
be the value given in the call to `throw'.  If there is no such call,
the code will be executed normally.

   Information pertaining to a catch is held in a `struct catchtag',
which is placed at the head of a linked list pointed to by `catchlist'.
`internal_catch()' is passed a C function to call (`Fprogn()' when
Lisp `catch' is called) and arguments to give it, and places a catch
around the function.  Each `struct catchtag' is held in the stack frame
of the `internal_catch()' instance that created the catch.

   `internal_catch()' is fairly straightforward.  It stores into the
`struct catchtag' the tag name and the current values of
`backtrace_list', `lisp_eval_depth', `gcprolist', and the offset into
the `specpdl' array, sets a jump point with `_setjmp()' (storing the
jump point into the `struct catchtag'), and calls the function.
Control will return to `internal_catch()' either when the function
exits normally or through a `_longjmp()' to this jump point.  In the
latter case, `throw' will store the value to be returned into the
`struct catchtag' before jumping.  When it's done, `internal_catch()'
removes the `struct catchtag' from the catchlist and returns the proper
value.

   `Fthrow()' goes up through the catchlist until it finds one with a
matching tag.  It then calls `unbind_catch()' to restore everything to
what it was when the appropriate catch was set, stores the return value
in the `struct catchtag', and jumps (with `_longjmp()') to its jump
point.

   `unbind_catch()' removes all catches from the catchlist until it
finds the correct one.  Some of the catches might have been placed for
error-trapping, and if so, the appropriate entries on the handlerlist
must be removed (see "errors").  `unbind_catch()' also restores the
values of `gcprolist', `backtrace_list', and `lisp_eval', and calls
`unbind_to()' to undo any specbindings created since the catch.

\1f
File: internals.info,  Node: Symbols and Variables,  Next: Buffers and Textual Representation,  Prev: Evaluation; Stack Frames; Bindings,  Up: Top

15 Symbols and Variables
************************

* Menu:

* Introduction to Symbols::
* Obarrays::
* Symbol Values::

\1f
File: internals.info,  Node: Introduction to Symbols,  Next: Obarrays,  Up: Symbols and Variables

15.1 Introduction to Symbols
============================

A symbol is basically just an object with four fields: a name (a
string), a value (some Lisp object), a function (some Lisp object), and
a property list (usually a list of alternating keyword/value pairs).
What makes symbols special is that there is usually only one symbol with
a given name, and the symbol is referred to by name.  This makes a
symbol a convenient way of calling up data by name, i.e. of implementing
variables. (The variable's value is stored in the "value slot".)
Similarly, functions are referenced by name, and the definition of the
function is stored in a symbol's "function slot".  This means that
there can be a distinct function and variable with the same name.  The
property list is used as a more general mechanism of associating
additional values with particular names, and once again the namespace is
independent of the function and variable namespaces.

\1f
File: internals.info,  Node: Obarrays,  Next: Symbol Values,  Prev: Introduction to Symbols,  Up: Symbols and Variables

15.2 Obarrays
=============

The identity of symbols with their names is accomplished through a
structure called an obarray, which is just a poorly-implemented hash
table mapping from strings to symbols whose name is that string. (I say
"poorly implemented" because an obarray appears in Lisp as a vector
with some hidden fields rather than as its own opaque type.  This is an
Emacs Lisp artifact that should be fixed.)

   Obarrays are implemented as a vector of some fixed size (which should
be a prime for best results), where each "bucket" of the vector
contains one or more symbols, threaded through a hidden `next' field in
the symbol.  Lookup of a symbol in an obarray, and adding a symbol to
an obarray, is accomplished through standard hash-table techniques.

   The standard Lisp function for working with symbols and obarrays is
`intern'.  This looks up a symbol in an obarray given its name; if it's
not found, a new symbol is automatically created with the specified
name, added to the obarray, and returned.  This is what happens when the
Lisp reader encounters a symbol (or more precisely, encounters the name
of a symbol) in some text that it is reading.  There is a standard
obarray called `obarray' that is used for this purpose, although the
Lisp programmer is free to create his own obarrays and `intern' symbols
in them.

   Note that, once a symbol is in an obarray, it stays there until
something is done about it, and the standard obarray `obarray' always
stays around, so once you use any particular variable name, a
corresponding symbol will stay around in `obarray' until you exit
XEmacs.

   Note that `obarray' itself is a variable, and as such there is a
symbol in `obarray' whose name is `"obarray"' and which contains
`obarray' as its value.

   Note also that this call to `intern' occurs only when in the Lisp
reader, not when the code is executed (at which point the symbol is
already around, stored as such in the definition of the function).

   You can create your own obarray using `make-vector' (this is
horrible but is an artifact) and intern symbols into that obarray.
Doing that will result in two or more symbols with the same name.
However, at most one of these symbols is in the standard `obarray': You
cannot have two symbols of the same name in any particular obarray.
Note that you cannot add a symbol to an obarray in any fashion other
than using `intern': i.e. you can't take an existing symbol and put it
in an existing obarray.  Nor can you change the name of an existing
symbol. (Since obarrays are vectors, you can violate the consistency of
things by storing directly into the vector, but let's ignore that
possibility.)

   Usually symbols are created by `intern', but if you really want, you
can explicitly create a symbol using `make-symbol', giving it some
name.  The resulting symbol is not in any obarray (i.e. it is
"uninterned"), and you can't add it to any obarray.  Therefore its
primary purpose is as a symbol to use in macros to avoid namespace
pollution.  It can also be used as a carrier of information, but cons
cells could probably be used just as well.

   You can also use `intern-soft' to look up a symbol but not create a
new one, and `unintern' to remove a symbol from an obarray.  This
returns the removed symbol. (Remember: You can't put the symbol back
into any obarray.) Finally, `mapatoms' maps over all of the symbols in
an obarray.

\1f
File: internals.info,  Node: Symbol Values,  Prev: Obarrays,  Up: Symbols and Variables

15.3 Symbol Values
==================

The value field of a symbol normally contains a Lisp object.  However,
a symbol can be "unbound", meaning that it logically has no value.
This is internally indicated by storing a special Lisp object, called
"the unbound marker" and stored in the global variable `Qunbound'.  The
unbound marker is of a special Lisp object type called
"symbol-value-magic".  It is impossible for the Lisp programmer to
directly create or access any object of this type.

   *You must not let any "symbol-value-magic" object escape to the Lisp
level.*  Printing any of these objects will cause the message `INTERNAL
EMACS BUG' to appear as part of the print representation.  (You may see
this normally when you call `debug_print()' from the debugger on a Lisp
object.) If you let one of these objects escape to the Lisp level, you
will violate a number of assumptions contained in the C code and make
the unbound marker not function right.

   When a symbol is created, its value field (and function field) are
set to `Qunbound'.  The Lisp programmer can restore these conditions
later using `makunbound' or `fmakunbound', and can query to see whether
the value of function fields are "bound" (i.e. have a value other than
`Qunbound') using `boundp' and `fboundp'.  The fields are set to a
normal Lisp object using `set' (or `setq') and `fset'.

   Other symbol-value-magic objects are used as special markers to
indicate variables that have non-normal properties.  This includes any
variables that are tied into C variables (setting the variable magically
sets some global variable in the C code, and likewise for retrieving the
variable's value), variables that magically tie into slots in the
current buffer, variables that are buffer-local, etc.  The
symbol-value-magic object is stored in the value cell in place of a
normal object, and the code to retrieve a symbol's value (i.e.
`symbol-value') knows how to do special things with them.  This means
that you should not just fetch the value cell directly if you want a
symbol's value.

   The exact workings of this are rather complex and involved and are
well-documented in comments in `buffer.c', `symbols.c', and `lisp.h'.

\1f
File: internals.info,  Node: Buffers and Textual Representation,  Next: MULE Character Sets and Encodings,  Prev: Symbols and Variables,  Up: Top

16 Buffers and Textual Representation
*************************************

* Menu:

* Introduction to Buffers::     A buffer holds a block of text such as a file.
* The Text in a Buffer::        Representation of the text in a buffer.
* Buffer Lists::                Keeping track of all buffers.
* Markers and Extents::         Tagging locations within a buffer.
* Bufbytes and Emchars::        Representation of individual characters.
* The Buffer Object::           The Lisp object corresponding to a buffer.

\1f
File: internals.info,  Node: Introduction to Buffers,  Next: The Text in a Buffer,  Up: Buffers and Textual Representation

16.1 Introduction to Buffers
============================

A buffer is logically just a Lisp object that holds some text.  In
this, it is like a string, but a buffer is optimized for frequent
insertion and deletion, while a string is not.  Furthermore:

  1. Buffers are "permanent" objects, i.e. once you create them, they
     remain around, and need to be explicitly deleted before they go
     away.

  2. Each buffer has a unique name, which is a string.  Buffers are
     normally referred to by name.  In this respect, they are like
     symbols.

  3. Buffers have a default insertion position, called "point".
     Inserting text (unless you explicitly give a position) goes at
     point, and moves point forward past the text.  This is what is
     going on when you type text into Emacs.

  4. Buffers have lots of extra properties associated with them.

  5. Buffers can be "displayed".  What this means is that there exist a
     number of "windows", which are objects that correspond to some
     visible section of your display, and each window has an associated
     buffer, and the current contents of the buffer are shown in that
     section of the display.  The redisplay mechanism (which takes care
     of doing this) knows how to look at the text of a buffer and come
     up with some reasonable way of displaying this.  Many of the
     properties of a buffer control how the buffer's text is displayed.

  6. One buffer is distinguished and called the "current buffer".  It is
     stored in the variable `current_buffer'.  Buffer operations operate
     on this buffer by default.  When you are typing text into a
     buffer, the buffer you are typing into is always `current_buffer'.
     Switching to a different window changes the current buffer.  Note
     that Lisp code can temporarily change the current buffer using
     `set-buffer' (often enclosed in a `save-excursion' so that the
     former current buffer gets restored when the code is finished).
     However, calling `set-buffer' will NOT cause a permanent change in
     the current buffer.  The reason for this is that the top-level
     event loop sets `current_buffer' to the buffer of the selected
     window, each time it finishes executing a user command.

   Make sure you understand the distinction between "current buffer"
and "buffer of the selected window", and the distinction between
"point" of the current buffer and "window-point" of the selected
window. (This latter distinction is explained in detail in the section
on windows.)

\1f
File: internals.info,  Node: The Text in a Buffer,  Next: Buffer Lists,  Prev: Introduction to Buffers,  Up: Buffers and Textual Representation

16.2 The Text in a Buffer
=========================

The text in a buffer consists of a sequence of zero or more characters.
A "character" is an integer that logically represents a letter,
number, space, or other unit of text.  Most of the characters that you
will typically encounter belong to the ASCII set of characters, but
there are also characters for various sorts of accented letters,
special symbols, Chinese and Japanese ideograms (i.e. Kanji, Katakana,
etc.), Cyrillic and Greek letters, etc.  The actual number of possible
characters is quite large.

   For now, we can view a character as some non-negative integer that
has some shape that defines how it typically appears (e.g. as an
uppercase A). (The exact way in which a character appears depends on the
font used to display the character.) The internal type of characters in
the C code is an `Emchar'; this is just an `int', but using a symbolic
type makes the code clearer.

   Between every character in a buffer is a "buffer position" or
"character position".  We can speak of the character before or after a
particular buffer position, and when you insert a character at a
particular position, all characters after that position end up at new
positions.  When we speak of the character "at" a position, we really
mean the character after the position.  (This schizophrenia between a
buffer position being "between" a character and "on" a character is
rampant in Emacs.)

   Buffer positions are numbered starting at 1.  This means that
position 1 is before the first character, and position 0 is not valid.
If there are N characters in a buffer, then buffer position N+1 is
after the last one, and position N+2 is not valid.

   The internal makeup of the Emchar integer varies depending on whether
we have compiled with MULE support.  If not, the Emchar integer is an
8-bit integer with possible values from 0 - 255.  0 - 127 are the
standard ASCII characters, while 128 - 255 are the characters from the
ISO-8859-1 character set.  If we have compiled with MULE support, an
Emchar is a 19-bit integer, with the various bits having meanings
according to a complex scheme that will be detailed later.  The
characters numbered 0 - 255 still have the same meanings as for the
non-MULE case, though.

   Internally, the text in a buffer is represented in a fairly simple
fashion: as a contiguous array of bytes, with a "gap" of some size in
the middle.  Although the gap is of some substantial size in bytes,
there is no text contained within it: From the perspective of the text
in the buffer, it does not exist.  The gap logically sits at some buffer
position, between two characters (or possibly at the beginning or end of
the buffer).  Insertion of text in a buffer at a particular position is
always accomplished by first moving the gap to that position (i.e.
through some block moving of text), then writing the text into the
beginning of the gap, thereby shrinking the gap.  If the gap shrinks
down to nothing, a new gap is created. (What actually happens is that a
new gap is "created" at the end of the buffer's text, which requires
nothing more than changing a couple of indices; then the gap is "moved"
to the position where the insertion needs to take place by moving up in
memory all the text after that position.)  Similarly, deletion occurs
by moving the gap to the place where the text is to be deleted, and
then simply expanding the gap to include the deleted text.
("Expanding" and "shrinking" the gap as just described means just that
the internal indices that keep track of where the gap is located are
changed.)

   Note that the total amount of memory allocated for a buffer text
never decreases while the buffer is live.  Therefore, if you load up a
20-megabyte file and then delete all but one character, there will be a
20-megabyte gap, which won't get any smaller (except by inserting
characters back again).  Once the buffer is killed, the memory allocated
for the buffer text will be freed, but it will still be sitting on the
heap, taking up virtual memory, and will not be released back to the
operating system. (However, if you have compiled XEmacs with rel-alloc,
the situation is different.  In this case, the space _will_ be released
back to the operating system.  However, this tends to result in a
noticeable speed penalty.)

   Astute readers may notice that the text in a buffer is represented as
an array of _bytes_, while (at least in the MULE case) an Emchar is a
19-bit integer, which clearly cannot fit in a byte.  This means (of
course) that the text in a buffer uses a different representation from
an Emchar: specifically, the 19-bit Emchar becomes a series of one to
four bytes.  The conversion between these two representations is complex
and will be described later.

   In the non-MULE case, everything is very simple: An Emchar is an
8-bit value, which fits neatly into one byte.

   If we are given a buffer position and want to retrieve the character
at that position, we need to follow these steps:

  1. Pretend there's no gap, and convert the buffer position into a
     "byte index" that indexes to the appropriate byte in the buffer's
     stream of textual bytes.  By convention, byte indices begin at 1,
     just like buffer positions.  In the non-MULE case, byte indices
     and buffer positions are identical, since one character equals one
     byte.

  2. Convert the byte index into a "memory index", which takes the gap
     into account.  The memory index is a direct index into the block of
     memory that stores the text of a buffer.  This basically just
     involves checking to see if the byte index is past the gap, and if
     so, adding the size of the gap to it.  By convention, memory
     indices begin at 1, just like buffer positions and byte indices,
     and when referring to the position that is "at" the gap, we always
     use the memory position at the _beginning_, not at the end, of the
     gap.

  3. Fetch the appropriate bytes at the determined memory position.

  4. Convert these bytes into an Emchar.

   In the non-Mule case, (3) and (4) boil down to a simple one-byte
memory access.

   Note that we have defined three types of positions in a buffer:

  1. "buffer positions" or "character positions", typedef `Bufpos'

  2. "byte indices", typedef `Bytind'

  3. "memory indices", typedef `Memind'

   All three typedefs are just `int's, but defining them this way makes
things a lot clearer.

   Most code works with buffer positions.  In particular, all Lisp code
that refers to text in a buffer uses buffer positions.  Lisp code does
not know that byte indices or memory indices exist.

   Finally, we have a typedef for the bytes in a buffer.  This is a
`Bufbyte', which is an unsigned char.  Referring to them as Bufbytes
underscores the fact that we are working with a string of bytes in the
internal Emacs buffer representation rather than in one of a number of
possible alternative representations (e.g. EUC-encoded text, etc.).

\1f
File: internals.info,  Node: Buffer Lists,  Next: Markers and Extents,  Prev: The Text in a Buffer,  Up: Buffers and Textual Representation

16.3 Buffer Lists
=================

Recall earlier that buffers are "permanent" objects, i.e.  that they
remain around until explicitly deleted.  This entails that there is a
list of all the buffers in existence.  This list is actually an
assoc-list (mapping from the buffer's name to the buffer) and is stored
in the global variable `Vbuffer_alist'.

   The order of the buffers in the list is important: the buffers are
ordered approximately from most-recently-used to least-recently-used.
Switching to a buffer using `switch-to-buffer', `pop-to-buffer', etc.
and switching windows using `other-window', etc.  usually brings the
new current buffer to the front of the list.  `switch-to-buffer',
`other-buffer', etc. look at the beginning of the list to find an
alternative buffer to suggest.  You can also explicitly move a buffer
to the end of the list using `bury-buffer'.

   In addition to the global ordering in `Vbuffer_alist', each frame
has its own ordering of the list.  These lists always contain the same
elements as in `Vbuffer_alist' although possibly in a different order.
`buffer-list' normally returns the list for the selected frame.  This
allows you to work in separate frames without things interfering with
each other.

   The standard way to look up a buffer given a name is `get-buffer',
and the standard way to create a new buffer is `get-buffer-create',
which looks up a buffer with a given name, creating a new one if
necessary.  These operations correspond exactly with the symbol
operations `intern-soft' and `intern', respectively.  You can also
force a new buffer to be created using `generate-new-buffer', which
takes a name and (if necessary) makes a unique name from this by
appending a number, and then creates the buffer.  This is basically
like the symbol operation `gensym'.

\1f
File: internals.info,  Node: Markers and Extents,  Next: Bufbytes and Emchars,  Prev: Buffer Lists,  Up: Buffers and Textual Representation

16.4 Markers and Extents
========================

Among the things associated with a buffer are things that are logically
attached to certain buffer positions.  This can be used to keep track
of a buffer position when text is inserted and deleted, so that it
remains at the same spot relative to the text around it; to assign
properties to particular sections of text; etc.  There are two such
objects that are useful in this regard: they are "markers" and
"extents".

   A "marker" is simply a flag placed at a particular buffer position,
which is moved around as text is inserted and deleted.  Markers are
used for all sorts of purposes, such as the `mark' that is the other
end of textual regions to be cut, copied, etc.

   An "extent" is similar to two markers plus some associated
properties, and is used to keep track of regions in a buffer as text is
inserted and deleted, and to add properties (e.g. fonts) to particular
regions of text.  The external interface of extents is explained
elsewhere.

   The important thing here is that markers and extents simply contain
buffer positions in them as integers, and every time text is inserted or
deleted, these positions must be updated.  In order to minimize the
amount of shuffling that needs to be done, the positions in markers and
extents (there's one per marker, two per extent) are stored in Meminds.
This means that they only need to be moved when the text is physically
moved in memory; since the gap structure tries to minimize this, it also
minimizes the number of marker and extent indices that need to be
adjusted.  Look in `insdel.c' for the details of how this works.

   One other important distinction is that markers are "temporary"
while extents are "permanent".  This means that markers disappear as
soon as there are no more pointers to them, and correspondingly, there
is no way to determine what markers are in a buffer if you are just
given the buffer.  Extents remain in a buffer until they are detached
(which could happen as a result of text being deleted) or the buffer is
deleted, and primitives do exist to enumerate the extents in a buffer.

\1f
File: internals.info,  Node: Bufbytes and Emchars,  Next: The Buffer Object,  Prev: Markers and Extents,  Up: Buffers and Textual Representation

16.5 Bufbytes and Emchars
=========================

Not yet documented.

\1f
File: internals.info,  Node: The Buffer Object,  Prev: Bufbytes and Emchars,  Up: Buffers and Textual Representation

16.6 The Buffer Object
======================

Buffers contain fields not directly accessible by the Lisp programmer.
We describe them here, naming them by the names used in the C code.
Many are accessible indirectly in Lisp programs via Lisp primitives.

`name'
     The buffer name is a string that names the buffer.  It is
     guaranteed to be unique.  *Note Buffer Names: (lispref)Buffer
     Names.

`save_modified'
     This field contains the time when the buffer was last saved, as an
     integer.  *Note Buffer Modification: (lispref)Buffer Modification.

`modtime'
     This field contains the modification time of the visited file.  It
     is set when the file is written or read.  Every time the buffer is
     written to the file, this field is compared to the modification
     time of the file.  *Note Buffer Modification: (lispref)Buffer
     Modification.

`auto_save_modified'
     This field contains the time when the buffer was last auto-saved.

`last_window_start'
     This field contains the `window-start' position in the buffer as of
     the last time the buffer was displayed in a window.

`undo_list'
     This field points to the buffer's undo list.  *Note Undo:
     (lispref)Undo.

`syntax_table_v'
     This field contains the syntax table for the buffer.  *Note Syntax
     Tables: (lispref)Syntax Tables.

`downcase_table'
     This field contains the conversion table for converting text to
     lower case.  *Note Case Tables: (lispref)Case Tables.

`upcase_table'
     This field contains the conversion table for converting text to
     upper case.  *Note Case Tables: (lispref)Case Tables.

`case_canon_table'
     This field contains the conversion table for canonicalizing text
     for case-folding search.  *Note Case Tables: (lispref)Case Tables.

`case_eqv_table'
     This field contains the equivalence table for case-folding search.
     *Note Case Tables: (lispref)Case Tables.

`display_table'
     This field contains the buffer's display table, or `nil' if it
     doesn't have one.  *Note Display Tables: (lispref)Display Tables.

`markers'
     This field contains the chain of all markers that currently point
     into the buffer.  Deletion of text in the buffer, and motion of
     the buffer's gap, must check each of these markers and perhaps
     update it.  *Note Markers: (lispref)Markers.

`backed_up'
     This field is a flag that tells whether a backup file has been
     made for the visited file of this buffer.

`mark'
     This field contains the mark for the buffer.  The mark is a marker,
     hence it is also included on the list `markers'.  *Note The Mark:
     (lispref)The Mark.

`mark_active'
     This field is non-`nil' if the buffer's mark is active.

`local_var_alist'
     This field contains the association list describing the variables
     local in this buffer, and their values, with the exception of
     local variables that have special slots in the buffer object.
     (Those slots are omitted from this table.)  *Note Buffer-Local
     Variables: (lispref)Buffer-Local Variables.

`modeline_format'
     This field contains a Lisp object which controls how to display
     the mode line for this buffer.  *Note Modeline Format:
     (lispref)Modeline Format.

`base_buffer'
     This field holds the buffer's base buffer (if it is an indirect
     buffer), or `nil'.

\1f
File: internals.info,  Node: MULE Character Sets and Encodings,  Next: The Lisp Reader and Compiler,  Prev: Buffers and Textual Representation,  Up: Top

17 MULE Character Sets and Encodings
************************************

Recall that there are two primary ways that text is represented in
XEmacs.  The "buffer" representation sees the text as a series of bytes
(Bufbytes), with a variable number of bytes used per character.  The
"character" representation sees the text as a series of integers
(Emchars), one per character.  The character representation is a cleaner
representation from a theoretical standpoint, and is thus used in many
cases when lots of manipulations on a string need to be done.  However,
the buffer representation is the standard representation used in both
Lisp strings and buffers, and because of this, it is the "default"
representation that text comes in.  The reason for using this
representation is that it's compact and is compatible with ASCII.

* Menu:

* Character Sets::
* Encodings::
* Internal Mule Encodings::
* CCL::

\1f
File: internals.info,  Node: Character Sets,  Next: Encodings,  Up: MULE Character Sets and Encodings

17.1 Character Sets
===================

A character set (or "charset") is an ordered set of characters.  A
particular character in a charset is indexed using one or more
"position codes", which are non-negative integers.  The number of
position codes needed to identify a particular character in a charset is
called the "dimension" of the charset.  In XEmacs/Mule, all charsets
have dimension 1 or 2, and the size of all charsets (except for a few
special cases) is either 94, 96, 94 by 94, or 96 by 96.  The range of
position codes used to index characters from any of these types of
character sets is as follows:

     Charset type            Position code 1         Position code 2
     ------------------------------------------------------------
     94                      33 - 126                N/A
     96                      32 - 127                N/A
     94x94                   33 - 126                33 - 126
     96x96                   32 - 127                32 - 127

   Note that in the above cases position codes do not start at an
expected value such as 0 or 1.  The reason for this will become clear
later.

   For example, Latin-1 is a 96-character charset, and JISX0208 (the
Japanese national character set) is a 94x94-character charset.

   [Note that, although the ranges above define the _valid_ position
codes for a charset, some of the slots in a particular charset may in
fact be empty.  This is the case for JISX0208, for example, where (e.g.)
all the slots whose first position code is in the range 118 - 127 are
empty.]

   There are three charsets that do not follow the above rules.  All of
them have one dimension, and have ranges of position codes as follows:

     Charset name            Position code 1
     ------------------------------------
     ASCII                   0 - 127
     Control-1               0 - 31
     Composite               0 - some large number

   (The upper bound of the position code for composite characters has
not yet been determined, but it will probably be at least 16,383).

   ASCII is the union of two subsidiary character sets: Printing-ASCII
(the printing ASCII character set, consisting of position codes 33 -
126, like for a standard 94-character charset) and Control-ASCII (the
non-printing characters that would appear in a binary file with codes 0
- 32 and 127).

   Control-1 contains the non-printing characters that would appear in a
binary file with codes 128 - 159.

   Composite contains characters that are generated by overstriking one
or more characters from other charsets.

   Note that some characters in ASCII, and all characters in Control-1,
are "control" (non-printing) characters.  These have no printed
representation but instead control some other function of the printing
(e.g. TAB or 8 moves the current character position to the next tab
stop).  All other characters in all charsets are "graphic" (printing)
characters.

   When a binary file is read in, the bytes in the file are assigned to
character sets as follows:

     Bytes           Character set           Range
     --------------------------------------------------
     0 - 127         ASCII                   0 - 127
     128 - 159       Control-1               0 - 31
     160 - 255       Latin-1                 32 - 127

   This is a bit ad-hoc but gets the job done.

\1f
File: internals.info,  Node: Encodings,  Next: Internal Mule Encodings,  Prev: Character Sets,  Up: MULE Character Sets and Encodings

17.2 Encodings
==============

An "encoding" is a way of numerically representing characters from one
or more character sets.  If an encoding only encompasses one character
set, then the position codes for the characters in that character set
could be used directly.  This is not possible, however, if more than
one character set is to be used in the encoding.

   For example, the conversion detailed above between bytes in a binary
file and characters is effectively an encoding that encompasses the
three character sets ASCII, Control-1, and Latin-1 in a stream of 8-bit
bytes.

   Thus, an encoding can be viewed as a way of encoding characters from
a specified group of character sets using a stream of bytes, each of
which contains a fixed number of bits (but not necessarily 8, as in the
common usage of "byte").

   Here are descriptions of a couple of common encodings:

* Menu:

* Japanese EUC (Extended Unix Code)::
* JIS7::

\1f
File: internals.info,  Node: Japanese EUC (Extended Unix Code),  Next: JIS7,  Up: Encodings

17.2.1 Japanese EUC (Extended Unix Code)
----------------------------------------

This encompasses the character sets Printing-ASCII, Japanese-JISX0201,
and Japanese-JISX0208-Kana (half-width katakana, the right half of
JISX0201).  It uses 8-bit bytes.

   Note that Printing-ASCII and Japanese-JISX0201-Kana are 94-character
charsets, while Japanese-JISX0208 is a 94x94-character charset.

   The encoding is as follows:

     Character set            Representation (PC=position-code)
     -------------            --------------
     Printing-ASCII           PC1
     Japanese-JISX0201-Kana   0x8E       | PC1 + 0x80
     Japanese-JISX0208        PC1 + 0x80 | PC2 + 0x80
     Japanese-JISX0212        PC1 + 0x80 | PC2 + 0x80

\1f
File: internals.info,  Node: JIS7,  Prev: Japanese EUC (Extended Unix Code),  Up: Encodings

17.2.2 JIS7
-----------

This encompasses the character sets Printing-ASCII,
Japanese-JISX0201-Roman (the left half of JISX0201; this character set
is very similar to Printing-ASCII and is a 94-character charset),
Japanese-JISX0208, and Japanese-JISX0201-Kana.  It uses 7-bit bytes.

   Unlike Japanese EUC, this is a "modal" encoding, which means that
there are multiple states that the encoding can be in, which affect how
the bytes are to be interpreted.  Special sequences of bytes (called
"escape sequences") are used to change states.

   The encoding is as follows:

     Character set              Representation (PC=position-code)
     -------------              --------------
     Printing-ASCII             PC1
     Japanese-JISX0201-Roman    PC1
     Japanese-JISX0201-Kana     PC1
     Japanese-JISX0208          PC1 PC2


     Escape sequence   ASCII equivalent   Meaning
     ---------------   ----------------   -------
     0x1B 0x28 0x4A    ESC ( J            invoke Japanese-JISX0201-Roman
     0x1B 0x28 0x49    ESC ( I            invoke Japanese-JISX0201-Kana
     0x1B 0x24 0x42    ESC $ B            invoke Japanese-JISX0208
     0x1B 0x28 0x42    ESC ( B            invoke Printing-ASCII

   Initially, Printing-ASCII is invoked.

\1f
File: internals.info,  Node: Internal Mule Encodings,  Next: CCL,  Prev: Encodings,  Up: MULE Character Sets and Encodings

17.3 Internal Mule Encodings
============================

In XEmacs/Mule, each character set is assigned a unique number, called a
"leading byte".  This is used in the encodings of a character.  Leading
bytes are in the range 0x80 - 0xFF (except for ASCII, which has a
leading byte of 0), although some leading bytes are reserved.

   Charsets whose leading byte is in the range 0x80 - 0x9F are called
"official" and are used for built-in charsets.  Other charsets are
called "private" and have leading bytes in the range 0xA0 - 0xFF; these
are user-defined charsets.

   More specifically:

     Character set           Leading byte
     -------------           ------------
     ASCII                   0
     Composite               0x80
     Dimension-1 Official    0x81 - 0x8D
                               (0x8E is free)
     Control-1               0x8F
     Dimension-2 Official    0x90 - 0x99
                               (0x9A - 0x9D are free;
                                0x9E and 0x9F are reserved)
     Dimension-1 Private     0xA0 - 0xEF
     Dimension-2 Private     0xF0 - 0xFF

   There are two internal encodings for characters in XEmacs/Mule.  One
is called "string encoding" and is an 8-bit encoding that is used for
representing characters in a buffer or string.  It uses 1 to 4 bytes per
character.  The other is called "character encoding" and is a 19-bit
encoding that is used for representing characters individually in a
variable.

   (In the following descriptions, we'll ignore composite characters for
the moment.  We also give a general (structural) overview first,
followed later by the exact details.)

* Menu:

* Internal String Encoding::
* Internal Character Encoding::

\1f
File: internals.info,  Node: Internal String Encoding,  Next: Internal Character Encoding,  Up: Internal Mule Encodings

17.3.1 Internal String Encoding
-------------------------------

ASCII characters are encoded using their position code directly.  Other
characters are encoded using their leading byte followed by their
position code(s) with the high bit set.  Characters in private character
sets have their leading byte prefixed with a "leading byte prefix",
which is either 0x9E or 0x9F. (No character sets are ever assigned these
leading bytes.) Specifically:

     Character set           Encoding (PC=position-code, LB=leading-byte)
     -------------           --------
     ASCII                   PC-1 |
     Control-1               LB   |  PC1 + 0xA0 |
     Dimension-1 official    LB   |  PC1 + 0x80 |
     Dimension-1 private     0x9E |  LB         | PC1 + 0x80 |
     Dimension-2 official    LB   |  PC1 + 0x80 | PC2 + 0x80 |
     Dimension-2 private     0x9F |  LB         | PC1 + 0x80 | PC2 + 0x80

   The basic characteristic of this encoding is that the first byte of
all characters is in the range 0x00 - 0x9F, and the second and
following bytes of all characters is in the range 0xA0 - 0xFF.  This
means that it is impossible to get out of sync, or more specifically:

  1. Given any byte position, the beginning of the character it is
     within can be determined in constant time.

  2. Given any byte position at the beginning of a character, the
     beginning of the next character can be determined in constant time.

  3. Given any byte position at the beginning of a character, the
     beginning of the previous character can be determined in constant
     time.

  4. Textual searches can simply treat encoded strings as if they were
     encoded in a one-byte-per-character fashion rather than the actual
     multi-byte encoding.

   None of the standard non-modal encodings meet all of these
conditions.  For example, EUC satisfies only (2) and (3), while
Shift-JIS and Big5 (not yet described) satisfy only (2). (All non-modal
encodings must satisfy (2), in order to be unambiguous.)

\1f
File: internals.info,  Node: Internal Character Encoding,  Prev: Internal String Encoding,  Up: Internal Mule Encodings

17.3.2 Internal Character Encoding
----------------------------------

One 19-bit word represents a single character.  The word is separated
into three fields:

     Bit number:     18 17 16 15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
                     <------------> <------------------> <------------------>
     Field:                1                  2                    3

   Note that fields 2 and 3 hold 7 bits each, while field 1 holds 5
bits.

     Character set           Field 1         Field 2         Field 3
     -------------           -------         -------         -------
     ASCII                      0               0              PC1
        range:                                                   (00 - 7F)
     Control-1                  0               1              PC1
        range:                                                   (00 - 1F)
     Dimension-1 official       0            LB - 0x80         PC1
        range:                                    (01 - 0D)      (20 - 7F)
     Dimension-1 private        0            LB - 0x80         PC1
        range:                                    (20 - 6F)      (20 - 7F)
     Dimension-2 official    LB - 0x8F         PC1             PC2
        range:                    (01 - 0A)       (20 - 7F)      (20 - 7F)
     Dimension-2 private     LB - 0xE1         PC1             PC2
        range:                    (0F - 1E)       (20 - 7F)      (20 - 7F)
     Composite                 0x1F             ?               ?

   Note that character codes 0 - 255 are the same as the "binary
encoding" described above.

\1f
File: internals.info,  Node: CCL,  Prev: Internal Mule Encodings,  Up: MULE Character Sets and Encodings

17.4 CCL
========

     CCL PROGRAM SYNTAX:
          CCL_PROGRAM := (CCL_MAIN_BLOCK
                          [ CCL_EOF_BLOCK ])

          CCL_MAIN_BLOCK := CCL_BLOCK
          CCL_EOF_BLOCK := CCL_BLOCK

          CCL_BLOCK := STATEMENT | (STATEMENT [STATEMENT ...])
          STATEMENT :=
                  SET | IF | BRANCH | LOOP | REPEAT | BREAK
                  | READ | WRITE

          SET := (REG = EXPRESSION) | (REG SELF_OP EXPRESSION)
                 | INT-OR-CHAR

          EXPRESSION := ARG | (EXPRESSION OP ARG)

          IF := (if EXPRESSION CCL_BLOCK CCL_BLOCK)
          BRANCH := (branch EXPRESSION CCL_BLOCK [CCL_BLOCK ...])
          LOOP := (loop STATEMENT [STATEMENT ...])
          BREAK := (break)
          REPEAT := (repeat)
                  | (write-repeat [REG | INT-OR-CHAR | string])
                  | (write-read-repeat REG [INT-OR-CHAR | string | ARRAY]?)
          READ := (read REG) | (read REG REG)
                  | (read-if REG ARITH_OP ARG CCL_BLOCK CCL_BLOCK)
                  | (read-branch REG CCL_BLOCK [CCL_BLOCK ...])
          WRITE := (write REG) | (write REG REG)
                  | (write INT-OR-CHAR) | (write STRING) | STRING
                  | (write REG ARRAY)
          END := (end)

          REG := r0 | r1 | r2 | r3 | r4 | r5 | r6 | r7
          ARG := REG | INT-OR-CHAR
          OP :=   + | - | * | / | % | & | '|' | ^ | << | >> | <8 | >8 | //
                  | < | > | == | <= | >= | !=
          SELF_OP :=
                  += | -= | *= | /= | %= | &= | '|=' | ^= | <<= | >>=
          ARRAY := '[' INT-OR-CHAR ... ']'
          INT-OR-CHAR := INT | CHAR

     MACHINE CODE:

     The machine code consists of a vector of 32-bit words.
     The first such word specifies the start of the EOF section of the code;
     this is the code executed to handle any stuff that needs to be done
     (e.g. designating back to ASCII and left-to-right mode) after all
     other encoded/decoded data has been written out.  This is not used for
     charset CCL programs.

     REGISTER: 0..7  -- referred by RRR or rrr

     OPERATOR BIT FIELD (27-bit): XXXXXXXXXXXXXXX RRR TTTTT
             TTTTT (5-bit): operator type
             RRR (3-bit): register number
             XXXXXXXXXXXXXXXX (15-bit):
                     CCCCCCCCCCCCCCC: constant or address
                     000000000000rrr: register number

     AAAA:   00000 +
             00001 -
             00010 *
             00011 /
             00100 %
             00101 &
             00110 |
             00111 ~

             01000 <<
             01001 >>
             01010 <8
             01011 >8
             01100 //
             01101 not used
             01110 not used
             01111 not used

             10000 <
             10001 >
             10010 ==
             10011 <=
             10100 >=
             10101 !=

     OPERATORS:      TTTTT RRR XX..

     SetCS:          00000 RRR C...C      RRR = C...C
     SetCL:          00001 RRR .....      RRR = c...c
                     c.............c
     SetR:           00010 RRR ..rrr      RRR = rrr
     SetA:           00011 RRR ..rrr      RRR = array[rrr]
                     C.............C      size of array = C...C
                     c.............c      contents = c...c

     Jump:           00100 000 c...c      jump to c...c
     JumpCond:       00101 RRR c...c      if (!RRR) jump to c...c
     WriteJump:      00110 RRR c...c      Write1 RRR, jump to c...c
     WriteReadJump:  00111 RRR c...c      Write1, Read1 RRR, jump to c...c
     WriteCJump:     01000 000 c...c      Write1 C...C, jump to c...c
                     C...C
     WriteCReadJump: 01001 RRR c...c      Write1 C...C, Read1 RRR,
                     C.............C      and jump to c...c
     WriteSJump:     01010 000 c...c      WriteS, jump to c...c
                     C.............C
                     S.............S
                     ...
     WriteSReadJump: 01011 RRR c...c      WriteS, Read1 RRR, jump to c...c
                     C.............C
                     S.............S
                     ...
     WriteAReadJump: 01100 RRR c...c      WriteA, Read1 RRR, jump to c...c
                     C.............C      size of array = C...C
                     c.............c      contents = c...c
                     ...
     Branch:         01101 RRR C...C      if (RRR >= 0 && RRR < C..)
                     c.............c      branch to (RRR+1)th address
     Read1:          01110 RRR ...        read 1-byte to RRR
     Read2:          01111 RRR ..rrr      read 2-byte to RRR and rrr
     ReadBranch:     10000 RRR C...C      Read1 and Branch
                     c.............c
                     ...
     Write1:         10001 RRR .....      write 1-byte RRR
     Write2:         10010 RRR ..rrr      write 2-byte RRR and rrr
     WriteC:         10011 000 .....      write 1-char C...CC
                     C.............C
     WriteS:         10100 000 .....      write C..-byte of string
                     C.............C
                     S.............S
                     ...
     WriteA:         10101 RRR .....      write array[RRR]
                     C.............C      size of array = C...C
                     c.............c      contents = c...c
                     ...
     End:            10110 000 .....      terminate the execution

     SetSelfCS:      10111 RRR C...C      RRR AAAAA= C...C
                     ..........AAAAA
     SetSelfCL:      11000 RRR .....      RRR AAAAA= c...c
                     c.............c
                     ..........AAAAA
     SetSelfR:       11001 RRR ..Rrr      RRR AAAAA= rrr
                     ..........AAAAA
     SetExprCL:      11010 RRR ..Rrr      RRR = rrr AAAAA c...c
                     c.............c
                     ..........AAAAA
     SetExprR:       11011 RRR ..rrr      RRR = rrr AAAAA Rrr
                     ............Rrr
                     ..........AAAAA
     JumpCondC:      11100 RRR c...c      if !(RRR AAAAA C..) jump to c...c
                     C.............C
                     ..........AAAAA
     JumpCondR:      11101 RRR c...c      if !(RRR AAAAA rrr) jump to c...c
                     ............rrr
                     ..........AAAAA
     ReadJumpCondC:  11110 RRR c...c      Read1 and JumpCondC
                     C.............C
                     ..........AAAAA
     ReadJumpCondR:  11111 RRR c...c      Read1 and JumpCondR
                     ............rrr
                     ..........AAAAA

\1f
File: internals.info,  Node: The Lisp Reader and Compiler,  Next: Lstreams,  Prev: MULE Character Sets and Encodings,  Up: Top

18 The Lisp Reader and Compiler
*******************************

Not yet documented.

\1f
File: internals.info,  Node: Lstreams,  Next: Consoles; Devices; Frames; Windows,  Prev: The Lisp Reader and Compiler,  Up: Top

19 Lstreams
***********

An "lstream" is an internal Lisp object that provides a generic
buffering stream implementation.  Conceptually, you send data to the
stream or read data from the stream, not caring what's on the other end
of the stream.  The other end could be another stream, a file
descriptor, a stdio stream, a fixed block of memory, a reallocating
block of memory, etc.  The main purpose of the stream is to provide a
standard interface and to do buffering.  Macros are defined to read or
write characters, so the calling functions do not have to worry about
blocking data together in order to achieve efficiency.

* Menu:

* Creating an Lstream::         Creating an lstream object.
* Lstream Types::               Different sorts of things that are streamed.
* Lstream Functions::           Functions for working with lstreams.
* Lstream Methods::             Creating new lstream types.

\1f
File: internals.info,  Node: Creating an Lstream,  Next: Lstream Types,  Up: Lstreams

19.1 Creating an Lstream
========================

Lstreams come in different types, depending on what is being interfaced
to.  Although the primitive for creating new lstreams is
`Lstream_new()', generally you do not call this directly.  Instead, you
call some type-specific creation function, which creates the lstream
and initializes it as appropriate for the particular type.

   All lstream creation functions take a MODE argument, specifying what
mode the lstream should be opened as.  This controls whether the
lstream is for input and output, and optionally whether data should be
blocked up in units of MULE characters.  Note that some types of
lstreams can only be opened for input; others only for output; and
others can be opened either way.  #### Richard Mlynarik thinks that
there should be a strict separation between input and output streams,
and he's probably right.

   MODE is a string, one of

`"r"'
     Open for reading.

`"w"'
     Open for writing.

`"rc"'
     Open for reading, but "read" never returns partial MULE characters.

`"wc"'
     Open for writing, but never writes partial MULE characters.

\1f
File: internals.info,  Node: Lstream Types,  Next: Lstream Functions,  Prev: Creating an Lstream,  Up: Lstreams

19.2 Lstream Types
==================

stdio

filedesc

lisp-string

fixed-buffer

resizing-buffer

dynarr

lisp-buffer

print

decoding

encoding

\1f
File: internals.info,  Node: Lstream Functions,  Next: Lstream Methods,  Prev: Lstream Types,  Up: Lstreams

19.3 Lstream Functions
======================

 -- Function: Lstream * Lstream_new (Lstream_implementation *IMP, const
          char *MODE)
     Allocate and return a new Lstream.  This function is not really
     meant to be called directly; rather, each stream type should
     provide its own stream creation function, which creates the stream
     and does any other necessary creation stuff (e.g. opening a file).

 -- Function: void Lstream_set_buffering (Lstream *LSTR,
          Lstream_buffering BUFFERING, int BUFFERING_SIZE)
     Change the buffering of a stream.  See `lstream.h'.  By default the
     buffering is `STREAM_BLOCK_BUFFERED'.

 -- Function: int Lstream_flush (Lstream *LSTR)
     Flush out any pending unwritten data in the stream.  Clear any
     buffered input data.  Returns 0 on success, -1 on error.

 -- Macro: int Lstream_putc (Lstream *STREAM, int C)
     Write out one byte to the stream.  This is a macro and so it is
     very efficient.  The C argument is only evaluated once but the
     STREAM argument is evaluated more than once.  Returns 0 on
     success, -1 on error.

 -- Macro: int Lstream_getc (Lstream *STREAM)
     Read one byte from the stream.  This is a macro and so it is very
     efficient.  The STREAM argument is evaluated more than once.
     Return value is -1 for EOF or error.

 -- Macro: void Lstream_ungetc (Lstream *STREAM, int C)
     Push one byte back onto the input queue.  This will be the next
     byte read from the stream.  Any number of bytes can be pushed back
     and will be read in the reverse order they were pushed back--most
     recent first. (This is necessary for consistency--if there are a
     number of bytes that have been unread and I read and unread a
     byte, it needs to be the first to be read again.) This is a macro
     and so it is very efficient.  The C argument is only evaluated
     once but the STREAM argument is evaluated more than once.

 -- Function: int Lstream_fputc (Lstream *STREAM, int C)
 -- Function: int Lstream_fgetc (Lstream *STREAM)
 -- Function: void Lstream_fungetc (Lstream *STREAM, int C)
     Function equivalents of the above macros.

 -- Function: ssize_t Lstream_read (Lstream *STREAM, void *DATA, size_t
          SIZE)
     Read SIZE bytes of DATA from the stream.  Return the number of
     bytes read.  0 means EOF. -1 means an error occurred and no bytes
     were read.

 -- Function: ssize_t Lstream_write (Lstream *STREAM, void *DATA,
          size_t SIZE)
     Write SIZE bytes of DATA to the stream.  Return the number of
     bytes written.  -1 means an error occurred and no bytes were
     written.

 -- Function: void Lstream_unread (Lstream *STREAM, void *DATA, size_t
          SIZE)
     Push back SIZE bytes of DATA onto the input queue.  The next call
     to `Lstream_read()' with the same size will read the same bytes
     back.  Note that this will be the case even if there is other
     pending unread data.

 -- Function: int Lstream_close (Lstream *STREAM)
     Close the stream.  All data will be flushed out.

 -- Function: void Lstream_reopen (Lstream *STREAM)
     Reopen a closed stream.  This enables I/O on it again.  This is not
     meant to be called except from a wrapper routine that reinitializes
     variables and such--the close routine may well have freed some
     necessary storage structures, for example.

 -- Function: void Lstream_rewind (Lstream *STREAM)
     Rewind the stream to the beginning.

\1f
File: internals.info,  Node: Lstream Methods,  Prev: Lstream Functions,  Up: Lstreams

19.4 Lstream Methods
====================

 -- Lstream Method: ssize_t reader (Lstream *STREAM, unsigned char
          *DATA, size_t SIZE)
     Read some data from the stream's end and store it into DATA, which
     can hold SIZE bytes.  Return the number of bytes read.  A return
     value of 0 means no bytes can be read at this time.  This may be
     because of an EOF, or because there is a granularity greater than
     one byte that the stream imposes on the returned data, and SIZE is
     less than this granularity. (This will happen frequently for
     streams that need to return whole characters, because
     `Lstream_read()' calls the reader function repeatedly until it has
     the number of bytes it wants or until 0 is returned.)  The lstream
     functions do not treat a 0 return as EOF or do anything special;
     however, the calling function will interpret any 0 it gets back as
     EOF.  This will normally not happen unless the caller calls
     `Lstream_read()' with a very small size.

     This function can be `NULL' if the stream is output-only.

 -- Lstream Method: ssize_t writer (Lstream *STREAM, const unsigned
          char *DATA, size_t SIZE)
     Send some data to the stream's end.  Data to be sent is in DATA
     and is SIZE bytes.  Return the number of bytes sent.  This
     function can send and return fewer bytes than is passed in; in that
     case, the function will just be called again until there is no
     data left or 0 is returned.  A return value of 0 means that no
     more data can be currently stored, but there is no error; the data
     will be squirreled away until the writer can accept data. (This is
     useful, e.g., if you're dealing with a non-blocking file
     descriptor and are getting `EWOULDBLOCK' errors.)  This function
     can be `NULL' if the stream is input-only.

 -- Lstream Method: int rewinder (Lstream *STREAM)
     Rewind the stream.  If this is `NULL', the stream is not seekable.

 -- Lstream Method: int seekable_p (Lstream *STREAM)
     Indicate whether this stream is seekable--i.e. it can be rewound.
     This method is ignored if the stream does not have a rewind
     method.  If this method is not present, the result is determined
     by whether a rewind method is present.

 -- Lstream Method: int flusher (Lstream *STREAM)
     Perform any additional operations necessary to flush the data in
     this stream.

 -- Lstream Method: int pseudo_closer (Lstream *STREAM)

 -- Lstream Method: int closer (Lstream *STREAM)
     Perform any additional operations necessary to close this stream
     down.  May be `NULL'.  This function is called when
     `Lstream_close()' is called or when the stream is
     garbage-collected.  When this function is called, all pending data
     in the stream will already have been written out.

 -- Lstream Method: Lisp_Object marker (Lisp_Object LSTREAM, void
          (*MARKFUN) (Lisp_Object))
     Mark this object for garbage collection.  Same semantics as a
     standard `Lisp_Object' marker.  This function can be `NULL'.

\1f
File: internals.info,  Node: Consoles; Devices; Frames; Windows,  Next: The Redisplay Mechanism,  Prev: Lstreams,  Up: Top

20 Consoles; Devices; Frames; Windows
*************************************

* Menu:

* Introduction to Consoles; Devices; Frames; Windows::
* Point::
* Window Hierarchy::
* The Window Object::

\1f
File: internals.info,  Node: Introduction to Consoles; Devices; Frames; Windows,  Next: Point,  Up: Consoles; Devices; Frames; Windows

20.1 Introduction to Consoles; Devices; Frames; Windows
=======================================================

A window-system window that you see on the screen is called a "frame"
in Emacs terminology.  Each frame is subdivided into one or more
non-overlapping panes, called (confusingly) "windows".  Each window
displays the text of a buffer in it. (See above on Buffers.) Note that
buffers and windows are independent entities: Two or more windows can
be displaying the same buffer (potentially in different locations), and
a buffer can be displayed in no windows.

   A single display screen that contains one or more frames is called a
"display".  Under most circumstances, there is only one display.
However, more than one display can exist, for example if you have a
"multi-headed" console, i.e. one with a single keyboard but multiple
displays. (Typically in such a situation, the various displays act like
one large display, in that the mouse is only in one of them at a time,
and moving the mouse off of one moves it into another.) In some cases,
the different displays will have different characteristics, e.g. one
color and one mono.

   XEmacs can display frames on multiple displays.  It can even deal
simultaneously with frames on multiple keyboards (called "consoles" in
XEmacs terminology).  Here is one case where this might be useful: You
are using XEmacs on your workstation at work, and leave it running.
Then you go home and dial in on a TTY line, and you can use the
already-running XEmacs process to display another frame on your local
TTY.

   Thus, there is a hierarchy console -> display -> frame -> window.
There is a separate Lisp object type for each of these four concepts.
Furthermore, there is logically a "selected console", "selected
display", "selected frame", and "selected window".  Each of these
objects is distinguished in various ways, such as being the default
object for various functions that act on objects of that type.  Note
that every containing object remembers the "selected" object among the
objects that it contains: e.g. not only is there a selected window, but
every frame remembers the last window in it that was selected, and
changing the selected frame causes the remembered window within it to
become the selected window.  Similar relationships apply for consoles
to devices and devices to frames.

\1f
File: internals.info,  Node: Point,  Next: Window Hierarchy,  Prev: Introduction to Consoles; Devices; Frames; Windows,  Up: Consoles; Devices; Frames; Windows

20.2 Point
==========

Recall that every buffer has a current insertion position, called
"point".  Now, two or more windows may be displaying the same buffer,
and the text cursor in the two windows (i.e. `point') can be in two
different places.  You may ask, how can that be, since each buffer has
only one value of `point'?  The answer is that each window also has a
value of `point' that is squirreled away in it.  There is only one
selected window, and the value of "point" in that buffer corresponds to
that window.  When the selected window is changed from one window to
another displaying the same buffer, the old value of `point' is stored
into the old window's "point" and the value of `point' from the new
window is retrieved and made the value of `point' in the buffer.  This
means that `window-point' for the selected window is potentially
inaccurate, and if you want to retrieve the correct value of `point'
for a window, you must special-case on the selected window and retrieve
the buffer's point instead.  This is related to why
`save-window-excursion' does not save the selected window's value of
`point'.

\1f
File: internals.info,  Node: Window Hierarchy,  Next: The Window Object,  Prev: Point,  Up: Consoles; Devices; Frames; Windows

20.3 Window Hierarchy
=====================

If a frame contains multiple windows (panes), they are always created
by splitting an existing window along the horizontal or vertical axis.
Terminology is a bit confusing here: to "split a window horizontally"
means to create two side-by-side windows, i.e. to make a _vertical_ cut
in a window.  Likewise, to "split a window vertically" means to create
two windows, one above the other, by making a _horizontal_ cut.

   If you split a window and then split again along the same axis, you
will end up with a number of panes all arranged along the same axis.
The precise way in which the splits were made should not be important,
and this is reflected internally.  Internally, all windows are arranged
in a tree, consisting of two types of windows, "combination" windows
(which have children, and are covered completely by those children) and
"leaf" windows, which have no children and are visible.  Every
combination window has two or more children, all arranged along the same
axis.  There are (logically) two subtypes of windows, depending on
whether their children are horizontally or vertically arrayed.  There is
always one root window, which is either a leaf window (if the frame
contains only one window) or a combination window (if the frame contains
more than one window).  In the latter case, the root window will have
two or more children, either horizontally or vertically arrayed, and
each of those children will be either a leaf window or another
combination window.

   Here are some rules:

  1. Horizontal combination windows can never have children that are
     horizontal combination windows; same for vertical.

  2. Only leaf windows can be split (obviously) and this splitting does
     one of two things: (a) turns the leaf window into a combination
     window and creates two new leaf children, or (b) turns the leaf
     window into one of the two new leaves and creates the other leaf.
     Rule (1) dictates which of these two outcomes happens.

  3. Every combination window must have at least two children.

  4. Leaf windows can never become combination windows.  They can be
     deleted, however.  If this results in a violation of (3), the
     parent combination window also gets deleted.

  5. All functions that accept windows must be prepared to accept
     combination windows, and do something sane (e.g. signal an error
     if so).  Combination windows _do_ escape to the Lisp level.

  6. All windows have three fields governing their contents: these are
     "hchild" (a list of horizontally-arrayed children), "vchild" (a
     list of vertically-arrayed children), and "buffer" (the buffer
     contained in a leaf window).  Exactly one of these will be
     non-`nil'.  Remember that "horizontally-arrayed" means
     "side-by-side" and "vertically-arrayed" means "one above the
     other".

  7. Leaf windows also have markers in their `start' (the first buffer
     position displayed in the window) and `pointm' (the window's
     stashed value of `point'--see above) fields, while combination
     windows have `nil' in these fields.

  8. The list of children for a window is threaded through the `next'
     and `prev' fields of each child window.

  9. *Deleted windows can be undeleted*.  This happens as a result of
     restoring a window configuration, and is unlike frames, displays,
     and consoles, which, once deleted, can never be restored.
     Deleting a window does nothing except set a special `dead' bit to
     1 and clear out the `next', `prev', `hchild', and `vchild' fields,
     for GC purposes.

 10. Most frames actually have two top-level windows--one for the
     minibuffer and one (the "root") for everything else.  The modeline
     (if present) separates these two.  The `next' field of the root
     points to the minibuffer, and the `prev' field of the minibuffer
     points to the root.  The other `next' and `prev' fields are `nil',
     and the frame points to both of these windows.  Minibuffer-less
     frames have no minibuffer window, and the `next' and `prev' of the
     root window are `nil'.  Minibuffer-only frames have no root
     window, and the `next' of the minibuffer window is `nil' but the
     `prev' points to itself. (#### This is an artifact that should be
     fixed.)

\1f
File: internals.info,  Node: The Window Object,  Prev: Window Hierarchy,  Up: Consoles; Devices; Frames; Windows

20.4 The Window Object
======================

Windows have the following accessible fields:

`frame'
     The frame that this window is on.

`mini_p'
     Non-`nil' if this window is a minibuffer window.

`buffer'
     The buffer that the window is displaying.  This may change often
     during the life of the window.

`dedicated'
     Non-`nil' if this window is dedicated to its buffer.

`pointm'
     This is the value of point in the current buffer when this window
     is selected; when it is not selected, it retains its previous
     value.

`start'
     The position in the buffer that is the first character to be
     displayed in the window.

`force_start'
     If this flag is non-`nil', it says that the window has been
     scrolled explicitly by the Lisp program.  This affects what the
     next redisplay does if point is off the screen: instead of
     scrolling the window to show the text around point, it moves point
     to a location that is on the screen.

`last_modified'
     The `modified' field of the window's buffer, as of the last time a
     redisplay completed in this window.

`last_point'
     The buffer's value of point, as of the last time a redisplay
     completed in this window.

`left'
     This is the left-hand edge of the window, measured in columns.
     (The leftmost column on the screen is column 0.)

`top'
     This is the top edge of the window, measured in lines.  (The top
     line on the screen is line 0.)

`height'
     The height of the window, measured in lines.

`width'
     The width of the window, measured in columns.

`next'
     This is the window that is the next in the chain of siblings.  It
     is `nil' in a window that is the rightmost or bottommost of a
     group of siblings.

`prev'
     This is the window that is the previous in the chain of siblings.
     It is `nil' in a window that is the leftmost or topmost of a group
     of siblings.

`parent'
     Internally, XEmacs arranges windows in a tree; each group of
     siblings has a parent window whose area includes all the siblings.
     This field points to a window's parent.

     Parent windows do not display buffers, and play little role in
     display except to shape their child windows.  Emacs Lisp programs
     usually have no access to the parent windows; they operate on the
     windows at the leaves of the tree, which actually display buffers.

`hscroll'
     This is the number of columns that the display in the window is
     scrolled horizontally to the left.  Normally, this is 0.

`use_time'
     This is the last time that the window was selected.  The function
     `get-lru-window' uses this field.

`display_table'
     The window's display table, or `nil' if none is specified for it.

`update_mode_line'
     Non-`nil' means this window's mode line needs to be updated.

`base_line_number'
     The line number of a certain position in the buffer, or `nil'.
     This is used for displaying the line number of point in the mode
     line.

`base_line_pos'
     The position in the buffer for which the line number is known, or
     `nil' meaning none is known.

`region_showing'
     If the region (or part of it) is highlighted in this window, this
     field holds the mark position that made one end of that region.
     Otherwise, this field is `nil'.

\1f
File: internals.info,  Node: The Redisplay Mechanism,  Next: Extents,  Prev: Consoles; Devices; Frames; Windows,  Up: Top

21 The Redisplay Mechanism
**************************

The redisplay mechanism is one of the most complicated sections of
XEmacs, especially from a conceptual standpoint.  This is doubly so
because, unlike for the basic aspects of the Lisp interpreter, the
computer science theories of how to efficiently handle redisplay are not
well-developed.

   When working with the redisplay mechanism, remember the Golden Rules
of Redisplay:

  1. It Is Better To Be Correct Than Fast.

  2. Thou Shalt Not Run Elisp From Within Redisplay.

  3. It Is Better To Be Fast Than Not To Be.

* Menu:

* Critical Redisplay Sections::
* Line Start Cache::
* Redisplay Piece by Piece::

\1f
File: internals.info,  Node: Critical Redisplay Sections,  Next: Line Start Cache,  Up: The Redisplay Mechanism

21.1 Critical Redisplay Sections
================================

Within this section, we are defenseless and assume that the following
cannot happen:

  1. garbage collection

  2. Lisp code evaluation

  3. frame size changes

   We ensure (3) by calling `hold_frame_size_changes()', which will
cause any pending frame size changes to get put on hold till after the
end of the critical section.  (1) follows automatically if (2) is met.
#### Unfortunately, there are some places where Lisp code can be called
within this section.  We need to remove them.

   If `Fsignal()' is called during this critical section, we will
`abort()'.

   If garbage collection is called during this critical section, we
simply return. #### We should abort instead.

   #### If a frame-size change does occur we should probably actually
be preempting redisplay.

\1f
File: internals.info,  Node: Line Start Cache,  Next: Redisplay Piece by Piece,  Prev: Critical Redisplay Sections,  Up: The Redisplay Mechanism

21.2 Line Start Cache
=====================

The traditional scrolling code in Emacs breaks in a variable height
world.  It depends on the key assumption that the number of lines that
can be displayed at any given time is fixed.  This led to a complete
separation of the scrolling code from the redisplay code.  In order to
fully support variable height lines, the scrolling code must actually be
tightly integrated with redisplay.  Only redisplay can determine how
many lines will be displayed on a screen for any given starting point.

   What is ideally wanted is a complete list of the starting buffer
position for every possible display line of a buffer along with the
height of that display line.  Maintaining such a full list would be very
expensive.  We settle for having it include information for all areas
which we happen to generate anyhow (i.e. the region currently being
displayed) and for those areas we need to work with.

   In order to ensure that the cache accurately represents what
redisplay would actually show, it is necessary to invalidate it in many
situations.  If the buffer changes, the starting positions may no longer
be correct.  If a face or an extent has changed then the line heights
may have altered.  These events happen frequently enough that the cache
can end up being constantly disabled.  With this potentially constant
invalidation when is the cache ever useful?

   Even if the cache is invalidated before every single usage, it is
necessary.  Scrolling often requires knowledge about display lines which
are actually above or below the visible region.  The cache provides a
convenient light-weight method of storing this information for multiple
display regions.  This knowledge is necessary for the scrolling code to
always obey the First Golden Rule of Redisplay.

   If the cache already contains all of the information that the
scrolling routines happen to need so that it doesn't have to go
generate it, then we are able to obey the Third Golden Rule of
Redisplay.  The first thing we do to help out the cache is to always
add the displayed region.  This region had to be generated anyway, so
the cache ends up getting the information basically for free.  In those
cases where a user is simply scrolling around viewing a buffer there is
a high probability that this is sufficient to always provide the needed
information.  The second thing we can do is be smart about invalidating
the cache.

   TODO--Be smart about invalidating the cache.  Potential places:

   * Insertions at end-of-line which don't cause line-wraps do not
     alter the starting positions of any display lines.  These types of
     buffer modifications should not invalidate the cache.  This is
     actually a large optimization for redisplay speed as well.

   * Buffer modifications frequently only affect the display of lines
     at and below where they occur.  In these situations we should only
     invalidate the part of the cache starting at where the
     modification occurs.

   In case you're wondering, the Second Golden Rule of Redisplay is not
applicable.

\1f
File: internals.info,  Node: Redisplay Piece by Piece,  Prev: Line Start Cache,  Up: The Redisplay Mechanism

21.3 Redisplay Piece by Piece
=============================

As you can begin to see redisplay is complex and also not well
documented. Chuck no longer works on XEmacs so this section is my take
on the workings of redisplay.

   Redisplay happens in three phases:

  1. Determine desired display in area that needs redisplay.
     Implemented by `redisplay.c'

  2. Compare desired display with current display Implemented by
     `redisplay-output.c'

  3. Output changes Implemented by `redisplay-output.c',
     `redisplay-x.c', `redisplay-msw.c' and `redisplay-tty.c'

   Steps 1 and 2 are device-independent and relatively complex.  Step 3
is mostly device-dependent.

   Determining the desired display

   Display attributes are stored in `display_line' structures. Each
`display_line' consists of a set of `display_block''s and each
`display_block' contains a number of `rune''s. Generally dynarr's of
`display_line''s are held by each window representing the current
display and the desired display.

   The `display_line' structures are tightly tied to buffers which
presents a problem for redisplay as this connection is bogus for the
modeline. Hence the `display_line' generation routines are duplicated
for generating the modeline. This means that the modeline display code
has many bugs that the standard redisplay code does not.

   The guts of `display_line' generation are in `create_text_block',
which creates a single display line for the desired locale. This
incrementally parses the characters on the current line and generates
redisplay structures for each.

   Gutter redisplay is different. Because the data to display is stored
in a string we cannot use `create_text_block'. Instead we use
`create_text_string_block' which performs the same function as
`create_text_block' but for strings. Many of the complexities of
`create_text_block' to do with cursor handling and selective display
have been removed.

\1f
File: internals.info,  Node: Extents,  Next: Faces,  Prev: The Redisplay Mechanism,  Up: Top

22 Extents
**********

* Menu:

* Introduction to Extents::     Extents are ranges over text, with properties.
* Extent Ordering::             How extents are ordered internally.
* Format of the Extent Info::   The extent information in a buffer or string.
* Zero-Length Extents::         A weird special case.
* Mathematics of Extent Ordering::  A rigorous foundation.
* Extent Fragments::            Cached information useful for redisplay.

\1f
File: internals.info,  Node: Introduction to Extents,  Next: Extent Ordering,  Up: Extents

22.1 Introduction to Extents
============================

Extents are regions over a buffer, with a start and an end position
denoting the region of the buffer included in the extent.  In addition,
either end can be closed or open, meaning that the endpoint is or is
not logically included in the extent.  Insertion of a character at a
closed endpoint causes the character to go inside the extent; insertion
at an open endpoint causes the character to go outside.

   Extent endpoints are stored using memory indices (see `insdel.c'),
to minimize the amount of adjusting that needs to be done when
characters are inserted or deleted.

   (Formerly, extent endpoints at the gap could be either before or
after the gap, depending on the open/closedness of the endpoint.  The
intent of this was to make it so that insertions would automatically go
inside or out of extents as necessary with no further work needing to
be done.  It didn't work out that way, however, and just ended up
complexifying and buggifying all the rest of the code.)

\1f
File: internals.info,  Node: Extent Ordering,  Next: Format of the Extent Info,  Prev: Introduction to Extents,  Up: Extents

22.2 Extent Ordering
====================

Extents are compared using memory indices.  There are two orderings for
extents and both orders are kept current at all times.  The normal or
"display" order is as follows:

     Extent A is ``less than'' extent B,
     that is, earlier in the display order,
       if:    A-start < B-start,
       or if: A-start = B-start, and A-end > B-end

   So if two extents begin at the same position, the larger of them is
the earlier one in the display order (`EXTENT_LESS' is true).

   For the e-order, the same thing holds:

     Extent A is ``less than'' extent B in e-order,
     that is, later in the buffer,
       if:    A-end < B-end,
       or if: A-end = B-end, and A-start > B-start

   So if two extents end at the same position, the smaller of them is
the earlier one in the e-order (`EXTENT_E_LESS' is true).

   The display order and the e-order are complementary orders: any
theorem about the display order also applies to the e-order if you swap
all occurrences of "display order" and "e-order", "less than" and
"greater than", and "extent start" and "extent end".

\1f
File: internals.info,  Node: Format of the Extent Info,  Next: Zero-Length Extents,  Prev: Extent Ordering,  Up: Extents

22.3 Format of the Extent Info
==============================

An extent-info structure consists of a list of the buffer or string's
extents and a "stack of extents" that lists all of the extents over a
particular position.  The stack-of-extents info is used for
optimization purposes--it basically caches some info that might be
expensive to compute.  Certain otherwise hard computations are easy
given the stack of extents over a particular position, and if the stack
of extents over a nearby position is known (because it was calculated
at some prior point in time), it's easy to move the stack of extents to
the proper position.

   Given that the stack of extents is an optimization, and given that
it requires memory, a string's stack of extents is wiped out each time
a garbage collection occurs.  Therefore, any time you retrieve the
stack of extents, it might not be there.  If you need it to be there,
use the `_force' version.

   Similarly, a string may or may not have an extent_info structure.
(Generally it won't if there haven't been any extents added to the
string.) So use the `_force' version if you need the extent_info
structure to be there.

   A list of extents is maintained as a double gap array: one gap array
is ordered by start index (the "display order") and the other is
ordered by end index (the "e-order").  Note that positions in an extent
list should logically be conceived of as referring _to_ a particular
extent (as is the norm in programs) rather than sitting between two
extents.  Note also that callers of these functions should not be aware
of the fact that the extent list is implemented as an array, except for
the fact that positions are integers (this should be generalized to
handle integers and linked list equally well).

\1f
File: internals.info,  Node: Zero-Length Extents,  Next: Mathematics of Extent Ordering,  Prev: Format of the Extent Info,  Up: Extents

22.4 Zero-Length Extents
========================

Extents can be zero-length, and will end up that way if their endpoints
are explicitly set that way or if their detachable property is `nil'
and all the text in the extent is deleted. (The exception is open-open
zero-length extents, which are barred from existing because there is no
sensible way to define their properties.  Deletion of the text in an
open-open extent causes it to be converted into a closed-open extent.)
Zero-length extents are primarily used to represent annotations, and
behave as follows:

  1. Insertion at the position of a zero-length extent expands the
     extent if both endpoints are closed; goes after the extent if it
     is closed-open; and goes before the extent if it is open-closed.

  2. Deletion of a character on a side of a zero-length extent whose
     corresponding endpoint is closed causes the extent to be detached
     if it is detachable; if the extent is not detachable or the
     corresponding endpoint is open, the extent remains in the buffer,
     moving as necessary.

   Note that closed-open, non-detachable zero-length extents behave
exactly like markers and that open-closed, non-detachable zero-length
extents behave like the "point-type" marker in Mule.

\1f
File: internals.info,  Node: Mathematics of Extent Ordering,  Next: Extent Fragments,  Prev: Zero-Length Extents,  Up: Extents

22.5 Mathematics of Extent Ordering
===================================

The extents in a buffer are ordered by "display order" because that is
that order that the redisplay mechanism needs to process them in.  The
e-order is an auxiliary ordering used to facilitate operations over
extents.  The operations that can be performed on the ordered list of
extents in a buffer are

  1. Locate where an extent would go if inserted into the list.

  2. Insert an extent into the list.

  3. Remove an extent from the list.

  4. Map over all the extents that overlap a range.

   (4) requires being able to determine the first and last extents that
overlap a range.

   NOTE: "overlap" is used as follows:

   * two ranges overlap if they have at least one point in common.
     Whether the endpoints are open or closed makes a difference here.

   * a point overlaps a range if the point is contained within the
     range; this is equivalent to treating a point P as the range [P,
     P].

   * In the case of an _extent_ overlapping a point or range, the extent
     is normally treated as having closed endpoints.  This applies
     consistently in the discussion of stacks of extents and such below.
     Note that this definition of overlap is not necessarily consistent
     with the extents that `map-extents' maps over, since `map-extents'
     sometimes pays attention to whether the endpoints of an extents
     are open or closed.  But for our purposes, it greatly simplifies
     things to treat all extents as having closed endpoints.

   First, define >, <, <=, etc. as applied to extents to mean
comparison according to the display order.  Comparison between an
extent E and an index I means comparison between E and the range [I, I].

   Also define e>, e<, e<=, etc. to mean comparison according to the
e-order.

   For any range R, define R(0) to be the starting index of the range
and R(1) to be the ending index of the range.

   For any extent E, define E(next) to be the extent directly following
E, and E(prev) to be the extent directly preceding E.  Assume E(next)
and E(prev) can be determined from E in constant time.  (This is
because we store the extent list as a doubly linked list.)

   Similarly, define E(e-next) and E(e-prev) to be the extents directly
following and preceding E in the e-order.

   Now:

   Let R be a range.  Let F be the first extent overlapping R.  Let L
be the last extent overlapping R.

   Theorem 1: R(1) lies between L and L(next), i.e. L <= R(1) < L(next).

   This follows easily from the definition of display order.  The basic
reason that this theorem applies is that the display order sorts by
increasing starting index.

   Therefore, we can determine L just by looking at where we would
insert R(1) into the list, and if we know F and are moving forward over
extents, we can easily determine when we've hit L by comparing the
extent we're at to R(1).

     Theorem 2: F(e-prev) e< [1, R(0)] e<= F.

   This is the analog of Theorem 1, and applies because the e-order
sorts by increasing ending index.

   Therefore, F can be found in the same amount of time as operation
(1), i.e. the time that it takes to locate where an extent would go if
inserted into the e-order list.

   If the lists were stored as balanced binary trees, then operation (1)
would take logarithmic time, which is usually quite fast.  However,
currently they're stored as simple doubly-linked lists, and instead we
do some caching to try to speed things up.

   Define a "stack of extents" (or "SOE") as the set of extents
(ordered in the display order) that overlap an index I, together with
the SOE's "previous" extent, which is an extent that precedes I in the
e-order. (Hopefully there will not be very many extents between I and
the previous extent.)

   Now:

   Let I be an index, let S be the stack of extents on I, let F be the
first extent in S, and let P be S's previous extent.

   Theorem 3: The first extent in S is the first extent that overlaps
any range [I, J].

   Proof: Any extent that overlaps [I, J] but does not include I must
have a start index > I, and thus be greater than any extent in S.

   Therefore, finding the first extent that overlaps a range R is the
same as finding the first extent that overlaps R(0).

   Theorem 4: Let I2 be an index such that I2 > I, and let F2 be the
first extent that overlaps I2.  Then, either F2 is in S or F2 is
greater than any extent in S.

   Proof: If F2 does not include I then its start index is greater than
I and thus it is greater than any extent in S, including F.  Otherwise,
F2 includes I and thus is in S, and thus F2 >= F.

\1f
File: internals.info,  Node: Extent Fragments,  Prev: Mathematics of Extent Ordering,  Up: Extents

22.6 Extent Fragments
=====================

Imagine that the buffer is divided up into contiguous, non-overlapping
"runs" of text such that no extent starts or ends within a run (extents
that abut the run don't count).

   An extent fragment is a structure that holds data about the run that
contains a particular buffer position (if the buffer position is at the
junction of two runs, the run after the position is used)--the
beginning and end of the run, a list of all of the extents in that run,
the "merged face" that results from merging all of the faces
corresponding to those extents, the begin and end glyphs at the
beginning of the run, etc.  This is the information that redisplay needs
in order to display this run.

   Extent fragments have to be very quick to update to a new buffer
position when moving linearly through the buffer.  They rely on the
stack-of-extents code, which does the heavy-duty algorithmic work of
determining which extents overly a particular position.

\1f
File: internals.info,  Node: Faces,  Next: Glyphs,  Prev: Extents,  Up: Top

23 Faces
********

Not yet documented.

\1f
File: internals.info,  Node: Glyphs,  Next: Specifiers,  Prev: Faces,  Up: Top

24 Glyphs
*********

Glyphs are graphical elements that can be displayed in XEmacs buffers or
gutters. We use the term graphical element here in the broadest possible
sense since glyphs can be as mundane as text or as arcane as a native
tab widget.

   In XEmacs, glyphs represent the uninstantiated state of graphical
elements, i.e. they hold all the information necessary to produce an
image on-screen but the image need not exist at this stage, and multiple
screen images can be instantiated from a single glyph.

   Glyphs are lazily instantiated by calling one of the glyph
functions. This usually occurs within redisplay when `Fglyph_height' is
called. Instantiation causes an image-instance to be created and
cached. This cache is on a per-device basis for all glyphs except
widget-glyphs, and on a per-window basis for widgets-glyphs.  The
caching is done by `image_instantiate' and is necessary because it is
generally possible to display an image-instance in multiple domains.
For instance if we create a Pixmap, we can actually display this on
multiple windows - even though we only need a single Pixmap instance to
do this. If caching wasn't done then it would be necessary to create
image-instances for every displayable occurrence of a glyph - and every
usage - and this would be extremely memory and cpu intensive.

   Widget-glyphs (a.k.a native widgets) are not cached in this way.
This is because widget-glyph image-instances on screen are toolkit
windows, and thus cannot be reused in multiple XEmacs domains. Thus
widget-glyphs are cached on an XEmacs window basis.

   Any action on a glyph first consults the cache before actually
instantiating a widget.

24.1 Glyph Instantiation
========================

Glyph instantiation is a hairy topic and requires some explanation. The
guts of glyph instantiation is contained within `image_instantiate'. A
glyph contains an image which is a specifier. When a glyph function -
for instance `Fglyph_height' - asks for a property of the glyph that
can only be determined from its instantiated state, then the glyph
image is instantiated and an image instance created. The instantiation
process is governed by the specifier code and goes through a series of
steps:

   * Validation. Instantiation of image instances happens dynamically -
     often within the guts of redisplay. Thus it is often not feasible
     to catch instantiator errors at instantiation time. Instead the
     instantiator is validated at the time it is added to the image
     specifier. This function is defined by `image_validate' and at a
     simple level validates keyword value pairs.

   * Duplication. The specifier code by default takes a copy of the
     instantiator. This is reasonable for most specifiers but in the
     case of widget-glyphs can be problematic, since some of the
     properties in the instantiator - for instance callbacks - could
     cause infinite recursion in the copying process. Thus the image
     code defines a function - `image_copy_instantiator' - which will
     selectively copy values.  This is controlled by the way that a
     keyword is defined either using `IIFORMAT_VALID_KEYWORD' or
     `IIFORMAT_VALID_NONCOPY_KEYWORD'. Note that the image caching and
     redisplay code relies on instantiator copying to ensure that
     current and new instantiators are actually different rather than
     referring to the same thing.

   * Normalization. Once the instantiator has been copied it must be
     converted into a form that is viable at instantiation time. This
     can involve no changes at all, but typically involves things like
     converting file names to the actual data. This function is defined
     by `image_going_to_add' and `normalize_image_instantiator'.

   * Instantiation. When an image instance is actually required for
     display it is instantiated using `image_instantiate'. This
     involves calling instantiate methods that are specific to the type
     of image being instantiated.

   The final instantiation phase also involves a number of steps. In
order to understand these we need to describe a number of concepts.

   An image is instantiated in a "domain", where a domain can be any
one of a device, frame, window or image-instance. The domain gives the
image-instance context and identity and properties that affect the
appearance of the image-instance may be different for the same glyph
instantiated in different domains. An example is the face used to
display the image-instance.

   Although an image is instantiated in a particular domain the
instantiation domain is not necessarily the domain in which the
image-instance is cached. For example a pixmap can be instantiated in a
window be actually be cached on a per-device basis. The domain in which
the image-instance is actually cached is called the "governing-domain".
A governing-domain is currently either a device or a window.
Widget-glyphs and text-glyphs have a window as a governing-domain, all
other image-instances have a device as the governing-domain. The
governing domain for an image-instance is determined using the
governing_domain image-instance method.

24.2 Widget-Glyphs
==================

24.3 Widget-Glyphs in the MS-Windows Environment
================================================

To Do

24.4 Widget-Glyphs in the X Environment
=======================================

Widget-glyphs under X make heavy use of lwlib (*note Lucid Widget
Library::) for manipulating the native toolkit objects. This is
primarily so that different toolkits can be supported for
widget-glyphs, just as they are supported for features such as menubars
etc.

   Lwlib is extremely poorly documented and quite hairy so here is my
understanding of what goes on.

   Lwlib maintains a set of widget_instances which mirror the
hierarchical state of Xt widgets. I think this is so that widgets can
be updated and manipulated generically by the lwlib library. For
instance update_one_widget_instance can cope with multiple types of
widget and multiple types of toolkit. Each element in the widget
hierarchy is updated from its corresponding widget_instance by walking
the widget_instance tree recursively.

   This has desirable properties such as lw_modify_all_widgets which is
called from `glyphs-x.c' and updates all the properties of a widget
without having to know what the widget is or what toolkit it is from.
Unfortunately this also has hairy properties such as making the lwlib
code quite complex. And of course lwlib has to know at some level what
the widget is and how to set its properties.

\1f
File: internals.info,  Node: Specifiers,  Next: Menus,  Prev: Glyphs,  Up: Top

25 Specifiers
*************

Not yet documented.

\1f
File: internals.info,  Node: Menus,  Next: Subprocesses,  Prev: Specifiers,  Up: Top

26 Menus
********

A menu is set by setting the value of the variable `current-menubar'
(which may be buffer-local) and then calling `set-menubar-dirty-flag'
to signal a change.  This will cause the menu to be redrawn at the next
redisplay.  The format of the data in `current-menubar' is described in
`menubar.c'.

   Internally the data in current-menubar is parsed into a tree of
`widget_value's' (defined in `lwlib.h'); this is accomplished by the
recursive function `menu_item_descriptor_to_widget_value()', called by
`compute_menubar_data()'.  Such a tree is deallocated using
`free_widget_value()'.

   `update_screen_menubars()' is one of the external entry points.
This checks to see, for each screen, if that screen's menubar needs to
be updated.  This is the case if

  1. `set-menubar-dirty-flag' was called since the last redisplay.
     (This function sets the C variable menubar_has_changed.)

  2. The buffer displayed in the screen has changed.

  3. The screen has no menubar currently displayed.

   `set_screen_menubar()' is called for each such screen.  This
function calls `compute_menubar_data()' to create the tree of
widget_value's, then calls `lw_create_widget()',
`lw_modify_all_widgets()', and/or `lw_destroy_all_widgets()' to create
the X-Toolkit widget associated with the menu.

   `update_psheets()', the other external entry point, actually changes
the menus being displayed.  It uses the widgets fixed by
`update_screen_menubars()' and calls various X functions to ensure that
the menus are displayed properly.

   The menubar widget is set up so that `pre_activate_callback()' is
called when the menu is first selected (i.e. mouse button goes down),
and `menubar_selection_callback()' is called when an item is selected.
`pre_activate_callback()' calls the function in activate-menubar-hook,
which can change the menubar (this is described in `menubar.c').  If
the menubar is changed, `set_screen_menubars()' is called.
`menubar_selection_callback()' enqueues a menu event, putting in it a
function to call (either `eval' or `call-interactively') and its
argument, which is the callback function or form given in the menu's
description.

\1f
File: internals.info,  Node: Subprocesses,  Next: Interface to the X Window System,  Prev: Menus,  Up: Top

27 Subprocesses
***************

The fields of a process are:

`name'
     A string, the name of the process.

`command'
     A list containing the command arguments that were used to start
     this process.

`filter'
     A function used to accept output from the process instead of a
     buffer, or `nil'.

`sentinel'
     A function called whenever the process receives a signal, or `nil'.

`buffer'
     The associated buffer of the process.

`pid'
     An integer, the Unix process ID.

`childp'
     A flag, non-`nil' if this is really a child process.  It is `nil'
     for a network connection.

`mark'
     A marker indicating the position of the end of the last output
     from this process inserted into the buffer.  This is often but not
     always the end of the buffer.

`kill_without_query'
     If this is non-`nil', killing XEmacs while this process is still
     running does not ask for confirmation about killing the process.

`raw_status_low'
`raw_status_high'
     These two fields record 16 bits each of the process status
     returned by the `wait' system call.

`status'
     The process status, as `process-status' should return it.

`tick'
`update_tick'
     If these two fields are not equal, a change in the status of the
     process needs to be reported, either by running the sentinel or by
     inserting a message in the process buffer.

`pty_flag'
     Non-`nil' if communication with the subprocess uses a PTY; `nil'
     if it uses a pipe.

`infd'
     The file descriptor for input from the process.

`outfd'
     The file descriptor for output to the process.

`subtty'
     The file descriptor for the terminal that the subprocess is using.
     (On some systems, there is no need to record this, so the value is
     `-1'.)

`tty_name'
     The name of the terminal that the subprocess is using, or `nil' if
     it is using pipes.

\1f
File: internals.info,  Node: Interface to the X Window System,  Next: Index,  Prev: Subprocesses,  Up: Top

28 Interface to the X Window System
***********************************

Mostly undocumented.

* Menu:

* Lucid Widget Library::        An interface to various widget sets.

\1f
File: internals.info,  Node: Lucid Widget Library,  Up: Interface to the X Window System

28.1 Lucid Widget Library
=========================

Lwlib is extremely poorly documented and quite hairy.  The author(s)
blame that on X, Xt, and Motif, with some justice, but also sufficient
hypocrisy to avoid drawing the obvious conclusion about their own work.

   The Lucid Widget Library is composed of two more or less independent
pieces.  The first, as the name suggests, is a set of widgets.  These
widgets are intended to resemble and improve on widgets provided in the
Motif toolkit but not in the Athena widgets, including menubars and
scrollbars.  Recent additions by Andy Piper integrate some "modern"
widgets by Edward Falk, including checkboxes, radio buttons, progress
gauges, and index tab controls (aka notebooks).

   The second piece of the Lucid widget library is a generic interface
to several toolkits for X (including Xt, the Athena widget set, and
Motif, as well as the Lucid widgets themselves) so that core XEmacs
code need not know which widget set has been used to build the
graphical user interface.

* Menu:

* Generic Widget Interface::    The lwlib generic widget interface.
* Scrollbars::
* Menubars::
* Checkboxes and Radio Buttons::
* Progress Bars::
* Tab Controls::

\1f
File: internals.info,  Node: Generic Widget Interface,  Next: Scrollbars,  Up: Lucid Widget Library

28.1.1 Generic Widget Interface
-------------------------------

In general in any toolkit a widget may be a composite object.  In Xt,
all widgets have an X window that they manage, but typically a complex
widget will have widget children, each of which manages a subwindow of
the parent widget's X window.  These children may themselves be
composite widgets.  Thus a widget is actually a tree or hierarchy of
widgets.

   For each toolkit widget, lwlib maintains a tree of `widget_values'
which mirror the hierarchical state of Xt widgets (including Motif,
Athena, 3D Athena, and Falk's widget sets).  Each `widget_value' has
`contents' member, which points to the head of a linked list of its
children.  The linked list of siblings is chained through the `next'
member of `widget_value'.

                +-----------+
                | composite |
                +-----------+
                      |
                      | contents
                      V
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     |
                                     | contents
                                     V
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   The `widget_instance' structure maintains the inverse view of the
tree.  As for the `widget_value', siblings are chained through the
`next' member.  However, rather than naming children, the
`widget_instance' tree links to parents.

                +-----------+
                | composite |
                +-----------+
                      A
                      | parent
                      |
                  +-------+ next +-------+ next +-------+
                  | child |----->| child |----->| child |
                  +-------+      +-------+      +-------+
                                     A
                                     | parent
                                     |
                              +-------------+ next +-------------+
                              | grand child |----->| grand child |
                              +-------------+      +-------------+

     The `widget_value' hierarchy of a composite widget with two simple
     children and one composite child.

   This permits widgets derived from different toolkits to be updated
and manipulated generically by the lwlib library. For instance
`update_one_widget_instance' can cope with multiple types of widget and
multiple types of toolkit. Each element in the widget hierarchy is
updated from its corresponding `widget_value' by walking the
`widget_value' tree.  This has desirable properties.  For example,
`lw_modify_all_widgets' is called from `glyphs-x.c' and updates all the
properties of a widget without having to know what the widget is or
what toolkit it is from.  Unfortunately this also has its hairy
properties; the lwlib code quite complex. And of course lwlib has to
know at some level what the widget is and how to set its properties.

   The `widget_instance' structure also contains a pointer to the root
of its tree.  Widget instances are further confi

\1f
File: internals.info,  Node: Scrollbars,  Next: Menubars,  Prev: Generic Widget Interface,  Up: Lucid Widget Library

28.1.2 Scrollbars
-----------------

\1f
File: internals.info,  Node: Menubars,  Next: Checkboxes and Radio Buttons,  Prev: Scrollbars,  Up: Lucid Widget Library

28.1.3 Menubars
---------------

\1f
File: internals.info,  Node: Checkboxes and Radio Buttons,  Next: Progress Bars,  Prev: Menubars,  Up: Lucid Widget Library

28.1.4 Checkboxes and Radio Buttons
-----------------------------------

\1f
File: internals.info,  Node: Progress Bars,  Next: Tab Controls,  Prev: Checkboxes and Radio Buttons,  Up: Lucid Widget Library

28.1.5 Progress Bars
--------------------

\1f
File: internals.info,  Node: Tab Controls,  Prev: Progress Bars,  Up: Lucid Widget Library

28.1.6 Tab Controls
-------------------

\1f
File: internals.info,  Node: Index,  Prev: Interface to the X Window System,  Up: Top

Index
*****

\0\b[index\0\b]
* Menu:

* allocation from frob blocks:           Allocation from Frob Blocks.
                                                              (line   6)
* allocation of objects in XEmacs Lisp:  Allocation of Objects in XEmacs Lisp.
                                                              (line   6)
* allocation, introduction to:           Introduction to Allocation.
                                                              (line   6)
* allocation, low-level:                 Low-level allocation.
                                                              (line   6)
* Amdahl Corporation:                    XEmacs.              (line   6)
* Andreessen, Marc:                      XEmacs.              (line   6)
* asynchronous subprocesses:             Modules for Interfacing with the Operating System.
                                                              (line  18)
* bars, progress:                        Progress Bars.       (line   6)
* Baur, Steve:                           XEmacs.              (line   6)
* Benson, Eric:                          Lucid Emacs.         (line  28)
* binding; the specbinding stack; unwind-protects, dynamic: Dynamic Binding; The specbinding Stack; Unwind-Protects.
                                                              (line   6)
* bindings, evaluation; stack frames;:   Evaluation; Stack Frames; Bindings.
                                                              (line   6)
* bit vector:                            Bit Vector.          (line   6)
* bridge, playing:                       XEmacs From the Outside.
                                                              (line  34)
* Buchholz, Martin:                      XEmacs.              (line   6)
* Bufbyte:                               Character-Related Data Types.
                                                              (line  24)
* Bufbytes and Emchars:                  Bufbytes and Emchars.
                                                              (line   6)
* buffer lists:                          Buffer Lists.        (line   6)
* buffer object, the:                    The Buffer Object.   (line   6)
* buffer, the text in a:                 The Text in a Buffer.
                                                              (line   6)
* buffers and textual representation:    Buffers and Textual Representation.
                                                              (line   6)
* buffers, introduction to:              Introduction to Buffers.
                                                              (line   6)
* Bufpos:                                Character-Related Data Types.
                                                              (line  47)
* building, XEmacs from the perspective of: XEmacs From the Perspective of Building.
                                                              (line   6)
* buttons, checkboxes and radio:         Checkboxes and Radio Buttons.
                                                              (line   6)
* byte positions, working with character and: Working With Character and Byte Positions.
                                                              (line   6)
* Bytecount:                             Character-Related Data Types.
                                                              (line  59)
* bytecount_to_charcount:                Working With Character and Byte Positions.
                                                              (line  82)
* Bytind:                                Character-Related Data Types.
                                                              (line  59)
* C code, rules when writing new:        Rules When Writing New C Code.
                                                              (line   6)
* C vs. Lisp:                            The Lisp Language.   (line   6)
* callback routines, the event stream:   The Event Stream Callback Routines.
                                                              (line   6)
* caller-protects (GCPRO rule):          Writing Lisp Primitives.
                                                              (line 169)
* case table:                            Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line  44)
* catch and throw:                       Catch and Throw.     (line   6)
* CCL:                                   CCL.                 (line   6)
* character and byte positions, working with: Working With Character and Byte Positions.
                                                              (line   6)
* character encoding, internal:          Internal Character Encoding.
                                                              (line   6)
* character sets:                        Character Sets.      (line   6)
* character sets and encodings, Mule:    MULE Character Sets and Encodings.
                                                              (line   6)
* character-related data types:          Character-Related Data Types.
                                                              (line   6)
* characters, integers and:              Integers and Characters.
                                                              (line   6)
* Charcount:                             Character-Related Data Types.
                                                              (line  47)
* charcount_to_bytecount:                Working With Character and Byte Positions.
                                                              (line  88)
* charptr_emchar:                        Working With Character and Byte Positions.
                                                              (line  36)
* charptr_n_addr:                        Working With Character and Byte Positions.
                                                              (line  94)
* checkboxes and radio buttons:          Checkboxes and Radio Buttons.
                                                              (line   6)
* closer:                                Lstream Methods.     (line  53)
* closure:                               The XEmacs Object System (Abstractly Speaking).
                                                              (line  71)
* code, an example of Mule-aware:        An Example of Mule-Aware Code.
                                                              (line   6)
* code, general guidelines for writing Mule-aware: General Guidelines for Writing Mule-Aware Code.
                                                              (line   6)
* code, rules when writing new C:        Rules When Writing New C Code.
                                                              (line   6)
* coding conventions:                    A Reader's Guide to XEmacs Coding Conventions.
                                                              (line   6)
* coding for Mule:                       Coding for Mule.     (line   6)
* coding rules, general:                 General Coding Rules.
                                                              (line   6)
* coding rules, naming:                  A Reader's Guide to XEmacs Coding Conventions.
                                                              (line   6)
* command builder, dispatching events; the: Dispatching Events; The Command Builder.
                                                              (line   6)
* comments, writing good:                Writing Good Comments.
                                                              (line   6)
* Common Lisp:                           The Lisp Language.   (line   6)
* compact_string_chars:                  compact_string_chars.
                                                              (line   6)
* compiled function:                     Compiled Function.   (line   6)
* compiler, the Lisp reader and:         The Lisp Reader and Compiler.
                                                              (line   6)
* cons:                                  Cons.                (line   6)
* conservative garbage collection:       GCPROing.            (line 131)
* consoles; devices; frames; windows:    Consoles; Devices; Frames; Windows.
                                                              (line   6)
* consoles; devices; frames; windows, introduction to: Introduction to Consoles; Devices; Frames; Windows.
                                                              (line   6)
* control flow modules, editor-level:    Editor-Level Control Flow Modules.
                                                              (line   6)
* conversion to and from external data:  Conversion to and from External Data.
                                                              (line   6)
* converting events:                     Converting Events.   (line   6)
* copy-on-write:                         General Coding Rules.
                                                              (line  57)
* creating Lisp object types:            Techniques for XEmacs Developers.
                                                              (line 192)
* critical redisplay sections:           Critical Redisplay Sections.
                                                              (line   6)
* data dumping:                          Data dumping.        (line   6)
* data types, character-related:         Character-Related Data Types.
                                                              (line   6)
* DEC_CHARPTR:                           Working With Character and Byte Positions.
                                                              (line  72)
* developers, techniques for XEmacs:     Techniques for XEmacs Developers.
                                                              (line   6)
* devices; frames; windows, consoles;:   Consoles; Devices; Frames; Windows.
                                                              (line   6)
* devices; frames; windows, introduction to consoles;: Introduction to Consoles; Devices; Frames; Windows.
                                                              (line   6)
* Devin, Matthieu:                       Lucid Emacs.         (line  28)
* dispatching events; the command builder: Dispatching Events; The Command Builder.
                                                              (line   6)
* display order of extents:              Mathematics of Extent Ordering.
                                                              (line   6)
* display-related Lisp objects, modules for other: Modules for other Display-Related Lisp Objects.
                                                              (line   6)
* displayable Lisp objects, modules for the basic: Modules for the Basic Displayable Lisp Objects.
                                                              (line   6)
* dumping:                               Dumping.             (line   6)
* dumping address allocation:            Address allocation.  (line   6)
* dumping and its justification, what is: Dumping.            (line   9)
* dumping data descriptions:             Data descriptions.   (line   6)
* dumping object inventory:              Object inventory.    (line   6)
* dumping overview:                      Overview.            (line   6)
* dumping phase:                         Dumping phase.       (line   6)
* dumping, data:                         Data dumping.        (line   6)
* dumping, file loading:                 Reloading phase.     (line   9)
* dumping, object relocation:            Reloading phase.     (line  32)
* dumping, pointers:                     Pointers dumping.    (line   6)
* dumping, putting back the pdump_opaques: Reloading phase.   (line  21)
* dumping, putting back the pdump_root_objects and pdump_weak_object_chains: Reloading phase.
                                                              (line  39)
* dumping, putting back the pdump_root_struct_ptrs: Reloading phase.
                                                              (line  26)
* dumping, reloading phase:              Reloading phase.     (line   6)
* dumping, remaining issues:             Remaining issues.    (line   6)
* dumping, reorganize the hash tables:   Reloading phase.     (line  44)
* dumping, the header:                   The header.          (line   6)
* dynamic array:                         Low-Level Modules.   (line 146)
* dynamic binding; the specbinding stack; unwind-protects: Dynamic Binding; The specbinding Stack; Unwind-Protects.
                                                              (line   6)
* dynamic scoping:                       The Lisp Language.   (line   6)
* dynamic types:                         The Lisp Language.   (line   6)
* editing operations, modules for standard: Modules for Standard Editing Operations.
                                                              (line   6)
* Emacs 19, GNU:                         GNU Emacs 19.        (line   6)
* Emacs 20, GNU:                         GNU Emacs 20.        (line   6)
* Emacs, a history of:                   A History of Emacs.  (line   6)
* Emchar:                                Character-Related Data Types.
                                                              (line  13)
* Emchars, Bufbytes and:                 Bufbytes and Emchars.
                                                              (line   6)
* encoding, internal character:          Internal Character Encoding.
                                                              (line   6)
* encoding, internal string:             Internal String Encoding.
                                                              (line   6)
* encodings, internal Mule:              Internal Mule Encodings.
                                                              (line   6)
* encodings, Mule:                       Encodings.           (line   6)
* encodings, Mule character sets and:    MULE Character Sets and Encodings.
                                                              (line   6)
* Energize:                              Lucid Emacs.         (line   6)
* Epoch <1>:                             XEmacs.              (line   6)
* Epoch:                                 Lucid Emacs.         (line   6)
* error checking:                        Techniques for XEmacs Developers.
                                                              (line  15)
* EUC (Extended Unix Code), Japanese:    Japanese EUC (Extended Unix Code).
                                                              (line   6)
* evaluation:                            Evaluation.          (line   6)
* evaluation; stack frames; bindings:    Evaluation; Stack Frames; Bindings.
                                                              (line   6)
* event gathering mechanism, specifics of the: Specifics of the Event Gathering Mechanism.
                                                              (line   6)
* event loop functions, other:           Other Event Loop Functions.
                                                              (line   6)
* event loop, events and the:            Events and the Event Loop.
                                                              (line   6)
* event stream callback routines, the:   The Event Stream Callback Routines.
                                                              (line   6)
* event, specifics about the Lisp object: Specifics About the Emacs Event.
                                                              (line   6)
* events and the event loop:             Events and the Event Loop.
                                                              (line   6)
* events, converting:                    Converting Events.   (line   6)
* events, introduction to:               Introduction to Events.
                                                              (line   6)
* events, main loop:                     Main Loop.           (line   6)
* events; the command builder, dispatching: Dispatching Events; The Command Builder.
                                                              (line   6)
* Extbyte:                               Character-Related Data Types.
                                                              (line  66)
* Extcount:                              Character-Related Data Types.
                                                              (line  66)
* Extended Unix Code, Japanese EUC:      Japanese EUC (Extended Unix Code).
                                                              (line   6)
* extent fragments:                      Extent Fragments.    (line   6)
* extent info, format of the:            Format of the Extent Info.
                                                              (line   6)
* extent mathematics:                    Mathematics of Extent Ordering.
                                                              (line   6)
* extent ordering <1>:                   Mathematics of Extent Ordering.
                                                              (line   6)
* extent ordering:                       Extent Ordering.     (line   6)
* extents:                               Extents.             (line   6)
* extents, display order:                Mathematics of Extent Ordering.
                                                              (line   6)
* extents, introduction to:              Introduction to Extents.
                                                              (line   6)
* extents, markers and:                  Markers and Extents. (line   6)
* extents, zero-length:                  Zero-Length Extents. (line   6)
* external data, conversion to and from: Conversion to and from External Data.
                                                              (line   6)
* external widget:                       Modules for Interfacing with X Windows.
                                                              (line  93)
* faces:                                 Faces.               (line   6)
* file system, modules for interfacing with the: Modules for Interfacing with the File System.
                                                              (line   6)
* flusher:                               Lstream Methods.     (line  47)
* fragments, extent:                     Extent Fragments.    (line   6)
* frames; windows, consoles; devices;:   Consoles; Devices; Frames; Windows.
                                                              (line   6)
* frames; windows, introduction to consoles; devices;: Introduction to Consoles; Devices; Frames; Windows.
                                                              (line   6)
* Free Software Foundation:              A History of Emacs.  (line   6)
* frob blocks, allocation from:          Allocation from Frob Blocks.
                                                              (line   6)
* FSF:                                   A History of Emacs.  (line   6)
* FSF Emacs <1>:                         GNU Emacs 20.        (line   6)
* FSF Emacs:                             GNU Emacs 19.        (line   6)
* function, compiled:                    Compiled Function.   (line   6)
* garbage collection:                    Garbage Collection.  (line   6)
* garbage collection - step by step:     Garbage Collection - Step by Step.
                                                              (line   6)
* garbage collection protection <1>:     GCPROing.            (line   6)
* garbage collection protection:         Writing Lisp Primitives.
                                                              (line  15)
* garbage collection, conservative:      GCPROing.            (line 131)
* garbage collection, invocation:        Invocation.          (line   6)
* garbage_collect_1:                     garbage_collect_1.   (line   6)
* gc_sweep:                              gc_sweep.            (line   6)
* GCPROing:                              GCPROing.            (line   6)
* global Lisp variables, adding:         Adding Global Lisp Variables.
                                                              (line   6)
* glyph instantiation:                   Glyphs.              (line  40)
* glyphs:                                Glyphs.              (line   6)
* GNU Emacs 19:                          GNU Emacs 19.        (line   6)
* GNU Emacs 20:                          GNU Emacs 20.        (line   6)
* Gosling, James <1>:                    The Lisp Language.   (line   6)
* Gosling, James:                        Through Version 18.  (line   6)
* Great Usenet Renaming:                 Through Version 18.  (line   6)
* Hackers (Steven Levy):                 A History of Emacs.  (line   6)
* header files, inline functions:        Techniques for XEmacs Developers.
                                                              (line 146)
* hierarchy of windows:                  Window Hierarchy.    (line   6)
* history of Emacs, a:                   A History of Emacs.  (line   6)
* Illinois, University of:               XEmacs.              (line   6)
* INC_CHARPTR:                           Working With Character and Byte Positions.
                                                              (line  72)
* inline functions:                      Techniques for XEmacs Developers.
                                                              (line 108)
* inline functions, headers:             Techniques for XEmacs Developers.
                                                              (line 146)
* inside, XEmacs from the:               XEmacs From the Inside.
                                                              (line   6)
* instantiation, glyph:                  Glyphs.              (line  40)
* integers and characters:               Integers and Characters.
                                                              (line   6)
* interactive:                           Modules for Standard Editing Operations.
                                                              (line  93)
* interfacing with the file system, modules for: Modules for Interfacing with the File System.
                                                              (line   6)
* interfacing with the operating system, modules for: Modules for Interfacing with the Operating System.
                                                              (line   6)
* interfacing with X Windows, modules for: Modules for Interfacing with X Windows.
                                                              (line   6)
* internal character encoding:           Internal Character Encoding.
                                                              (line   6)
* internal Mule encodings:               Internal Mule Encodings.
                                                              (line   6)
* internal string encoding:              Internal String Encoding.
                                                              (line   6)
* internationalization, modules for:     Modules for Internationalization.
                                                              (line   6)
* interning:                             The XEmacs Object System (Abstractly Speaking).
                                                              (line 302)
* interpreter and object system, modules for other aspects of the Lisp: Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line   6)
* ITS (Incompatible Timesharing System): A History of Emacs.  (line   6)
* Japanese EUC (Extended Unix Code):     Japanese EUC (Extended Unix Code).
                                                              (line   6)
* Java:                                  The Lisp Language.   (line   6)
* Java vs. Lisp:                         The Lisp Language.   (line   6)
* JIS7:                                  JIS7.                (line   6)
* Jones, Kyle:                           XEmacs.              (line  45)
* Kaplan, Simon:                         XEmacs.              (line   6)
* Levy, Steven:                          A History of Emacs.  (line   6)
* library, Lucid Widget:                 Lucid Widget Library.
                                                              (line   6)
* line start cache:                      Line Start Cache.    (line   6)
* Lisp interpreter and object system, modules for other aspects of the: Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line   6)
* Lisp language, the:                    The Lisp Language.   (line   6)
* Lisp modules, basic:                   Basic Lisp Modules.  (line   6)
* Lisp object types, creating:           Techniques for XEmacs Developers.
                                                              (line 192)
* Lisp objects are represented in C, how: How Lisp Objects Are Represented in C.
                                                              (line   6)
* Lisp objects, allocation of in XEmacs: Allocation of Objects in XEmacs Lisp.
                                                              (line   6)
* Lisp objects, modules for other display-related: Modules for other Display-Related Lisp Objects.
                                                              (line   6)
* Lisp objects, modules for the basic displayable: Modules for the Basic Displayable Lisp Objects.
                                                              (line   6)
* Lisp primitives, writing:              Writing Lisp Primitives.
                                                              (line   6)
* Lisp reader and compiler, the:         The Lisp Reader and Compiler.
                                                              (line   6)
* Lisp vs. C:                            The Lisp Language.   (line   6)
* Lisp vs. Java:                         The Lisp Language.   (line   6)
* low-level allocation:                  Low-level allocation.
                                                              (line   6)
* low-level modules:                     Low-Level Modules.   (line   6)
* lrecords:                              lrecords.            (line   6)
* lstream:                               Modules for Interfacing with the File System.
                                                              (line  20)
* lstream functions:                     Lstream Functions.   (line   6)
* lstream methods:                       Lstream Methods.     (line   6)
* lstream types:                         Lstream Types.       (line   6)
* lstream, creating an:                  Creating an Lstream. (line   6)
* Lstream_close:                         Lstream Functions.   (line  68)
* Lstream_fgetc:                         Lstream Functions.   (line  45)
* Lstream_flush:                         Lstream Functions.   (line  19)
* Lstream_fputc:                         Lstream Functions.   (line  44)
* Lstream_fungetc:                       Lstream Functions.   (line  46)
* Lstream_getc:                          Lstream Functions.   (line  29)
* Lstream_new:                           Lstream Functions.   (line   8)
* Lstream_putc:                          Lstream Functions.   (line  23)
* Lstream_read:                          Lstream Functions.   (line  50)
* Lstream_reopen:                        Lstream Functions.   (line  71)
* Lstream_rewind:                        Lstream Functions.   (line  77)
* Lstream_set_buffering:                 Lstream Functions.   (line  15)
* Lstream_ungetc:                        Lstream Functions.   (line  34)
* Lstream_unread:                        Lstream Functions.   (line  62)
* Lstream_write:                         Lstream Functions.   (line  56)
* lstreams:                              Lstreams.            (line   6)
* Lucid Emacs:                           Lucid Emacs.         (line   6)
* Lucid Inc.:                            Lucid Emacs.         (line   6)
* Lucid Widget Library:                  Lucid Widget Library.
                                                              (line   6)
* macro hygiene:                         Techniques for XEmacs Developers.
                                                              (line  71)
* main loop:                             Main Loop.           (line   6)
* mark and sweep:                        Garbage Collection.  (line   6)
* mark method <1>:                       lrecords.            (line 107)
* mark method:                           Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line 132)
* mark_object:                           mark_object.         (line   6)
* marker <1>:                            Lstream Methods.     (line  61)
* marker:                                Marker.              (line   6)
* markers and extents:                   Markers and Extents. (line   6)
* mathematics of extent ordering:        Mathematics of Extent Ordering.
                                                              (line   6)
* MAX_EMCHAR_LEN:                        Working With Character and Byte Positions.
                                                              (line  14)
* menubars:                              Menubars.            (line   6)
* menus:                                 Menus.               (line   6)
* merging attempts:                      XEmacs.              (line  52)
* MIT:                                   A History of Emacs.  (line   6)
* Mlynarik, Richard:                     GNU Emacs 19.        (line  68)
* modules for interfacing with the file system: Modules for Interfacing with the File System.
                                                              (line   6)
* modules for interfacing with the operating system: Modules for Interfacing with the Operating System.
                                                              (line   6)
* modules for interfacing with X Windows: Modules for Interfacing with X Windows.
                                                              (line   6)
* modules for internationalization:      Modules for Internationalization.
                                                              (line   6)
* modules for other aspects of the Lisp interpreter and object system: Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line   6)
* modules for other display-related Lisp objects: Modules for other Display-Related Lisp Objects.
                                                              (line   6)
* modules for regression testing:        Modules for Regression Testing.
                                                              (line   6)
* modules for standard editing operations: Modules for Standard Editing Operations.
                                                              (line   6)
* modules for the basic displayable Lisp objects: Modules for the Basic Displayable Lisp Objects.
                                                              (line   6)
* modules for the redisplay mechanism:   Modules for the Redisplay Mechanism.
                                                              (line   6)
* modules, a summary of the various XEmacs: A Summary of the Various XEmacs Modules.
                                                              (line   6)
* modules, basic Lisp:                   Basic Lisp Modules.  (line   6)
* modules, editor-level control flow:    Editor-Level Control Flow Modules.
                                                              (line   6)
* modules, low-level:                    Low-Level Modules.   (line   6)
* MS-Windows environment, widget-glyphs in the: Glyphs.       (line 107)
* Mule character sets and encodings:     MULE Character Sets and Encodings.
                                                              (line   6)
* Mule encodings:                        Encodings.           (line   6)
* Mule encodings, internal:              Internal Mule Encodings.
                                                              (line   6)
* MULE merged XEmacs appears:            XEmacs.              (line  35)
* Mule, coding for:                      Coding for Mule.     (line   6)
* Mule-aware code, an example of:        An Example of Mule-Aware Code.
                                                              (line   6)
* Mule-aware code, general guidelines for writing: General Guidelines for Writing Mule-Aware Code.
                                                              (line   6)
* NAS:                                   Modules for Interfacing with the Operating System.
                                                              (line 135)
* native sound:                          Modules for Interfacing with the Operating System.
                                                              (line 131)
* network connections:                   Modules for Interfacing with the Operating System.
                                                              (line  37)
* network sound:                         Modules for Interfacing with the Operating System.
                                                              (line 135)
* Niksic, Hrvoje:                        XEmacs.              (line  45)
* obarrays:                              Obarrays.            (line   6)
* object system (abstractly speaking), the XEmacs: The XEmacs Object System (Abstractly Speaking).
                                                              (line   6)
* object system, modules for other aspects of the Lisp interpreter and: Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line   6)
* object types, creating Lisp:           Techniques for XEmacs Developers.
                                                              (line 192)
* object, the buffer:                    The Buffer Object.   (line   6)
* object, the window:                    The Window Object.   (line   6)
* objects are represented in C, how Lisp: How Lisp Objects Are Represented in C.
                                                              (line   6)
* objects in XEmacs Lisp, allocation of: Allocation of Objects in XEmacs Lisp.
                                                              (line   6)
* objects, modules for the basic displayable Lisp: Modules for the Basic Displayable Lisp Objects.
                                                              (line   6)
* operating system, modules for interfacing with the: Modules for Interfacing with the Operating System.
                                                              (line   6)
* outside, XEmacs from the:              XEmacs From the Outside.
                                                              (line   6)
* pane:                                  Modules for the Basic Displayable Lisp Objects.
                                                              (line  63)
* permanent objects:                     The XEmacs Object System (Abstractly Speaking).
                                                              (line 238)
* pi, calculating:                       XEmacs From the Outside.
                                                              (line  34)
* point:                                 Point.               (line   6)
* pointers dumping:                      Pointers dumping.    (line   6)
* positions, working with character and byte: Working With Character and Byte Positions.
                                                              (line   6)
* primitives, writing Lisp:              Writing Lisp Primitives.
                                                              (line   6)
* progress bars:                         Progress Bars.       (line   6)
* protection, garbage collection:        GCPROing.            (line   6)
* pseudo_closer:                         Lstream Methods.     (line  51)
* Purify:                                Techniques for XEmacs Developers.
                                                              (line   6)
* Quantify:                              Techniques for XEmacs Developers.
                                                              (line   6)
* radio buttons, checkboxes and:         Checkboxes and Radio Buttons.
                                                              (line   6)
* read syntax:                           The XEmacs Object System (Abstractly Speaking).
                                                              (line 252)
* read-eval-print:                       XEmacs From the Outside.
                                                              (line   6)
* reader:                                Lstream Methods.     (line   8)
* reader and compiler, the Lisp:         The Lisp Reader and Compiler.
                                                              (line   6)
* reader's guide:                        A Reader's Guide to XEmacs Coding Conventions.
                                                              (line   6)
* redisplay mechanism, modules for the:  Modules for the Redisplay Mechanism.
                                                              (line   6)
* redisplay mechanism, the:              The Redisplay Mechanism.
                                                              (line   6)
* redisplay piece by piece:              Redisplay Piece by Piece.
                                                              (line   6)
* redisplay sections, critical:          Critical Redisplay Sections.
                                                              (line   6)
* regression testing, modules for:       Modules for Regression Testing.
                                                              (line   6)
* reloading phase:                       Reloading phase.     (line   6)
* relocating allocator:                  Low-Level Modules.   (line 105)
* rename to XEmacs:                      XEmacs.              (line  21)
* represented in C, how Lisp objects are: How Lisp Objects Are Represented in C.
                                                              (line   6)
* rewinder:                              Lstream Methods.     (line  38)
* RMS:                                   A History of Emacs.  (line   6)
* scanner:                               Modules for Other Aspects of the Lisp Interpreter and Object System.
                                                              (line  52)
* scoping, dynamic:                      The Lisp Language.   (line   6)
* scrollbars:                            Scrollbars.          (line   6)
* seekable_p:                            Lstream Methods.     (line  41)
* selections:                            Modules for Interfacing with X Windows.
                                                              (line  55)
* set_charptr_emchar:                    Working With Character and Byte Positions.
                                                              (line  36)
* Sexton, Harlan:                        Lucid Emacs.         (line  28)
* sound, native:                         Modules for Interfacing with the Operating System.
                                                              (line 131)
* sound, network:                        Modules for Interfacing with the Operating System.
                                                              (line 135)
* SPARCWorks:                            XEmacs.              (line   6)
* specbinding stack; unwind-protects, dynamic binding; the: Dynamic Binding; The specbinding Stack; Unwind-Protects.
                                                              (line   6)
* special forms, simple:                 Simple Special Forms.
                                                              (line   6)
* specifiers:                            Specifiers.          (line   6)
* stack frames; bindings, evaluation;:   Evaluation; Stack Frames; Bindings.
                                                              (line   6)
* Stallman, Richard:                     A History of Emacs.  (line   6)
* string:                                String.              (line   6)
* string encoding, internal:             Internal String Encoding.
                                                              (line   6)
* subprocesses:                          Subprocesses.        (line   6)
* subprocesses, asynchronous:            Modules for Interfacing with the Operating System.
                                                              (line  18)
* subprocesses, synchronous:             Modules for Interfacing with the Operating System.
                                                              (line  13)
* Sun Microsystems:                      XEmacs.              (line   6)
* sweep_bit_vectors_1:                   sweep_bit_vectors_1. (line   6)
* sweep_lcrecords_1:                     sweep_lcrecords_1.   (line   6)
* sweep_strings:                         sweep_strings.       (line   6)
* symbol:                                Symbol.              (line   6)
* symbol values:                         Symbol Values.       (line   6)
* symbols and variables:                 Symbols and Variables.
                                                              (line   6)
* symbols, introduction to:              Introduction to Symbols.
                                                              (line   6)
* synchronous subprocesses:              Modules for Interfacing with the Operating System.
                                                              (line  13)
* tab controls:                          Tab Controls.        (line   6)
* taxes, doing:                          XEmacs From the Outside.
                                                              (line  34)
* techniques for XEmacs developers:      Techniques for XEmacs Developers.
                                                              (line   6)
* TECO:                                  A History of Emacs.  (line   6)
* temporary objects:                     The XEmacs Object System (Abstractly Speaking).
                                                              (line 238)
* testing, regression:                   Regression Testing XEmacs.
                                                              (line   6)
* text in a buffer, the:                 The Text in a Buffer.
                                                              (line   6)
* textual representation, buffers and:   Buffers and Textual Representation.
                                                              (line   6)
* Thompson, Chuck:                       XEmacs.              (line   6)
* throw, catch and:                      Catch and Throw.     (line   6)
* types, dynamic:                        The Lisp Language.   (line   6)
* types, lstream:                        Lstream Types.       (line   6)
* types, proper use of unsigned:         Proper Use of Unsigned Types.
                                                              (line   6)
* University of Illinois:                XEmacs.              (line   6)
* unsigned types, proper use of:         Proper Use of Unsigned Types.
                                                              (line   6)
* unwind-protects, dynamic binding; the specbinding stack;: Dynamic Binding; The specbinding Stack; Unwind-Protects.
                                                              (line   6)
* values, symbol:                        Symbol Values.       (line   6)
* variables, adding global Lisp:         Adding Global Lisp Variables.
                                                              (line   6)
* variables, symbols and:                Symbols and Variables.
                                                              (line   6)
* vector:                                Vector.              (line   6)
* vector, bit:                           Bit Vector.          (line   6)
* version 18, through:                   Through Version 18.  (line   6)
* version 19, GNU Emacs:                 GNU Emacs 19.        (line   6)
* version 20, GNU Emacs:                 GNU Emacs 20.        (line   6)
* widget interface, generic:             Generic Widget Interface.
                                                              (line   6)
* widget library, Lucid:                 Lucid Widget Library.
                                                              (line   6)
* widget-glyphs:                         Glyphs.              (line 104)
* widget-glyphs in the MS-Windows environment: Glyphs.        (line 107)
* widget-glyphs in the X environment:    Glyphs.              (line 112)
* Win-Emacs:                             XEmacs.              (line   6)
* window (in Emacs):                     Modules for the Basic Displayable Lisp Objects.
                                                              (line  63)
* window hierarchy:                      Window Hierarchy.    (line   6)
* window object, the:                    The Window Object.   (line   6)
* window point internals:                The Window Object.   (line  22)
* windows, consoles; devices; frames;:   Consoles; Devices; Frames; Windows.
                                                              (line   6)
* windows, introduction to consoles; devices; frames;: Introduction to Consoles; Devices; Frames; Windows.
                                                              (line   6)
* Wing, Ben:                             XEmacs.              (line   6)
* writer:                                Lstream Methods.     (line  26)
* writing good comments:                 Writing Good Comments.
                                                              (line   6)
* writing Lisp primitives:               Writing Lisp Primitives.
                                                              (line   6)
* writing Mule-aware code, general guidelines for: General Guidelines for Writing Mule-Aware Code.
                                                              (line   6)
* writing new C code, rules when:        Rules When Writing New C Code.
                                                              (line   6)
* X environment, widget-glyphs in the:   Glyphs.              (line 112)
* X Window System, interface to the:     Interface to the X Window System.
                                                              (line   6)
* X Windows, modules for interfacing with: Modules for Interfacing with X Windows.
                                                              (line   6)
* XEmacs:                                XEmacs.              (line   6)
* XEmacs from the inside:                XEmacs From the Inside.
                                                              (line   6)
* XEmacs from the outside:               XEmacs From the Outside.
                                                              (line   6)
* XEmacs from the perspective of building: XEmacs From the Perspective of Building.
                                                              (line   6)
* XEmacs goes it alone:                  XEmacs.              (line  45)
* XEmacs object system (abstractly speaking), the: The XEmacs Object System (Abstractly Speaking).
                                                              (line   6)
* Zawinski, Jamie:                       Lucid Emacs.         (line  28)
* zero-length extents:                   Zero-Length Extents. (line   6)