Contents in 1999-06-04-13 of release-21-2.

[chise/xemacs-chise.git.1] / man / internals / internals.texi

\input texinfo  @c -*-texinfo-*-
@c %**start of header
@setfilename ../../info/internals.info
@settitle XEmacs Internals Manual
@c %**end of header

@ifinfo
@dircategory XEmacs Editor
@direntry
* Internals: (internals).       XEmacs Internals Manual.
@end direntry

Copyright @copyright{} 1992 - 1996 Ben Wing.
Copyright @copyright{} 1996, 1997 Sun Microsystems.
Copyright @copyright{} 1994 - 1998 Free Software Foundation.
Copyright @copyright{} 1994, 1995 Board of Trustees, University of Illinois.


Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries copying permission notice
identical to this one except for the removal of this paragraph (this
paragraph not being relevant to the printed manual).

@end ignore
Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided that the
entire resulting derived work is distributed under the terms of a
permission notice identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions,
except that this permission notice may be stated in a translation
approved by the Foundation.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that the
section entitled ``GNU General Public License'' is included exactly as
in the original, and provided that the entire resulting derived work is
distributed under the terms of a permission notice identical to this
one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions,
except that the section entitled ``GNU General Public License'' may be
included in a translation approved by the Free Software Foundation
instead of in the original English.
@end ifinfo

@c Combine indices.
@synindex cp fn
@syncodeindex vr fn
@syncodeindex ky fn
@syncodeindex pg fn
@syncodeindex tp fn

@setchapternewpage odd
@finalout

@titlepage
@title XEmacs Internals Manual
@subtitle Version 1.2, October 1998

@author Ben Wing
@author Martin Buchholz
@author Hrvoje Niksic
@page
@vskip 0pt plus 1fill

@noindent
Copyright @copyright{} 1992 - 1996 Ben Wing. @*
Copyright @copyright{} 1996, 1997 Sun Microsystems, Inc. @*
Copyright @copyright{} 1994 - 1998 Free Software Foundation. @*
Copyright @copyright{} 1994, 1995 Board of Trustees, University of Illinois.

@sp 2
Version 1.2 @*
October 1998.@*

Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that the
section entitled ``GNU General Public License'' is included
exactly as in the original, and provided that the entire resulting
derived work is distributed under the terms of a permission notice
identical to this one.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions,
except that the section entitled ``GNU General Public License'' may be
included in a translation approved by the Free Software Foundation
instead of in the original English.
@end titlepage
@page

@node Top, A History of Emacs, (dir), (dir)

@ifinfo
This Info file contains v1.0 of the XEmacs Internals Manual.
@end ifinfo

@menu
* A History of Emacs::          Times, dates, important events.
* XEmacs From the Outside::     A broad conceptual overview.
* The Lisp Language::           An overview.
* XEmacs From the Perspective of Building::
* XEmacs From the Inside::
* The XEmacs Object System (Abstractly Speaking)::
* How Lisp Objects Are Represented in C::
* Rules When Writing New C Code::
* A Summary of the Various XEmacs Modules::
* Allocation of Objects in XEmacs Lisp::
* Events and the Event Loop::
* Evaluation; Stack Frames; Bindings::
* Symbols and Variables::
* Buffers and Textual Representation::
* MULE Character Sets and Encodings::
* The Lisp Reader and Compiler::
* Lstreams::
* Consoles; Devices; Frames; Windows::
* The Redisplay Mechanism::
* Extents::
* Faces and Glyphs::
* Specifiers::
* Menus::
* Subprocesses::
* Interface to X Windows::
* Index::                   Index including concepts, functions, variables,
                              and other terms.

      --- The Detailed Node Listing ---

Here are other nodes that are inferiors of those already listed,
mentioned here so you can get to them in one step:

A History of Emacs

* Through Version 18::          Unification prevails.
* Lucid Emacs::                 One version 19 Emacs.
* GNU Emacs 19::                The other version 19 Emacs.
* XEmacs::                      The continuation of Lucid Emacs.

Rules When Writing New C Code

* General Coding Rules::
* Writing Lisp Primitives::
* Adding Global Lisp Variables::
* Techniques for XEmacs Developers::

A Summary of the Various XEmacs Modules

* Low-Level Modules::
* Basic Lisp Modules::
* Modules for Standard Editing Operations::
* Editor-Level Control Flow Modules::
* Modules for the Basic Displayable Lisp Objects::
* Modules for other Display-Related Lisp Objects::
* Modules for the Redisplay Mechanism::
* Modules for Interfacing with the File System::
* Modules for Other Aspects of the Lisp Interpreter and Object System::
* Modules for Interfacing with the Operating System::
* Modules for Interfacing with X Windows::
* Modules for Internationalization::

Allocation of Objects in XEmacs Lisp

* Introduction to Allocation::
* Garbage Collection::
* GCPROing::
* Integers and Characters::
* Allocation from Frob Blocks::
* lrecords::
* Low-level allocation::
* Pure Space::
* Cons::
* Vector::
* Bit Vector::
* Symbol::
* Marker::
* String::
* Compiled Function::

Events and the Event Loop

* Introduction to Events::
* Main Loop::
* Specifics of the Event Gathering Mechanism::
* Specifics About the Emacs Event::
* The Event Stream Callback Routines::
* Other Event Loop Functions::
* Converting Events::
* Dispatching Events; The Command Builder::

Evaluation; Stack Frames; Bindings

* Evaluation::
* Dynamic Binding; The specbinding Stack; Unwind-Protects::
* Simple Special Forms::
* Catch and Throw::

Symbols and Variables

* Introduction to Symbols::
* Obarrays::
* Symbol Values::

Buffers and Textual Representation

* Introduction to Buffers::     A buffer holds a block of text such as a file.
* The Text in a Buffer::        Representation of the text in a buffer.
* Buffer Lists::                Keeping track of all buffers.
* Markers and Extents::         Tagging locations within a buffer.
* Bufbytes and Emchars::        Representation of individual characters.
* The Buffer Object::           The Lisp object corresponding to a buffer.

MULE Character Sets and Encodings

* Character Sets::
* Encodings::
* Internal Mule Encodings::

Encodings

* Japanese EUC (Extended Unix Code)::
* JIS7::

Internal Mule Encodings

* Internal String Encoding::
* Internal Character Encoding::

The Lisp Reader and Compiler

Lstreams

Consoles; Devices; Frames; Windows

* Introduction to Consoles; Devices; Frames; Windows::
* Point::
* Window Hierarchy::

The Redisplay Mechanism

* Critical Redisplay Sections::
* Line Start Cache::

Extents

* Introduction to Extents::     Extents are ranges over text, with properties.
* Extent Ordering::             How extents are ordered internally.
* Format of the Extent Info::   The extent information in a buffer or string.
* Zero-Length Extents::         A weird special case.
* Mathematics of Extent Ordering::      A rigorous foundation.
* Extent Fragments::            Cached information useful for redisplay.

Faces and Glyphs

Specifiers

Menus

Subprocesses

Interface to X Windows

@end menu

@node A History of Emacs, XEmacs From the Outside, Top, Top
@chapter A History of Emacs
@cindex history of Emacs
@cindex Hackers (Steven Levy)
@cindex Levy, Steven
@cindex ITS (Incompatible Timesharing System)
@cindex Stallman, Richard
@cindex RMS
@cindex MIT
@cindex TECO
@cindex FSF
@cindex Free Software Foundation

  XEmacs is a powerful, customizable text editor and development
environment.  It began as Lucid Emacs, which was in turn derived from
GNU Emacs, a program written by Richard Stallman of the Free Software
Foundation.  GNU Emacs dates back to the 1970's, and was modelled
after a package called ``Emacs'', written in 1976, that was a set of
macros on top of TECO, an old, old text editor written at MIT on the
DEC PDP 10 under one of the earliest time-sharing operating systems,
ITS (Incompatible Timesharing System). (ITS dates back well before
Unix.) ITS, TECO, and Emacs were products of a group of people at MIT
who called themselves ``hackers'', who shared an idealistic belief
system about the free exchange of information and were fanatical in
their devotion to and time spent with computers. (The hacker
subculture dates back to the late 1950's at MIT and is described in
detail in Steven Levy's book @cite{Hackers}.  This book also includes
a lot of information about Stallman himself and the development of
Lisp, a programming language developed at MIT that underlies Emacs.)

@menu
* Through Version 18::          Unification prevails.
* Lucid Emacs::                 One version 19 Emacs.
* GNU Emacs 19::                The other version 19 Emacs.
* GNU Emacs 20::                The other version 20 Emacs.
* XEmacs::                      The continuation of Lucid Emacs.
@end menu

@node Through Version 18
@section Through Version 18
@cindex Gosling, James
@cindex Great Usenet Renaming

  Although the history of the early versions of GNU Emacs is unclear,
the history is well-known from the middle of 1985.  A time line is:

@itemize @bullet
@item
GNU Emacs version 15 (15.34) was released sometime in 1984 or 1985 and
shared some code with a version of Emacs written by James Gosling (the
same James Gosling who later created the Java language).
@item
GNU Emacs version 16 (first released version was 16.56) was released on
July 15, 1985.  All Gosling code was removed due to potential copyright
problems with the code.
@item
version 16.57: released on September 16, 1985.
@item
versions 16.58, 16.59: released on September 17, 1985.
@item
version 16.60: released on September 19, 1985.  These later version 16's
incorporated patches from the net, esp. for getting Emacs to work under
System V.
@item
version 17.36 (first official v17 release) released on December 20,
1985.  Included a TeX-able user manual.  First official unpatched
version that worked on vanilla System V machines.
@item
version 17.43 (second official v17 release) released on January 25,
1986.
@item
version 17.45 released on January 30, 1986.
@item
version 17.46 released on February 4, 1986.
@item
version 17.48 released on February 10, 1986.
@item
version 17.49 released on February 12, 1986.
@item
version 17.55 released on March 18, 1986.
@item
version 17.57 released on March 27, 1986.
@item
version 17.58 released on April 4, 1986.
@item
version 17.61 released on April 12, 1986.
@item
version 17.63 released on May 7, 1986.
@item
version 17.64 released on May 12, 1986.
@item
version 18.24 (a beta version) released on October 2, 1986.
@item
version 18.30 (a beta version) released on November 15, 1986.
@item
version 18.31 (a beta version) released on November 23, 1986.
@item
version 18.32 (a beta version) released on December 7, 1986.
@item
version 18.33 (a beta version) released on December 12, 1986.
@item
version 18.35 (a beta version) released on January 5, 1987.
@item
version 18.36 (a beta version) released on January 21, 1987.
@item
January 27, 1987: The Great Usenet Renaming.  net.emacs is now
comp.emacs.
@item
version 18.37 (a beta version) released on February 12, 1987.
@item
version 18.38 (a beta version) released on March 3, 1987.
@item
version 18.39 (a beta version) released on March 14, 1987.
@item
version 18.40 (a beta version) released on March 18, 1987.
@item
version 18.41 (the first ``official'' release) released on March 22,
1987.
@item
version 18.45 released on June 2, 1987.
@item
version 18.46 released on June 9, 1987.
@item
version 18.47 released on June 18, 1987.
@item
version 18.48 released on September 3, 1987.
@item
version 18.49 released on September 18, 1987.
@item
version 18.50 released on February 13, 1988.
@item
version 18.51 released on May 7, 1988.
@item
version 18.52 released on September 1, 1988.
@item
version 18.53 released on February 24, 1989.
@item
version 18.54 released on April 26, 1989.
@item
version 18.55 released on August 23, 1989.  This is the earliest version
that is still available by FTP.
@item
version 18.56 released on January 17, 1991.
@item
version 18.57 released late January, 1991.
@item
version 18.58 released ?????.
@item
version 18.59 released October 31, 1992.
@end itemize

@node Lucid Emacs
@section Lucid Emacs
@cindex Lucid Emacs
@cindex Lucid Inc.
@cindex Energize
@cindex Epoch

  Lucid Emacs was developed by the (now-defunct) Lucid Inc., a maker of
C++ and Lisp development environments.  It began when Lucid decided they
wanted to use Emacs as the editor and cornerstone of their C++
development environment (called ``Energize'').  They needed many features
that were not available in the existing version of GNU Emacs (version
18.5something), in particular good and integrated support for GUI
elements such as mouse support, multiple fonts, multiple window-system
windows, etc.  A branch of GNU Emacs called Epoch, written at the
University of Illinois, existed that supplied many of these features;
however, Lucid needed more than what existed in Epoch.  At the time, the
Free Software Foundation was working on version 19 of Emacs (this was
sometime around 1991), which was planned to have similar features, and
so Lucid decided to work with the Free Software Foundation.  Their plan
was to add features that they needed, and coordinate with the FSF so
that the features would get included back into Emacs version 19.

  Delays in the release of version 19 occurred, however (resulting in it
finally being released more than a year after what was initially
planned), and Lucid encountered unexpected technical resistance in
getting their changes merged back into version 19, so they decided to
release their own version of Emacs, which became Lucid Emacs 19.0.

@cindex Zawinski, Jamie
@cindex Sexton, Harlan
@cindex Benson, Eric
@cindex Devin, Matthieu
  The initial authors of Lucid Emacs were Matthieu Devin, Harlan Sexton,
and Eric Benson, and the work was later taken over by Jamie Zawinski,
who became ``Mr. Lucid Emacs'' for many releases.

  A time line for Lucid Emacs/XEmacs is

@itemize @bullet
@item
version 19.0 shipped with Energize 1.0, April 1992.
@item
version 19.1 released June 4, 1992.
@item
version 19.2 released June 19, 1992.
@item
version 19.3 released September 9, 1992.
@item
version 19.4 released January 21, 1993.
@item
version 19.5 was a repackaging of 19.4 with a few bug fixes and
shipped with Energize 2.0.  Never released to the net.
@item
version 19.6 released April 9, 1993.
@item
version 19.7 was a repackaging of 19.6 with a few bug fixes and
shipped with Energize 2.1.  Never released to the net.
@item
version 19.8 released September 6, 1993.
@item
version 19.9 released January 12, 1994.
@item
version 19.10 released May 27, 1994.
@item
version 19.11 (first XEmacs) released September 13, 1994.
@item
version 19.12 released June 23, 1995.
@item
version 19.13 released September 1, 1995.
@item
version 19.14 released June 23, 1996.
@item
version 20.0 released February 9, 1997.
@item
version 19.15 released March 28, 1997.
@item
version 20.1 (not released to the net) April 15, 1997.
@item
version 20.2 released May 16, 1997.
@item
version 19.16 released October 31, 1997.
@item
version 20.3 (the first stable version of XEmacs 20.x) released November 30,
1997.
version 20.4 released February 28, 1998.
@end itemize

@node GNU Emacs 19
@section GNU Emacs 19
@cindex GNU Emacs 19
@cindex FSF Emacs

  About a year after the initial release of Lucid Emacs, the FSF
released a beta of their version of Emacs 19 (referred to here as ``GNU
Emacs'').  By this time, the current version of Lucid Emacs was
19.6. (Strangely, the first released beta from the FSF was GNU Emacs
19.7.) A time line for GNU Emacs version 19 is

@itemize @bullet
@item
version 19.8 (beta) released May 27, 1993.
@item
version 19.9 (beta) released May 27, 1993.
@item
version 19.10 (beta) released May 30, 1993.
@item
version 19.11 (beta) released June 1, 1993.
@item
version 19.12 (beta) released June 2, 1993.
@item
version 19.13 (beta) released June 8, 1993.
@item
version 19.14 (beta) released June 17, 1993.
@item
version 19.15 (beta) released June 19, 1993.
@item
version 19.16 (beta) released July 6, 1993.
@item
version 19.17 (beta) released late July, 1993.
@item
version 19.18 (beta) released August 9, 1993.
@item
version 19.19 (beta) released August 15, 1993.
@item
version 19.20 (beta) released November 17, 1993.
@item
version 19.21 (beta) released November 17, 1993.
@item
version 19.22 (beta) released November 28, 1993.
@item
version 19.23 (beta) released May 17, 1994.
@item
version 19.24 (beta) released May 16, 1994.
@item
version 19.25 (beta) released June 3, 1994.
@item
version 19.26 (beta) released September 11, 1994.
@item
version 19.27 (beta) released September 14, 1994.
@item
version 19.28 (first ``official'' release) released November 1, 1994.
@item
version 19.29 released June 21, 1995.
@item
version 19.30 released November 24, 1995.
@item
version 19.31 released May 25, 1996.
@item
version 19.32 released July 31, 1996.
@item
version 19.33 released August 11, 1996.
@item
version 19.34 released August 21, 1996.
@item
version 19.34b released September 6, 1996.
@end itemize

@cindex Mlynarik, Richard
  In some ways, GNU Emacs 19 was better than Lucid Emacs; in some ways,
worse.  Lucid soon began incorporating features from GNU Emacs 19 into
Lucid Emacs; the work was mostly done by Richard Mlynarik, who had been
working on and using GNU Emacs for a long time (back as far as version
16 or 17).

@node GNU Emacs 20
@section GNU Emacs 20
@cindex GNU Emacs 20
@cindex FSF Emacs

On February 2, 1997 work began on GNU Emacs to integrate Mule.  The first
release was made in September of that year.

A timeline for Emacs 20 is

@itemize @bullet
@item
version 20.1 released September 17, 1997.
@item
version 20.2 released September 20, 1997.
@item
version 20.3 released August 19, 1998.
@end itemize

@node XEmacs
@section XEmacs
@cindex XEmacs

@cindex Sun Microsystems
@cindex University of Illinois
@cindex Illinois, University of
@cindex SPARCWorks
@cindex Andreessen, Marc
@cindex Baur, Steve
@cindex Buchholz, Martin
@cindex Kaplan, Simon
@cindex Wing, Ben
@cindex Thompson, Chuck
@cindex Win-Emacs
@cindex Epoch
@cindex Amdahl Corporation
  Around the time that Lucid was developing Energize, Sun Microsystems
was developing their own development environment (called ``SPARCWorks'')
and also decided to use Emacs.  They joined forces with the Epoch team
at the University of Illinois and later with Lucid.  The maintainer of
the last-released version of Epoch was Marc Andreessen, but he dropped
out and the Epoch project, headed by Simon Kaplan, lured Chuck Thompson
away from a system administration job to become the primary Lucid Emacs
author for Epoch and Sun.  Chuck's area of specialty became the
redisplay engine (he replaced the old Lucid Emacs redisplay engine with
a ported version from Epoch and then later rewrote it from scratch).
Sun also hired Ben Wing (the author of Win-Emacs, a port of Lucid Emacs
to Microsoft Windows 3.1) in 1993, for what was initially a one-month
contract to fix some event problems but later became a many-year
involvement, punctuated by a six-month contract with Amdahl Corporation.

@cindex rename to XEmacs
  In 1994, Sun and Lucid agreed to rename Lucid Emacs to XEmacs (a name
not favorable to either company); the first release called XEmacs was
version 19.11.  In June 1994, Lucid folded and Jamie quit to work for
the newly formed Mosaic Communications Corp., later Netscape
Communications Corp. (co-founded by the same Marc Andreessen, who had
quit his Epoch job to work on a graphical browser for the World Wide
Web).  Chuck then become the primary maintainer of XEmacs, and put out
versions 19.11 through 19.14 in conjunction with Ben.  For 19.12 and
19.13, Chuck added the new redisplay and many other display improvements
and Ben added MULE support (support for Asian and other languages) and
redesigned most of the internal Lisp subsystems to better support the
MULE work and the various other features being added to XEmacs.  After
19.14 Chuck retired as primary maintainer and Steve Baur stepped in.

@cindex MULE merged XEmacs appears
  Soon after 19.13 was released, work began in earnest on the MULE
internationalization code and the source tree was divided into two
development paths.  The MULE version was initially called 19.20, but was
soon renamed to 20.0.  In 1996 Martin Buchholz of Sun Microsystems took
over the care and feeding of it and worked on it in parallel with the
19.14 development that was occurring at the same time.  After much work
by Martin, it was decided to release 20.0 ahead of 19.15 in February
1997.  The source tree remained divided until 20.2 when the version 19
source was finally retired at version 19.16.

@cindex Baur, Steve
@cindex Buchholz, Martin
@cindex Jones, Kyle
@cindex Niksic, Hrvoje
@cindex XEmacs goes it alone
  In 1997, Sun finally dropped all pretense of support for XEmacs and
Martin Buchholz left the company in November.  Since then, and mostly
for the previous year, because Steve Baur was never paid to work on
XEmacs, XEmacs has existed solely on the contributions of volunteers
from the Free Software Community.  Starting from 1997, Hrvoje Niksic and
Kyle Jones have figured prominently in XEmacs development.

@cindex merging attempts
  Many attempts have been made to merge XEmacs and GNU Emacs, but they
have consistently failed.

  A more detailed history is contained in the XEmacs About page.

@node XEmacs From the Outside, The Lisp Language, A History of Emacs, Top
@chapter XEmacs From the Outside
@cindex read-eval-print

  XEmacs appears to the outside world as an editor, but it is really a
Lisp environment.  At its heart is a Lisp interpreter; it also
``happens'' to contain many specialized object types (e.g. buffers,
windows, frames, events) that are useful for implementing an editor.
Some of these objects (in particular windows and frames) have
displayable representations, and XEmacs provides a function
@code{redisplay()} that ensures that the display of all such objects
matches their internal state.  Most of the time, a standard Lisp
environment is in a @dfn{read-eval-print} loop -- i.e. ``read some Lisp
code, execute it, and print the results''.  XEmacs has a similar loop:

@itemize @bullet
@item
read an event
@item
dispatch the event (i.e. ``do it'')
@item
redisplay
@end itemize

  Reading an event is done using the Lisp function @code{next-event},
which waits for something to happen (typically, the user presses a key
or moves the mouse) and returns an event object describing this.
Dispatching an event is done using the Lisp function
@code{dispatch-event}, which looks up the event in a keymap object (a
particular kind of object that associates an event with a Lisp function)
and calls that function.  The function ``does'' what the user has
requested by changing the state of particular frame objects, buffer
objects, etc.  Finally, @code{redisplay()} is called, which updates the
display to reflect those changes just made.  Thus is an ``editor'' born.

@cindex bridge, playing
@cindex taxes, doing
@cindex pi, calculating
  Note that you do not have to use XEmacs as an editor; you could just
as well make it do your taxes, compute pi, play bridge, etc.  You'd just
have to write functions to do those operations in Lisp.

@node The Lisp Language, XEmacs From the Perspective of Building, XEmacs From the Outside, Top
@chapter The Lisp Language
@cindex Lisp vs. C
@cindex C vs. Lisp
@cindex Lisp vs. Java
@cindex Java vs. Lisp
@cindex dynamic scoping
@cindex scoping, dynamic
@cindex dynamic types
@cindex types, dynamic
@cindex Java
@cindex Common Lisp
@cindex Gosling, James

  Lisp is a general-purpose language that is higher-level than C and in
many ways more powerful than C.  Powerful dialects of Lisp such as
Common Lisp are probably much better languages for writing very large
applications than is C. (Unfortunately, for many non-technical
reasons C and its successor C++ have become the dominant languages for
application development.  These languages are both inadequate for
extremely large applications, which is evidenced by the fact that newer,
larger programs are becoming ever harder to write and are requiring ever
more programmers despite great increases in C development environments;
and by the fact that, although hardware speeds and reliability have been
growing at an exponential rate, most software is still generally
considered to be slow and buggy.)

  The new Java language holds promise as a better general-purpose
development language than C.  Java has many features in common with
Lisp that are not shared by C (this is not a coincidence, since
Java was designed by James Gosling, a former Lisp hacker).  This
will be discussed more later.

For those used to C, here is a summary of the basic differences between
C and Lisp:

@enumerate
@item
Lisp has an extremely regular syntax.  Every function, expression,
and control statement is written in the form

@example
   (@var{func} @var{arg1} @var{arg2} ...)
@end example

This is as opposed to C, which writes functions as

@example
   func(@var{arg1}, @var{arg2}, ...)
@end example

but writes expressions involving operators as (e.g.)

@example
   @var{arg1} + @var{arg2}
@end example

and writes control statements as (e.g.)

@example
   while (@var{expr}) @{ @var{statement1}; @var{statement2}; ... @}
@end example

Lisp equivalents of the latter two would be

@example
   (+ @var{arg1} @var{arg2} ...)
@end example

and

@example
   (while @var{expr} @var{statement1} @var{statement2} ...)
@end example

@item
Lisp is a safe language.  Assuming there are no bugs in the Lisp
interpreter/compiler, it is impossible to write a program that ``core
dumps'' or otherwise causes the machine to execute an illegal
instruction.  This is very different from C, where perhaps the most
common outcome of a bug is exactly such a crash.  A corollary of this is that
the C operation of casting a pointer is impossible (and unnecessary) in
Lisp, and that it is impossible to access memory outside the bounds of
an array.

@item
Programs and data are written in the same form.  The
parenthesis-enclosing form described above for statements is the same
form used for the most common data type in Lisp, the list.  Thus, it is
possible to represent any Lisp program using Lisp data types, and for
one program to construct Lisp statements and then dynamically
@dfn{evaluate} them, or cause them to execute.

@item
All objects are @dfn{dynamically typed}.  This means that part of every
object is an indication of what type it is.  A Lisp program can
manipulate an object without knowing what type it is, and can query an
object to determine its type.  This means that, correspondingly,
variables and function parameters can hold objects of any type and are
not normally declared as being of any particular type.  This is opposed
to the @dfn{static typing} of C, where variables can hold exactly one
type of object and must be declared as such, and objects do not contain
an indication of their type because it's implicit in the variables they
are stored in.  It is possible in C to have a variable hold different
types of objects (e.g. through the use of @code{void *} pointers or
variable-argument functions), but the type information must then be
passed explicitly in some other fashion, leading to additional program
complexity.

@item
Allocated memory is automatically reclaimed when it is no longer in use.
This operation is called @dfn{garbage collection} and involves looking
through all variables to see what memory is being pointed to, and
reclaiming any memory that is not pointed to and is thus
``inaccessible'' and out of use.  This is as opposed to C, in which
allocated memory must be explicitly reclaimed using @code{free()}.  If
you simply drop all pointers to memory without freeing it, it becomes
``leaked'' memory that still takes up space.  Over a long period of
time, this can cause your program to grow and grow until it runs out of
memory.

@item
Lisp has built-in facilities for handling errors and exceptions.  In C,
when an error occurs, usually either the program exits entirely or the
routine in which the error occurs returns a value indicating this.  If
an error occurs in a deeply-nested routine, then every routine currently
called must unwind itself normally and return an error value back up to
the next routine.  This means that every routine must explicitly check
for an error in all the routines it calls; if it does not do so,
unexpected and often random behavior results.  This is an extremely
common source of bugs in C programs.  An alternative would be to do a
non-local exit using @code{longjmp()}, but that is often very dangerous
because the routines that were exited past had no opportunity to clean
up after themselves and may leave things in an inconsistent state,
causing a crash shortly afterwards.

Lisp provides mechanisms to make such non-local exits safe.  When an
error occurs, a routine simply signals that an error of a particular
class has occurred, and a non-local exit takes place.  Any routine can
trap errors occurring in routines it calls by registering an error
handler for some or all classes of errors. (If no handler is registered,
a default handler, generally installed by the top-level event loop, is
executed; this prints out the error and continues.) Routines can also
specify cleanup code (called an @dfn{unwind-protect}) that will be
called when control exits from a block of code, no matter how that exit
occurs -- i.e. even if a function deeply nested below it causes a
non-local exit back to the top level.

Note that this facility has appeared in some recent vintages of C, in
particular Visual C++ and other PC compilers written for the Microsoft
Win32 API.

@item
In Emacs Lisp, local variables are @dfn{dynamically scoped}.  This means
that if you declare a local variable in a particular function, and then
call another function, that subfunction can ``see'' the local variable
you declared.  This is actually considered a bug in Emacs Lisp and in
all other early dialects of Lisp, and was corrected in Common Lisp. (In
Common Lisp, you can still declare dynamically scoped variables if you
want to -- they are sometimes useful -- but variables by default are
@dfn{lexically scoped} as in C.)
@end enumerate

For those familiar with Lisp, Emacs Lisp is modelled after MacLisp, an
early dialect of Lisp developed at MIT (no relation to the Macintosh
computer).  There is a Common Lisp compatibility package available for
Emacs that provides many of the features of Common Lisp.

The Java language is derived in many ways from C, and shares a similar
syntax, but has the following features in common with Lisp (and different
from C):

@enumerate
@item
Java is a safe language, like Lisp.
@item
Java provides garbage collection, like Lisp.
@item
Java has built-in facilities for handling errors and exceptions, like
Lisp.
@item
Java has a type system that combines the best advantages of both static
and dynamic typing.  Objects (except very simple types) are explicitly
marked with their type, as in dynamic typing; but there is a hierarchy
of types and functions are declared to accept only certain types, thus
providing the increased compile-time error-checking of static typing.
@end enumerate

The Java language also has some negative attributes:

@enumerate
@item
Java uses the edit/compile/run model of software development.  This
makes it hard to use interactively.  For example, to use Java like
@code{bc} it is necessary to write a special purpose, albeit tiny,
application.  In Emacs Lisp, a calculator comes built-in without any
effort - one can always just type an expression in the @code{*scratch*}
buffer.
@item
Java tries too hard to enforce, not merely enable, portability, making
ordinary access to standard OS facilities painful.  Java has an
@dfn{agenda}.  I think this is why @code{chdir} is not part of standard
Java, which is inexcusable.
@end enumerate

Unfortunately, there is no perfect language.  Static typing allows a
compiler to catch programmer errors and produce more efficient code, but
makes programming more tedious and less fun.  For the forseeable future,
an Ideal Editing and Programming Environment (and that is what XEmacs
aspires to) will be programmable in multiple languages: high level ones
like Lisp for user customization and prototyping, and lower level ones
for infrastructure and industrial strength applications.  If I had my
way, XEmacs would be friendly towards the Python, Scheme, C++, ML,
etc... communities.  But there are serious technical difficulties to
achieving that goal.

The word @dfn{application} in the previous paragraph was used
intentionally.  XEmacs implements an API for programs written in Lisp
that makes it a full-fledged application platform, very much like an OS
inside the real OS.

@node XEmacs From the Perspective of Building, XEmacs From the Inside, The Lisp Language, Top
@chapter XEmacs From the Perspective of Building

The heart of XEmacs is the Lisp environment, which is written in C.
This is contained in the @file{src/} subdirectory.  Underneath
@file{src/} are two subdirectories of header files: @file{s/} (header
files for particular operating systems) and @file{m/} (header files for
particular machine types).  In practice the distinction between the two
types of header files is blurred.  These header files define or undefine
certain preprocessor constants and macros to indicate particular
characteristics of the associated machine or operating system.  As part
of the configure process, one @file{s/} file and one @file{m/} file is
identified for the particular environment in which XEmacs is being
built.

XEmacs also contains a great deal of Lisp code.  This implements the
operations that make XEmacs useful as an editor as well as just a Lisp
environment, and also contains many add-on packages that allow XEmacs to
browse directories, act as a mail and Usenet news reader, compile Lisp
code, etc.  There is actually more Lisp code than C code associated with
XEmacs, but much of the Lisp code is peripheral to the actual operation
of the editor.  The Lisp code all lies in subdirectories underneath the
@file{lisp/} directory.

The @file{lwlib/} directory contains C code that implements a
generalized interface onto different X widget toolkits and also
implements some widgets of its own that behave like Motif widgets but
are faster, free, and in some cases more powerful.  The code in this
directory compiles into a library and is mostly independent from XEmacs.

The @file{etc/} directory contains various data files associated with
XEmacs.  Some of them are actually read by XEmacs at startup; others
merely contain useful information of various sorts.

The @file{lib-src/} directory contains C code for various auxiliary
programs that are used in connection with XEmacs.  Some of them are used
during the build process; others are used to perform certain functions
that cannot conveniently be placed in the XEmacs executable (e.g. the
@file{movemail} program for fetching mail out of @file{/var/spool/mail},
which must be setgid to @file{mail} on many systems; and the
@file{gnuclient} program, which allows an external script to communicate
with a running XEmacs process).

The @file{man/} directory contains the sources for the XEmacs
documentation.  It is mostly in a form called Texinfo, which can be
converted into either a printed document (by passing it through @TeX{})
or into on-line documentation called @dfn{info files}.

The @file{info/} directory contains the results of formatting the XEmacs
documentation as @dfn{info files}, for on-line use.  These files are
used when you enter the Info system using @kbd{C-h i} or through the
Help menu.

The @file{dynodump/} directory contains auxiliary code used to build
XEmacs on Solaris platforms.

The other directories contain various miscellaneous code and information
that is not normally used or needed.

The first step of building involves running the @file{configure} program
and passing it various parameters to specify any optional features you
want and compiler arguments and such, as described in the @file{INSTALL}
file.  This determines what the build environment is, chooses the
appropriate @file{s/} and @file{m/} file, and runs a series of tests to
determine many details about your environment, such as which library
functions are available and exactly how they work.  The reason for
running these tests is that it allows XEmacs to be compiled on a much
wider variety of platforms than those that the XEmacs developers happen
to be familiar with, including various sorts of hybrid platforms.  This
is especially important now that many operating systems give you a great
deal of control over exactly what features you want installed, and allow
for easy upgrading of parts of a system without upgrading the rest.  It
would be impossible to pre-determine and pre-specify the information for
all possible configurations.

In fact, the @file{s/} and @file{m/} files are basically @emph{evil},
since they contain unmaintainable platform-specific hard-coded
information.  XEmacs has been moving in the direction of having all
system-specific information be determined dynamically by
@file{configure}.  Perhaps someday we can @code{rm -rf src/s src/m}.

When configure is done running, it generates @file{Makefile}s and
@file{GNUmakefile}s and the file @file{src/config.h} (which describes
the features of your system) from template files.  You then run
@file{make}, which compiles the auxiliary code and programs in
@file{lib-src/} and @file{lwlib/} and the main XEmacs executable in
@file{src/}.  The result of compiling and linking is an executable
called @file{temacs}, which is @emph{not} the final XEmacs executable.
@file{temacs} by itself is not intended to function as an editor or even
display any windows on the screen, and if you simply run it, it will
exit immediately.  The @file{Makefile} runs @file{temacs} with certain
options that cause it to initialize itself, read in a number of basic
Lisp files, and then dump itself out into a new executable called
@file{xemacs}.  This new executable has been pre-initialized and
contains pre-digested Lisp code that is necessary for the editor to
function (this includes most basic editing functions,
e.g. @code{kill-line}, that can be defined in terms of other Lisp
primitives; some initialization code that is called when certain
objects, such as frames, are created; and all of the standard
keybindings and code for the actions they result in).  This executable,
@file{xemacs}, is the executable that you run to use the XEmacs editor.

Although @file{temacs} is not intended to be run as an editor, it can,
by using the incantation @code{temacs -batch -l loadup.el run-temacs}.
This is useful when the dumping procedure described above is broken, or
when using certain program debugging tools such as Purify.  These tools
get mighty confused by the tricks played by the XEmacs build process,
such as allocation memory in one process, and freeing it in the next.

@node XEmacs From the Inside, The XEmacs Object System (Abstractly Speaking), XEmacs From the Perspective of Building, Top
@chapter XEmacs From the Inside

Internally, XEmacs is quite complex, and can be very confusing.  To
simplify things, it can be useful to think of XEmacs as containing an
event loop that ``drives'' everything, and a number of other subsystems,
such as a Lisp engine and a redisplay mechanism.  Each of these other
subsystems exists simultaneously in XEmacs, and each has a certain
state.  The flow of control continually passes in and out of these
different subsystems in the course of normal operation of the editor.

It is important to keep in mind that, most of the time, the editor is
``driven'' by the event loop.  Except during initialization and batch
mode, all subsystems are entered directly or indirectly through the
event loop, and ultimately, control exits out of all subsystems back up
to the event loop.  This cycle of entering a subsystem, exiting back out
to the event loop, and starting another iteration of the event loop
occurs once each keystroke, mouse motion, etc.

If you're trying to understand a particular subsystem (other than the
event loop), think of it as a ``daemon'' process or ``servant'' that is
responsible for one particular aspect of a larger system, and
periodically receives commands or environment changes that cause it to
do something.  Ultimately, these commands and environment changes are
always triggered by the event loop.  For example:

@itemize @bullet
@item
The window and frame mechanism is responsible for keeping track of what
windows and frames exist, what buffers are in them, etc.  It is
periodically given commands (usually from the user) to make a change to
the current window/frame state: i.e. create a new frame, delete a
window, etc.

@item
The buffer mechanism is responsible for keeping track of what buffers
exist and what text is in them.  It is periodically given commands
(usually from the user) to insert or delete text, create a buffer, etc.
When it receives a text-change command, it notifies the redisplay
mechanism.

@item
The redisplay mechanism is responsible for making sure that windows and
frames are displayed correctly.  It is periodically told (by the event
loop) to actually ``do its job'', i.e. snoop around and see what the
current state of the environment (mostly of the currently-existing
windows, frames, and buffers) is, and make sure that that state matches
what's actually displayed.  It keeps lots and lots of information around
(such as what is actually being displayed currently, and what the
environment was last time it checked) so that it can minimize the work
it has to do.  It is also helped along in that whenever a relevant
change to the environment occurs, the redisplay mechanism is told about
this, so it has a pretty good idea of where it has to look to find
possible changes and doesn't have to look everywhere.

@item
The Lisp engine is responsible for executing the Lisp code in which most
user commands are written.  It is entered through a call to @code{eval}
or @code{funcall}, which occurs as a result of dispatching an event from
the event loop.  The functions it calls issue commands to the buffer
mechanism, the window/frame subsystem, etc.

@item
The Lisp allocation subsystem is responsible for keeping track of Lisp
objects.  It is given commands from the Lisp engine to allocate objects,
garbage collect, etc.
@end itemize

etc.

  The important idea here is that there are a number of independent
subsystems each with its own responsibility and persistent state, just
like different employees in a company, and each subsystem is
periodically given commands from other subsystems.  Commands can flow
from any one subsystem to any other, but there is usually some sort of
hierarchy, with all commands originating from the event subsystem.

  XEmacs is entered in @code{main()}, which is in @file{emacs.c}.  When
this is called the first time (in a properly-invoked @file{temacs}), it
does the following:

@enumerate
@item
It does some very basic environment initializations, such as determining
where it and its directories (e.g. @file{lisp/} and @file{etc/}) reside
and setting up signal handlers.
@item
It initializes the entire Lisp interpreter.
@item
It sets the initial values of many built-in variables (including many
variables that are visible to Lisp programs), such as the global keymap
object and the built-in faces (a face is an object that describes the
display characteristics of text).  This involves creating Lisp objects
and thus is dependent on step (2).
@item
It performs various other initializations that are relevant to the
particular environment it is running in, such as retrieving environment
variables, determining the current date and the user who is running the
program, examining its standard input, creating any necessary file
descriptors, etc.
@item
At this point, the C initialization is complete.  A Lisp program that
was specified on the command line (usually @file{loadup.el}) is called
(temacs is normally invoked as @code{temacs -batch -l loadup.el dump}).
@file{loadup.el} loads all of the other Lisp files that are needed for
the operation of the editor, calls the @code{dump-emacs} function to
write out @file{xemacs}, and then kills the temacs process.
@end enumerate

  When @file{xemacs} is then run, it only redoes steps (1) and (4)
above; all variables already contain the values they were set to when
the executable was dumped, and all memory that was allocated with
@code{malloc()} is still around. (XEmacs knows whether it is being run
as @file{xemacs} or @file{temacs} because it sets the global variable
@code{initialized} to 1 after step (4) above.) At this point,
@file{xemacs} calls a Lisp function to do any further initialization,
which includes parsing the command-line (the C code can only do limited
command-line parsing, which includes looking for the @samp{-batch} and
@samp{-l} flags and a few other flags that it needs to know about before
initialization is complete), creating the first frame (or @dfn{window}
in standard window-system parlance), running the user's init file
(usually the file @file{.emacs} in the user's home directory), etc.  The
function to do this is usually called @code{normal-top-level};
@file{loadup.el} tells the C code about this function by setting its
name as the value of the Lisp variable @code{top-level}.

  When the Lisp initialization code is done, the C code enters the event
loop, and stays there for the duration of the XEmacs process.  The code
for the event loop is contained in @file{keyboard.c}, and is called
@code{Fcommand_loop_1()}.  Note that this event loop could very well be
written in Lisp, and in fact a Lisp version exists; but apparently,
doing this makes XEmacs run noticeably slower.

  Notice how much of the initialization is done in Lisp, not in C.
In general, XEmacs tries to move as much code as is possible
into Lisp.  Code that remains in C is code that implements the
Lisp interpreter itself, or code that needs to be very fast, or
code that needs to do system calls or other such stuff that
needs to be done in C, or code that needs to have access to
``forbidden'' structures. (One conscious aspect of the design of
Lisp under XEmacs is a clean separation between the external
interface to a Lisp object's functionality and its internal
implementation.  Part of this design is that Lisp programs
are forbidden from accessing the contents of the object other
than through using a standard API.  In this respect, XEmacs Lisp
is similar to modern Lisp dialects but differs from GNU Emacs,
which tends to expose the implementation and allow Lisp
programs to look at it directly.  The major advantage of
hiding the implementation is that it allows the implementation
to be redesigned without affecting any Lisp programs, including
those that might want to be ``clever'' by looking directly at
the object's contents and possibly manipulating them.)

  Moving code into Lisp makes the code easier to debug and maintain and
makes it much easier for people who are not XEmacs developers to
customize XEmacs, because they can make a change with much less chance
of obscure and unwanted interactions occurring than if they were to
change the C code.

@node The XEmacs Object System (Abstractly Speaking), How Lisp Objects Are Represented in C, XEmacs From the Inside, Top
@chapter The XEmacs Object System (Abstractly Speaking)

  At the heart of the Lisp interpreter is its management of objects.
XEmacs Lisp contains many built-in objects, some of which are
simple and others of which can be very complex; and some of which
are very common, and others of which are rarely used or are only
used internally. (Since the Lisp allocation system, with its
automatic reclamation of unused storage, is so much more convenient
than @code{malloc()} and @code{free()}, the C code makes extensive use of it
in its internal operations.)

  The basic Lisp objects are

@table @code
@item integer
28 or 31 bits of precision, or 60 or 63 bits on 64-bit machines; the
reason for this is described below when the internal Lisp object
representation is described.
@item float
Same precision as a double in C.
@item cons
A simple container for two Lisp objects, used to implement lists and
most other data structures in Lisp.
@item char
An object representing a single character of text; chars behave like
integers in many ways but are logically considered text rather than
numbers and have a different read syntax. (the read syntax for a char
contains the char itself or some textual encoding of it -- for example,
a Japanese Kanji character might be encoded as @samp{^[$(B#&^[(B} using the
ISO-2022 encoding standard -- rather than the numerical representation
of the char; this way, if the mapping between chars and integers
changes, which is quite possible for Kanji characters and other extended
characters, the same character will still be created.  Note that some
primitives confuse chars and integers.  The worst culprit is @code{eq},
which makes a special exception and considers a char to be @code{eq} to
its integer equivalent, even though in no other case are objects of two
different types @code{eq}.  The reason for this monstrosity is
compatibility with existing code; the separation of char from integer
came fairly recently.)
@item symbol
An object that contains Lisp objects and is referred to by name;
symbols are used to implement variables and named functions
and to provide the equivalent of preprocessor constants in C.
@item vector
A one-dimensional array of Lisp objects providing constant-time access
to any of the objects; access to an arbitrary object in a vector is
faster than for lists, but the operations that can be done on a vector
are more limited.
@item string
Self-explanatory; behaves much like a vector of chars
but has a different read syntax and is stored and manipulated
more compactly.
@item bit-vector
A vector of bits; similar to a string in spirit.
@item compiled-function
An object containing compiled Lisp code, known as @dfn{byte code}.
@item subr
A Lisp primitive, i.e. a Lisp-callable function implemented in C.
@end table

@cindex closure
Note that there is no basic ``function'' type, as in more powerful
versions of Lisp (where it's called a @dfn{closure}).  XEmacs Lisp does
not provide the closure semantics implemented by Common Lisp and Scheme.
The guts of a function in XEmacs Lisp are represented in one of four
ways: a symbol specifying another function (when one function is an
alias for another), a list (whose first element must be the symbol
@code{lambda}) containing the function's source code, a
compiled-function object, or a subr object. (In other words, given a
symbol specifying the name of a function, calling @code{symbol-function}
to retrieve the contents of the symbol's function cell will return one
of these types of objects.)

XEmacs Lisp also contains numerous specialized objects used to implement
the editor:

@table @code
@item buffer
Stores text like a string, but is optimized for insertion and deletion
and has certain other properties that can be set.
@item frame
An object with various properties whose displayable representation is a
@dfn{window} in window-system parlance.
@item window
A section of a frame that displays the contents of a buffer;
often called a @dfn{pane} in window-system parlance.
@item window-configuration
An object that represents a saved configuration of windows in a frame.
@item device
An object representing a screen on which frames can be displayed;
equivalent to a @dfn{display} in the X Window System and a @dfn{TTY} in
character mode.
@item face
An object specifying the appearance of text or graphics; it has
properties such as font, foreground color, and background color.
@item marker
An object that refers to a particular position in a buffer and moves
around as text is inserted and deleted to stay in the same relative
position to the text around it.
@item extent
Similar to a marker but covers a range of text in a buffer; can also
specify properties of the text, such as a face in which the text is to
be displayed, whether the text is invisible or unmodifiable, etc.
@item event
Generated by calling @code{next-event} and contains information
describing a particular event happening in the system, such as the user
pressing a key or a process terminating.
@item keymap
An object that maps from events (described using lists, vectors, and
symbols rather than with an event object because the mapping is for
classes of events, rather than individual events) to functions to
execute or other events to recursively look up; the functions are
described by name, using a symbol, or using lists to specify the
function's code.
@item glyph
An object that describes the appearance of an image (e.g.  pixmap) on
the screen; glyphs can be attached to the beginning or end of extents
and in some future version of XEmacs will be able to be inserted
directly into a buffer.
@item process
An object that describes a connection to an externally-running process.
@end table

  There are some other, less-commonly-encountered general objects:

@table @code
@item hash-table
An object that maps from an arbitrary Lisp object to another arbitrary
Lisp object, using hashing for fast lookup.
@item obarray
A limited form of hash-table that maps from strings to symbols; obarrays
are used to look up a symbol given its name and are not actually their
own object type but are kludgily represented using vectors with hidden
fields (this representation derives from GNU Emacs).
@item specifier
A complex object used to specify the value of a display property; a
default value is given and different values can be specified for
particular frames, buffers, windows, devices, or classes of device.
@item char-table
An object that maps from chars or classes of chars to arbitrary Lisp
objects; internally char tables use a complex nested-vector
representation that is optimized to the way characters are represented
as integers.
@item range-table
An object that maps from ranges of integers to arbitrary Lisp objects.
@end table

  And some strange special-purpose objects:

@table @code
@item charset
@itemx coding-system
Objects used when MULE, or multi-lingual/Asian-language, support is
enabled.
@item color-instance
@itemx font-instance
@itemx image-instance
An object that encapsulates a window-system resource; instances are
mostly used internally but are exposed on the Lisp level for cleanness
of the specifier model and because it's occasionally useful for Lisp
program to create or query the properties of instances.
@item subwindow
An object that encapsulate a @dfn{subwindow} resource, i.e. a
window-system child window that is drawn into by an external process;
this object should be integrated into the glyph system but isn't yet,
and may change form when this is done.
@item tooltalk-message
@itemx tooltalk-pattern
Objects that represent resources used in the ToolTalk interprocess
communication protocol.
@item toolbar-button
An object used in conjunction with the toolbar.
@end table

  And objects that are only used internally:

@table @code
@item opaque
A generic object for encapsulating arbitrary memory; this allows you the
generality of @code{malloc()} and the convenience of the Lisp object
system.
@item lstream
A buffering I/O stream, used to provide a unified interface to anything
that can accept output or provide input, such as a file descriptor, a
stdio stream, a chunk of memory, a Lisp buffer, a Lisp string, etc.;
it's a Lisp object to make its memory management more convenient.
@item char-table-entry
Subsidiary objects in the internal char-table representation.
@item extent-auxiliary
@itemx menubar-data
@itemx toolbar-data
Various special-purpose objects that are basically just used to
encapsulate memory for particular subsystems, similar to the more
general ``opaque'' object.
@item symbol-value-forward
@itemx symbol-value-buffer-local
@itemx symbol-value-varalias
@itemx symbol-value-lisp-magic
Special internal-only objects that are placed in the value cell of a
symbol to indicate that there is something special with this variable --
e.g. it has no value, it mirrors another variable, or it mirrors some C
variable; there is really only one kind of object, called a
@dfn{symbol-value-magic}, but it is sort-of halfway kludged into
semi-different object types.
@end table

@cindex permanent objects
@cindex temporary objects
  Some types of objects are @dfn{permanent}, meaning that once created,
they do not disappear until explicitly destroyed, using a function such
as @code{delete-buffer}, @code{delete-window}, @code{delete-frame}, etc.
Others will disappear once they are not longer used, through the garbage
collection mechanism.  Buffers, frames, windows, devices, and processes
are among the objects that are permanent.  Note that some objects can go
both ways: Faces can be created either way; extents are normally
permanent, but detached extents (extents not referring to any text, as
happens to some extents when the text they are referring to is deleted)
are temporary.  Note that some permanent objects, such as faces and
coding systems, cannot be deleted.  Note also that windows are unique in
that they can be @emph{undeleted} after having previously been
deleted. (This happens as a result of restoring a window configuration.)

@cindex read syntax
  Note that many types of objects have a @dfn{read syntax}, i.e. a way of
specifying an object of that type in Lisp code.  When you load a Lisp
file, or type in code to be evaluated, what really happens is that the
function @code{read} is called, which reads some text and creates an object
based on the syntax of that text; then @code{eval} is called, which
possibly does something special; then this loop repeats until there's
no more text to read. (@code{eval} only actually does something special
with symbols, which causes the symbol's value to be returned,
similar to referencing a variable; and with conses [i.e. lists],
which cause a function invocation.  All other values are returned
unchanged.)

  The read syntax

@example
17297
@end example

converts to an integer whose value is 17297.

@example
1.983e-4
@end example

converts to a float whose value is 1.983e-4, or .0001983.

@example
?b
@end example

converts to a char that represents the lowercase letter b.

@example
?^[$(B#&^[(B
@end example

(where @samp{^[} actually is an @samp{ESC} character) converts to a
particular Kanji character when using an ISO2022-based coding system for
input. (To decode this goo: @samp{ESC} begins an escape sequence;
@samp{ESC $ (} is a class of escape sequences meaning ``switch to a
94x94 character set''; @samp{ESC $ ( B} means ``switch to Japanese
Kanji''; @samp{#} and @samp{&} collectively index into a 94-by-94 array
of characters [subtract 33 from the ASCII value of each character to get
the corresponding index]; @samp{ESC (} is a class of escape sequences
meaning ``switch to a 94 character set''; @samp{ESC (B} means ``switch
to US ASCII''.  It is a coincidence that the letter @samp{B} is used to
denote both Japanese Kanji and US ASCII.  If the first @samp{B} were
replaced with an @samp{A}, you'd be requesting a Chinese Hanzi character
from the GB2312 character set.)

@example
"foobar"
@end example

converts to a string.

@example
foobar
@end example

converts to a symbol whose name is @code{"foobar"}.  This is done by
looking up the string equivalent in the global variable
@code{obarray}, whose contents should be an obarray.  If no symbol
is found, a new symbol with the name @code{"foobar"} is automatically
created and added to @code{obarray}; this process is called
@dfn{interning} the symbol.
@cindex interning

@example
(foo . bar)
@end example

converts to a cons cell containing the symbols @code{foo} and @code{bar}.

@example
(1 a 2.5)
@end example

converts to a three-element list containing the specified objects
(note that a list is actually a set of nested conses; see the
XEmacs Lisp Reference).

@example
[1 a 2.5]
@end example

converts to a three-element vector containing the specified objects.

@example
#[... ... ... ...]
@end example

converts to a compiled-function object (the actual contents are not
shown since they are not relevant here; look at a file that ends with
@file{.elc} for examples).

@example
#*01110110
@end example

converts to a bit-vector.

@example
#s(hash-table ... ...)
@end example

converts to a hash table (the actual contents are not shown).

@example
#s(range-table ... ...)
@end example

converts to a range table (the actual contents are not shown).

@example
#s(char-table ... ...)
@end example

converts to a char table (the actual contents are not shown).

Note that the @code{#s()} syntax is the general syntax for structures,
which are not really implemented in XEmacs Lisp but should be.

When an object is printed out (using @code{print} or a related
function), the read syntax is used, so that the same object can be read
in again.

The other objects do not have read syntaxes, usually because it does not
really make sense to create them in this fashion (i.e.  processes, where
it doesn't make sense to have a subprocess created as a side effect of
reading some Lisp code), or because they can't be created at all
(e.g. subrs).  Permanent objects, as a rule, do not have a read syntax;
nor do most complex objects, which contain too much state to be easily
initialized through a read syntax.

@node How Lisp Objects Are Represented in C, Rules When Writing New C Code, The XEmacs Object System (Abstractly Speaking), Top
@chapter How Lisp Objects Are Represented in C

Lisp objects are represented in C using a 32-bit or 64-bit machine word
(depending on the processor; i.e. DEC Alphas use 64-bit Lisp objects and
most other processors use 32-bit Lisp objects).  The representation
stuffs a pointer together with a tag, as follows:

@example
 [ 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 ]
 [ 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 ]

   <---> ^ <------------------------------------------------------>
    tag  |       a pointer to a structure, or an integer
         |
       mark bit
@end example

The tag describes the type of the Lisp object.  For integers and chars,
the lower 28 bits contain the value of the integer or char; for all
others, the lower 28 bits contain a pointer.  The mark bit is used
during garbage-collection, and is always 0 when garbage collection is
not happening. (The way that garbage collection works, basically, is that it
loops over all places where Lisp objects could exist -- this includes
all global variables in C that contain Lisp objects [including
@code{Vobarray}, the C equivalent of @code{obarray}; through this, all
Lisp variables will get marked], plus various other places -- and
recursively scans through the Lisp objects, marking each object it finds
by setting the mark bit.  Then it goes through the lists of all objects
allocated, freeing the ones that are not marked and turning off the mark
bit of the ones that are marked.)

Lisp objects use the typedef @code{Lisp_Object}, but the actual C type
used for the Lisp object can vary.  It can be either a simple type
(@code{long} on the DEC Alpha, @code{int} on other machines) or a
structure whose fields are bit fields that line up properly (actually, a
union of structures is used).  Generally the simple integral type is
preferable because it ensures that the compiler will actually use a
machine word to represent the object (some compilers will use more
general and less efficient code for unions and structs even if they can
fit in a machine word).  The union type, however, has the advantage of
stricter type checking (if you accidentally pass an integer where a Lisp
object is desired, you get a compile error), and it makes it easier to
decode Lisp objects when debugging.  The choice of which type to use is
determined by the preprocessor constant @code{USE_UNION_TYPE} which is
defined via the @code{--use-union-type} option to @code{configure}.

@cindex record type

Note that there are only eight types that the tag can represent, but
many more actual types than this.  This is handled by having one of the
tag types specify a meta-type called a @dfn{record}; for all such
objects, the first four bytes of the pointed-to structure indicate what
the actual type is.

Note also that having 28 bits for pointers and integers restricts a lot
of things to 256 megabytes of memory. (Basically, enough pointers and
indices and whatnot get stuffed into Lisp objects that the total amount
of memory used by XEmacs can't grow above 256 megabytes.  In older
versions of XEmacs and GNU Emacs, the tag was 5 bits wide, allowing for
32 types, which was more than the actual number of types that existed at
the time, and no ``record'' type was necessary.  However, this limited
the editor to 64 megabytes total, which some users who edited large
files might conceivably exceed.)

Also, note that there is an implicit assumption here that all pointers
are low enough that the top bits are all zero and can just be chopped
off.  On standard machines that allocate memory from the bottom up (and
give each process its own address space), this works fine.  Some
machines, however, put the data space somewhere else in memory
(e.g. beginning at 0x80000000).  Those machines cope by defining
@code{DATA_SEG_BITS} in the corresponding @file{m/} or @file{s/} file to
the proper mask.  Then, pointers retrieved from Lisp objects are
automatically OR'ed with this value prior to being used.

A corollary of the previous paragraph is that @strong{(pointers to)
stack-allocated structures cannot be put into Lisp objects}.  The stack
is generally located near the top of memory; if you put such a pointer
into a Lisp object, it will get its top bits chopped off, and you will
lose.

Actually, there's an alternative representation of a @code{Lisp_Object},
invented by Kyle Jones, that is used when the
@code{--use-minimal-tagbits} option to @code{configure} is used.  In
this case the 2 lower bits are used for the tag bits.  This
representation assumes that pointers to structs are always aligned to
multiples of 4, so the lower 2 bits are always zero.

@example
 [ 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 ]
 [ 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 ]

   <---------------------------------------------------------> <->
            a pointer to a structure, or an integer            tag
@end example

A tag of 00 is used for all pointer object types, a tag of 10 is used
for characters, and the other two tags 01 and 11 are joined together to
form the integer object type.  The markbit is moved to part of the
structure being pointed at (integers and chars do not need to be marked,
since no memory is allocated).  This representation has these
advantages:

@enumerate
@item
31 bits can be used for Lisp Integers.
@item
@emph{Any} pointer can be represented directly, and no bit masking
operations are necessary.
@end enumerate

The disadvantages are:

@enumerate
@item
An extra level of indirection is needed when accessing the object types
that were not record types.  So checking whether a Lisp object is a cons
cell becomes a slower operation.
@item
Mark bits can no longer be stored directly in Lisp objects, so another
place for them must be found.  This means that a cons cell requires more
memory than merely room for 2 lisp objects, leading to extra memory use.
@end enumerate

Various macros are used to construct Lisp objects and extract the
components.  Macros of the form @code{XINT()}, @code{XCHAR()},
@code{XSTRING()}, @code{XSYMBOL()}, etc. mask out the pointer/integer
field and cast it to the appropriate type.  All of the macros that
construct pointers will @code{OR} with @code{DATA_SEG_BITS} if
necessary.  @code{XINT()} needs to be a bit tricky so that negative
numbers are properly sign-extended: Usually it does this by shifting the
number four bits to the left and then four bits to the right.  This
assumes that the right-shift operator does an arithmetic shift (i.e. it
leaves the most-significant bit as-is rather than shifting in a zero, so
that it mimics a divide-by-two even for negative numbers).  Not all
machines/compilers do this, and on the ones that don't, a more
complicated definition is selected by defining
@code{EXPLICIT_SIGN_EXTEND}.

Note that when @code{ERROR_CHECK_TYPECHECK} is defined, the extractor
macros become more complicated -- they check the tag bits and/or the
type field in the first four bytes of a record type to ensure that the
object is really of the correct type.  This is great for catching places
where an incorrect type is being dereferenced -- this typically results
in a pointer being dereferenced as the wrong type of structure, with
unpredictable (and sometimes not easily traceable) results.

There are similar @code{XSET@var{TYPE}()} macros that construct a Lisp
object.  These macros are of the form @code{XSET@var{TYPE}
(@var{lvalue}, @var{result})},
i.e. they have to be a statement rather than just used in an expression.
The reason for this is that standard C doesn't let you ``construct'' a
structure (but GCC does).  Granted, this sometimes isn't too convenient;
for the case of integers, at least, you can use the function
@code{make_int()}, which constructs and @emph{returns} an integer
Lisp object.  Note that the @code{XSET@var{TYPE}()} macros are also
affected by @code{ERROR_CHECK_TYPECHECK} and make sure that the
structure is of the right type in the case of record types, where the
type is contained in the structure.

The C programmer is responsible for @strong{guaranteeing} that a
Lisp_Object is is the correct type before using the @code{X@var{TYPE}}
macros.  This is especially important in the case of lists.  Use
@code{XCAR} and @code{XCDR} if a Lisp_Object is certainly a cons cell,
else use @code{Fcar()} and @code{Fcdr()}.  Trust other C code, but not
Lisp code.  On the other hand, if XEmacs has an internal logic error,
it's better to crash immediately, so sprinkle ``unreachable''
@code{abort()}s liberally about the source code.

@node Rules When Writing New C Code, A Summary of the Various XEmacs Modules, How Lisp Objects Are Represented in C, Top
@chapter Rules When Writing New C Code

The XEmacs C Code is extremely complex and intricate, and there are many
rules that are more or less consistently followed throughout the code.
Many of these rules are not obvious, so they are explained here.  It is
of the utmost importance that you follow them.  If you don't, you may
get something that appears to work, but which will crash in odd
situations, often in code far away from where the actual breakage is.

@menu
* General Coding Rules::
* Writing Lisp Primitives::
* Adding Global Lisp Variables::
* Coding for Mule::
* Techniques for XEmacs Developers::
@end menu

@node General Coding Rules
@section General Coding Rules

The C code is actually written in a dialect of C called @dfn{Clean C},
meaning that it can be compiled, mostly warning-free, with either a C or
C++ compiler.  Coding in Clean C has several advantages over plain C.
C++ compilers are more nit-picking, and a number of coding errors have
been found by compiling with C++.  The ability to use both C and C++
tools means that a greater variety of development tools are available to
the developer.

Almost every module contains a @code{syms_of_*()} function and a
@code{vars_of_*()} function.  The former declares any Lisp primitives
you have defined and defines any symbols you will be using.  The latter
declares any global Lisp variables you have added and initializes global
C variables in the module.  For each such function, declare it in
@file{symsinit.h} and make sure it's called in the appropriate place in
@file{emacs.c}.  @strong{Important}: There are stringent requirements on
exactly what can go into these functions.  See the comment in
@file{emacs.c}.  The reason for this is to avoid obscure unwanted
interactions during initialization.  If you don't follow these rules,
you'll be sorry!  If you want to do anything that isn't allowed, create
a @code{complex_vars_of_*()} function for it.  Doing this is tricky,
though: You have to make sure your function is called at the right time
so that all the initialization dependencies work out.

Every module includes @file{<config.h>} (angle brackets so that
@samp{--srcdir} works correctly; @file{config.h} may or may not be in
the same directory as the C sources) and @file{lisp.h}.  @file{config.h}
must always be included before any other header files (including
system header files) to ensure that certain tricks played by various
@file{s/} and @file{m/} files work out correctly.

@strong{All global and static variables that are to be modifiable must
be declared uninitialized.}  This means that you may not use the
``declare with initializer'' form for these variables, such as @code{int
some_variable = 0;}.  The reason for this has to do with some kludges
done during the dumping process: If possible, the initialized data
segment is re-mapped so that it becomes part of the (unmodifiable) code
segment in the dumped executable.  This allows this memory to be shared
among multiple running XEmacs processes.  XEmacs is careful to place as
much constant data as possible into initialized variables (in
particular, into what's called the @dfn{pure space} -- see below) during
the @file{temacs} phase.

@cindex copy-on-write
@strong{Please note:} This kludge only works on a few systems nowadays,
and is rapidly becoming irrelevant because most modern operating systems
provide @dfn{copy-on-write} semantics.  All data is initially shared
between processes, and a private copy is automatically made (on a
page-by-page basis) when a process first attempts to write to a page of
memory.

Formerly, there was a requirement that static variables not be declared
inside of functions.  This had to do with another hack along the same
vein as what was just described: old USG systems put statically-declared
variables in the initialized data space, so those header files had a
@code{#define static} declaration. (That way, the data-segment remapping
described above could still work.) This fails badly on static variables
inside of functions, which suddenly become automatic variables;
therefore, you weren't supposed to have any of them.  This awful kludge
has been removed in XEmacs because

@enumerate
@item
almost all of the systems that used this kludge ended up having
to disable the data-segment remapping anyway;
@item
the only systems that didn't were extremely outdated ones;
@item
this hack completely messed up inline functions.
@end enumerate

The C source code makes heavy use of C preprocessor macros.  One popular
macro style is:

@example
#define FOO(var, value) do @{           \
  Lisp_Object FOO_value = (value);      \
  ... /* compute using FOO_value */     \
  (var) = bar;                          \
@} while (0)
@end example

The @code{do @{...@} while (0)} is a standard trick to allow FOO to have
statement semantics, so that it can safely be used within an @code{if}
statement in C, for example.  Multiple evaluation is prevented by
copying a supplied argument into a local variable, so that
@code{FOO(var,fun(1))} only calls @code{fun} once.

Lisp lists are popular data structures in the C code as well as in
Elisp.  There are two sets of macros that iterate over lists.
@code{EXTERNAL_LIST_LOOP_@var{n}} should be used when the list has been
supplied by the user, and cannot be trusted to be acyclic and
nil-terminated.  A @code{malformed-list} or @code{circular-list} error
will be generated if the list being iterated over is not entirely
kosher.  @code{LIST_LOOP_@var{n}}, on the other hand, is faster and less
safe, and can be used only on trusted lists.

Related macros are @code{GET_EXTERNAL_LIST_LENGTH} and
@code{GET_LIST_LENGTH}, which calculate the length of a list, and in the
case of @code{GET_EXTERNAL_LIST_LENGTH}, validating the properness of
the list.  The macros @code{EXTERNAL_LIST_LOOP_DELETE_IF} and
@code{LIST_LOOP_DELETE_IF} delete elements from a lisp list satisfying some
predicate.

@node Writing Lisp Primitives
@section Writing Lisp Primitives

Lisp primitives are Lisp functions implemented in C.  The details of
interfacing the C function so that Lisp can call it are handled by a few
C macros.  The only way to really understand how to write new C code is
to read the source, but we can explain some things here.

An example of a special form is the definition of @code{prog1}, from
@file{eval.c}.  (An ordinary function would have the same general
appearance.)

@cindex garbage collection protection
@smallexample
@group
DEFUN ("prog1", Fprog1, 1, UNEVALLED, 0, /*
Similar to `progn', but the value of the first form is returned.
\(prog1 FIRST BODY...): All the arguments are evaluated sequentially.
The value of FIRST is saved during evaluation of the remaining args,
whose values are discarded.
*/
       (args))
@{
  /* This function can GC */
  REGISTER Lisp_Object val, form, tail;
  struct gcpro gcpro1;

  val = Feval (XCAR (args));

  GCPRO1 (val);

  LIST_LOOP_3 (form, XCDR (args), tail)
    Feval (form);

  UNGCPRO;
  return val;
@}
@end group
@end smallexample

  Let's start with a precise explanation of the arguments to the
@code{DEFUN} macro.  Here is a template for them:

@example
@group
DEFUN (@var{lname}, @var{fname}, @var{min_args}, @var{max_args}, @var{interactive}, /*
@var{docstring}
*/
   (@var{arglist}))
@end group
@end example

@table @var
@item lname
This string is the name of the Lisp symbol to define as the function
name; in the example above, it is @code{"prog1"}.

@item fname
This is the C function name for this function.  This is the name that is
used in C code for calling the function.  The name is, by convention,
@samp{F} prepended to the Lisp name, with all dashes (@samp{-}) in the
Lisp name changed to underscores.  Thus, to call this function from C
code, call @code{Fprog1}.  Remember that the arguments are of type
@code{Lisp_Object}; various macros and functions for creating values of
type @code{Lisp_Object} are declared in the file @file{lisp.h}.

Primitives whose names are special characters (e.g. @code{+} or
@code{<}) are named by spelling out, in some fashion, the special
character: e.g. @code{Fplus()} or @code{Flss()}.  Primitives whose names
begin with normal alphanumeric characters but also contain special
characters are spelled out in some creative way, e.g. @code{let*}
becomes @code{FletX()}.

Each function also has an associated structure that holds the data for
the subr object that represents the function in Lisp.  This structure
conveys the Lisp symbol name to the initialization routine that will
create the symbol and store the subr object as its definition.  The C
variable name of this structure is always @samp{S} prepended to the
@var{fname}.  You hardly ever need to be aware of the existence of this
structure, since @code{DEFUN} plus @code{DEFSUBR} takes care of all the
details.

@item min_args
This is the minimum number of arguments that the function requires.  The
function @code{prog1} allows a minimum of one argument.

@item max_args
This is the maximum number of arguments that the function accepts, if
there is a fixed maximum.  Alternatively, it can be @code{UNEVALLED},
indicating a special form that receives unevaluated arguments, or
@code{MANY}, indicating an unlimited number of evaluated arguments (the
C equivalent of @code{&rest}).  Both @code{UNEVALLED} and @code{MANY}
are macros.  If @var{max_args} is a number, it may not be less than
@var{min_args} and it may not be greater than 8. (If you need to add a
function with more than 8 arguments, use the @code{MANY} form.  Resist
the urge to edit the definition of @code{DEFUN} in @file{lisp.h}.  If
you do it anyways, make sure to also add another clause to the switch
statement in @code{primitive_funcall().})

@item interactive
This is an interactive specification, a string such as might be used as
the argument of @code{interactive} in a Lisp function.  In the case of
@code{prog1}, it is 0 (a null pointer), indicating that @code{prog1}
cannot be called interactively.  A value of @code{""} indicates a
function that should receive no arguments when called interactively.

@item docstring
This is the documentation string.  It is written just like a
documentation string for a function defined in Lisp; in particular, the
first line should be a single sentence.  Note how the documentation
string is enclosed in a comment, none of the documentation is placed on
the same lines as the comment-start and comment-end characters, and the
comment-start characters are on the same line as the interactive
specification.  @file{make-docfile}, which scans the C files for
documentation strings, is very particular about what it looks for, and
will not properly extract the doc string if it's not in this exact format.

In order to make both @file{etags} and @file{make-docfile} happy, make
sure that the @code{DEFUN} line contains the @var{lname} and
@var{fname}, and that the comment-start characters for the doc string
are on the same line as the interactive specification, and put a newline
directly after them (and before the comment-end characters).

@item arglist
This is the comma-separated list of arguments to the C function.  For a
function with a fixed maximum number of arguments, provide a C argument
for each Lisp argument.  In this case, unlike regular C functions, the
types of the arguments are not declared; they are simply always of type
@code{Lisp_Object}.

The names of the C arguments will be used as the names of the arguments
to the Lisp primitive as displayed in its documentation, modulo the same
concerns described above for @code{F...} names (in particular,
underscores in the C arguments become dashes in the Lisp arguments).

There is one additional kludge: A trailing `_' on the C argument is
discarded when forming the Lisp argument.  This allows C language
reserved words (like @code{default}) or global symbols (like
@code{dirname}) to be used as argument names without compiler warnings
or errors.

A Lisp function with @w{@var{max_args} = @code{UNEVALLED}} is a
@w{@dfn{special form}}; its arguments are not evaluated.  Instead it
receives one argument of type @code{Lisp_Object}, a (Lisp) list of the
unevaluated arguments, conventionally named @code{(args)}.

When a Lisp function has no upper limit on the number of arguments,
specify @w{@var{max_args} = @code{MANY}}.  In this case its implementation in
C actually receives exactly two arguments: the number of Lisp arguments
(an @code{int}) and the address of a block containing their values (a
@w{@code{Lisp_Object *}}).  In this case only are the C types specified
in the @var{arglist}: @w{@code{(int nargs, Lisp_Object *args)}}.

@end table

Within the function @code{Fprog1} itself, note the use of the macros
@code{GCPRO1} and @code{UNGCPRO}.  @code{GCPRO1} is used to ``protect''
a variable from garbage collection---to inform the garbage collector
that it must look in that variable and regard the object pointed at by
its contents as an accessible object.  This is necessary whenever you
call @code{Feval} or anything that can directly or indirectly call
@code{Feval} (this includes the @code{QUIT} macro!).  At such a time,
any Lisp object that you intend to refer to again must be protected
somehow.  @code{UNGCPRO} cancels the protection of the variables that
are protected in the current function.  It is necessary to do this
explicitly.

The macro @code{GCPRO1} protects just one local variable.  If you want
to protect two, use @code{GCPRO2} instead; repeating @code{GCPRO1} will
not work.  Macros @code{GCPRO3} and @code{GCPRO4} also exist.

These macros implicitly use local variables such as @code{gcpro1}; you
must declare these explicitly, with type @code{struct gcpro}.  Thus, if
you use @code{GCPRO2}, you must declare @code{gcpro1} and @code{gcpro2}.

@cindex caller-protects (@code{GCPRO} rule)
Note also that the general rule is @dfn{caller-protects}; i.e. you are
only responsible for protecting those Lisp objects that you create.  Any
objects passed to you as arguments should have been protected by whoever
created them, so you don't in general have to protect them.

In particular, the arguments to any Lisp primitive are always
automatically @code{GCPRO}ed, when called ``normally'' from Lisp code or
bytecode.  So only a few Lisp primitives that are called frequently from
C code, such as @code{Fprogn} protect their arguments as a service to
their caller.  You don't need to protect your arguments when writing a
new @code{DEFUN}.

@code{GCPRO}ing is perhaps the trickiest and most error-prone part of
XEmacs coding.  It is @strong{extremely} important that you get this
right and use a great deal of discipline when writing this code.
@xref{GCPROing, ,@code{GCPRO}ing}, for full details on how to do this.

What @code{DEFUN} actually does is declare a global structure of type
@code{Lisp_Subr} whose name begins with capital @samp{SF} and which
contains information about the primitive (e.g. a pointer to the
function, its minimum and maximum allowed arguments, a string describing
its Lisp name); @code{DEFUN} then begins a normal C function declaration
using the @code{F...} name.  The Lisp subr object that is the function
definition of a primitive (i.e. the object in the function slot of the
symbol that names the primitive) actually points to this @samp{SF}
structure; when @code{Feval} encounters a subr, it looks in the
structure to find out how to call the C function.

Defining the C function is not enough to make a Lisp primitive
available; you must also create the Lisp symbol for the primitive (the
symbol is @dfn{interned}; @pxref{Obarrays}) and store a suitable subr
object in its function cell. (If you don't do this, the primitive won't
be seen by Lisp code.) The code looks like this:

@example
DEFSUBR (@var{fname});
@end example

@noindent
Here @var{fname} is the same name you used as the second argument to
@code{DEFUN}.

This call to @code{DEFSUBR} should go in the @code{syms_of_*()} function
at the end of the module.  If no such function exists, create it and
make sure to also declare it in @file{symsinit.h} and call it from the
appropriate spot in @code{main()}.  @xref{General Coding Rules}.

Note that C code cannot call functions by name unless they are defined
in C.  The way to call a function written in Lisp from C is to use
@code{Ffuncall}, which embodies the Lisp function @code{funcall}.  Since
the Lisp function @code{funcall} accepts an unlimited number of
arguments, in C it takes two: the number of Lisp-level arguments, and a
one-dimensional array containing their values.  The first Lisp-level
argument is the Lisp function to call, and the rest are the arguments to
pass to it.  Since @code{Ffuncall} can call the evaluator, you must
protect pointers from garbage collection around the call to
@code{Ffuncall}. (However, @code{Ffuncall} explicitly protects all of
its parameters, so you don't have to protect any pointers passed as
parameters to it.)

The C functions @code{call0}, @code{call1}, @code{call2}, and so on,
provide handy ways to call a Lisp function conveniently with a fixed
number of arguments.  They work by calling @code{Ffuncall}.

@file{eval.c} is a very good file to look through for examples;
@file{lisp.h} contains the definitions for important macros and
functions.

@node Adding Global Lisp Variables
@section Adding Global Lisp Variables

Global variables whose names begin with @samp{Q} are constants whose
value is a symbol of a particular name.  The name of the variable should
be derived from the name of the symbol using the same rules as for Lisp
primitives.  These variables are initialized using a call to
@code{defsymbol()} in the @code{syms_of_*()} function. (This call
interns a symbol, sets the C variable to the resulting Lisp object, and
calls @code{staticpro()} on the C variable to tell the
garbage-collection mechanism about this variable.  What
@code{staticpro()} does is add a pointer to the variable to a large
global array; when garbage-collection happens, all pointers listed in
the array are used as starting points for marking Lisp objects.  This is
important because it's quite possible that the only current reference to
the object is the C variable.  In the case of symbols, the
@code{staticpro()} doesn't matter all that much because the symbol is
contained in @code{obarray}, which is itself @code{staticpro()}ed.
However, it's possible that a naughty user could do something like
uninterning the symbol out of @code{obarray} or even setting
@code{obarray} to a different value [although this is likely to make
XEmacs crash!].)

  @strong{Please note:} It is potentially deadly if you declare a
@samp{Q...}  variable in two different modules.  The two calls to
@code{defsymbol()} are no problem, but some linkers will complain about
multiply-defined symbols.  The most insidious aspect of this is that
often the link will succeed anyway, but then the resulting executable
will sometimes crash in obscure ways during certain operations!  To
avoid this problem, declare any symbols with common names (such as
@code{text}) that are not obviously associated with this particular
module in the module @file{general.c}.

  Global variables whose names begin with @samp{V} are variables that
contain Lisp objects.  The convention here is that all global variables
of type @code{Lisp_Object} begin with @samp{V}, and all others don't
(including integer and boolean variables that have Lisp
equivalents). Most of the time, these variables have equivalents in
Lisp, but some don't.  Those that do are declared this way by a call to
@code{DEFVAR_LISP()} in the @code{vars_of_*()} initializer for the
module.  What this does is create a special @dfn{symbol-value-forward}
Lisp object that contains a pointer to the C variable, intern a symbol
whose name is as specified in the call to @code{DEFVAR_LISP()}, and set
its value to the symbol-value-forward Lisp object; it also calls
@code{staticpro()} on the C variable to tell the garbage-collection
mechanism about the variable.  When @code{eval} (or actually
@code{symbol-value}) encounters this special object in the process of
retrieving a variable's value, it follows the indirection to the C
variable and gets its value.  @code{setq} does similar things so that
the C variable gets changed.

  Whether or not you @code{DEFVAR_LISP()} a variable, you need to
initialize it in the @code{vars_of_*()} function; otherwise it will end
up as all zeroes, which is the integer 0 (@emph{not} @code{nil}), and
this is probably not what you want.  Also, if the variable is not
@code{DEFVAR_LISP()}ed, @strong{you must call} @code{staticpro()} on the
C variable in the @code{vars_of_*()} function.  Otherwise, the
garbage-collection mechanism won't know that the object in this variable
is in use, and will happily collect it and reuse its storage for another
Lisp object, and you will be the one who's unhappy when you can't figure
out how your variable got overwritten.

@node Coding for Mule
@section Coding for Mule
@cindex Coding for Mule

Although Mule support is not compiled by default in XEmacs, many people
are using it, and we consider it crucial that new code works correctly
with multibyte characters.  This is not hard; it is only a matter of
following several simple user-interface guidelines.  Even if you never
compile with Mule, with a little practice you will find it quite easy
to code Mule-correctly.

Note that these guidelines are not necessarily tied to the current Mule
implementation; they are also a good idea to follow on the grounds of
code generalization for future I18N work.

@menu
* Character-Related Data Types::
* Working With Character and Byte Positions::
* Conversion to and from External Data::
* General Guidelines for Writing Mule-Aware Code::
* An Example of Mule-Aware Code::
@end menu

@node Character-Related Data Types
@subsection Character-Related Data Types

First, let's review the basic character-related datatypes used by
XEmacs.  Note that the separate @code{typedef}s are not mandatory in the
current implementation (all of them boil down to @code{unsigned char} or
@code{int}), but they improve clarity of code a great deal, because one
glance at the declaration can tell the intended use of the variable.

@table @code
@item Emchar
@cindex Emchar
An @code{Emchar} holds a single Emacs character.

Obviously, the equality between characters and bytes is lost in the Mule
world.  Characters can be represented by one or more bytes in the
buffer, and @code{Emchar} is the C type large enough to hold any
character.

Without Mule support, an @code{Emchar} is equivalent to an
@code{unsigned char}.

@item Bufbyte
@cindex Bufbyte
The data representing the text in a buffer or string is logically a set
of @code{Bufbyte}s.

XEmacs does not work with character formats all the time; when reading
characters from the outside, it decodes them to an internal format, and
likewise encodes them when writing.  @code{Bufbyte} (in fact
@code{unsigned char}) is the basic unit of XEmacs internal buffers and
strings format.

One character can correspond to one or more @code{Bufbyte}s.  In the
current implementation, an ASCII character is represented by the same
@code{Bufbyte}, and extended characters are represented by a sequence of
@code{Bufbyte}s.

Without Mule support, a @code{Bufbyte} is equivalent to an
@code{Emchar}.

@item Bufpos
@itemx Charcount
@cindex Bufpos
@cindex Charcount
A @code{Bufpos} represents a character position in a buffer or string.
A @code{Charcount} represents a number (count) of characters.
Logically, subtracting two @code{Bufpos} values yields a
@code{Charcount} value.  Although all of these are @code{typedef}ed to
@code{int}, we use them in preference to @code{int} to make it clear
what sort of position is being used.

@code{Bufpos} and @code{Charcount} values are the only ones that are
ever visible to Lisp.

@item Bytind
@itemx Bytecount
@cindex Bytind
@cindex Bytecount
A @code{Bytind} represents a byte position in a buffer or string.  A
@code{Bytecount} represents the distance between two positions in bytes.
The relationship between @code{Bytind} and @code{Bytecount} is the same
as the relationship between @code{Bufpos} and @code{Charcount}.

@item Extbyte
@itemx Extcount
@cindex Extbyte
@cindex Extcount
When dealing with the outside world, XEmacs works with @code{Extbyte}s,
which are equivalent to @code{unsigned char}.  Obviously, an
@code{Extcount} is the distance between two @code{Extbyte}s.  Extbytes
and Extcounts are not all that frequent in XEmacs code.
@end table

@node Working With Character and Byte Positions
@subsection Working With Character and Byte Positions

Now that we have defined the basic character-related types, we can look
at the macros and functions designed for work with them and for
conversion between them.  Most of these macros are defined in
@file{buffer.h}, and we don't discuss all of them here, but only the
most important ones.  Examining the existing code is the best way to
learn about them.

@table @code
@item MAX_EMCHAR_LEN
@cindex MAX_EMCHAR_LEN
This preprocessor constant is the maximum number of buffer bytes per
Emacs character, i.e. the byte length of an @code{Emchar}.  It is useful
when allocating temporary strings to keep a known number of characters.
For instance:

@example
@group
@{
  Charcount cclen;
  ...
  @{
    /* Allocate place for @var{cclen} characters. */
    Bufbyte *buf = (Bufbyte *)alloca (cclen * MAX_EMCHAR_LEN);
...
@end group
@end example

If you followed the previous section, you can guess that, logically,
multiplying a @code{Charcount} value with @code{MAX_EMCHAR_LEN} produces
a @code{Bytecount} value.

In the current Mule implementation, @code{MAX_EMCHAR_LEN} equals 4.
Without Mule, it is 1.

@item charptr_emchar
@itemx set_charptr_emchar
@cindex charptr_emchar
@cindex set_charptr_emchar
The @code{charptr_emchar} macro takes a @code{Bufbyte} pointer and
returns the @code{Emchar} stored at that position.  If it were a
function, its prototype would be:

@example
Emchar charptr_emchar (Bufbyte *p);
@end example

@code{set_charptr_emchar} stores an @code{Emchar} to the specified byte
position.  It returns the number of bytes stored:

@example
Bytecount set_charptr_emchar (Bufbyte *p, Emchar c);
@end example

It is important to note that @code{set_charptr_emchar} is safe only for
appending a character at the end of a buffer, not for overwriting a
character in the middle.  This is because the width of characters
varies, and @code{set_charptr_emchar} cannot resize the string if it
writes, say, a two-byte character where a single-byte character used to
reside.

A typical use of @code{set_charptr_emchar} can be demonstrated by this
example, which copies characters from buffer @var{buf} to a temporary
string of Bufbytes.

@example
@group
@{
  Bufpos pos;
  for (pos = beg; pos < end; pos++)
    @{
      Emchar c = BUF_FETCH_CHAR (buf, pos);
      p += set_charptr_emchar (buf, c);
    @}
@}
@end group
@end example

Note how @code{set_charptr_emchar} is used to store the @code{Emchar}
and increment the counter, at the same time.

@item INC_CHARPTR
@itemx DEC_CHARPTR
@cindex INC_CHARPTR
@cindex DEC_CHARPTR
These two macros increment and decrement a @code{Bufbyte} pointer,
respectively.  They will adjust the pointer by the appropriate number of
bytes according to the byte length of the character stored there.  Both
macros assume that the memory address is located at the beginning of a
valid character.

Without Mule support, @code{INC_CHARPTR (p)} and @code{DEC_CHARPTR (p)}
simply expand to @code{p++} and @code{p--}, respectively.

@item bytecount_to_charcount
@cindex bytecount_to_charcount
Given a pointer to a text string and a length in bytes, return the
equivalent length in characters.

@example
Charcount bytecount_to_charcount (Bufbyte *p, Bytecount bc);
@end example

@item charcount_to_bytecount
@cindex charcount_to_bytecount
Given a pointer to a text string and a length in characters, return the
equivalent length in bytes.

@example
Bytecount charcount_to_bytecount (Bufbyte *p, Charcount cc);
@end example

@item charptr_n_addr
@cindex charptr_n_addr
Return a pointer to the beginning of the character offset @var{cc} (in
characters) from @var{p}.

@example
Bufbyte *charptr_n_addr (Bufbyte *p, Charcount cc);
@end example
@end table

@node Conversion to and from External Data
@subsection Conversion to and from External Data

When an external function, such as a C library function, returns a
@code{char} pointer, you should almost never treat it as @code{Bufbyte}.
This is because these returned strings may contain 8bit characters which
can be misinterpreted by XEmacs, and cause a crash.  Likewise, when
exporting a piece of internal text to the outside world, you should
always convert it to an appropriate external encoding, lest the internal
stuff (such as the infamous \201 characters) leak out.

The interface to conversion between the internal and external
representations of text are the numerous conversion macros defined in
@file{buffer.h}.  Before looking at them, we'll look at the external
formats supported by these macros.

Currently meaningful formats are @code{FORMAT_BINARY},
@code{FORMAT_FILENAME}, @code{FORMAT_OS}, and @code{FORMAT_CTEXT}.  Here
is a description of these.

@table @code
@item FORMAT_BINARY
Binary format.  This is the simplest format and is what we use in the
absence of a more appropriate format.  This converts according to the
@code{binary} coding system:

@enumerate a
@item
On input, bytes 0--255 are converted into characters 0--255.
@item
On output, characters 0--255 are converted into bytes 0--255 and other
characters are converted into `X'.
@end enumerate

@item FORMAT_FILENAME
Format used for filenames.  In the original Mule, this is user-definable
with the @code{pathname-coding-system} variable.  For the moment, we
just use the @code{binary} coding system.

@item FORMAT_OS
Format used for the external Unix environment---@code{argv[]}, stuff
from @code{getenv()}, stuff from the @file{/etc/passwd} file, etc.

Perhaps should be the same as FORMAT_FILENAME.

@item FORMAT_CTEXT
Compound--text format.  This is the standard X format used for data
stored in properties, selections, and the like.  This is an 8-bit
no-lock-shift ISO2022 coding system.
@end table

The macros to convert between these formats and the internal format, and
vice versa, follow.

@table @code
@item GET_CHARPTR_INT_DATA_ALLOCA
@itemx GET_CHARPTR_EXT_DATA_ALLOCA
These two are the most basic conversion macros.
@code{GET_CHARPTR_INT_DATA_ALLOCA} converts external data to internal
format, and @code{GET_CHARPTR_EXT_DATA_ALLOCA} converts the other way
around.  The arguments each of these receives are @var{ptr} (pointer to
the text in external format), @var{len} (length of texts in bytes),
@var{fmt} (format of the external text), @var{ptr_out} (lvalue to which
new text should be copied), and @var{len_out} (lvalue which will be
assigned the length of the internal text in bytes).  The resulting text
is stored to a stack-allocated buffer.  If the text doesn't need
changing, these macros will do nothing, except for setting
@var{len_out}.

The macros above take many arguments which makes them unwieldy.  For
this reason, a number of convenience macros are defined with obvious
functionality, but accepting less arguments.  The general rule is that
macros with @samp{INT} in their name convert text to internal Emacs
representation, whereas the @samp{EXT} macros convert to external
representation.

@item GET_C_CHARPTR_INT_DATA_ALLOCA
@itemx GET_C_CHARPTR_EXT_DATA_ALLOCA
As their names imply, these macros work on C char pointers, which are
zero-terminated, and thus do not need @var{len} or @var{len_out}
parameters.

@item GET_STRING_EXT_DATA_ALLOCA
@itemx GET_C_STRING_EXT_DATA_ALLOCA
These two macros convert a Lisp string into an external representation.
The difference between them is that @code{GET_STRING_EXT_DATA_ALLOCA}
stores its output to a generic string, providing @var{len_out}, the
length of the resulting external string.  On the other hand,
@code{GET_C_STRING_EXT_DATA_ALLOCA} assumes that the caller will be
satisfied with output string being zero-terminated.

Note that for Lisp strings only one conversion direction makes sense.

@item GET_C_CHARPTR_EXT_BINARY_DATA_ALLOCA
@itemx GET_CHARPTR_EXT_BINARY_DATA_ALLOCA
@itemx GET_STRING_BINARY_DATA_ALLOCA
@itemx GET_C_STRING_BINARY_DATA_ALLOCA
@itemx GET_C_CHARPTR_EXT_FILENAME_DATA_ALLOCA
@itemx ...
These macros convert internal text to a specific external
representation, with the external format being encoded into the name of
the macro.  Note that the @code{GET_STRING_...} and
@code{GET_C_STRING...}  macros lack the @samp{EXT} tag, because they
only make sense in that direction.

@item GET_C_CHARPTR_INT_BINARY_DATA_ALLOCA
@itemx GET_CHARPTR_INT_BINARY_DATA_ALLOCA
@itemx GET_C_CHARPTR_INT_FILENAME_DATA_ALLOCA
@itemx ...
These macros convert external text of a specific format to its internal
representation, with the external format being incoded into the name of
the macro.
@end table

@node General Guidelines for Writing Mule-Aware Code
@subsection General Guidelines for Writing Mule-Aware Code

This section contains some general guidance on how to write Mule-aware
code, as well as some pitfalls you should avoid.

@table @emph
@item Never use @code{char} and @code{char *}.
In XEmacs, the use of @code{char} and @code{char *} is almost always a
mistake.  If you want to manipulate an Emacs character from ``C'', use
@code{Emchar}.  If you want to examine a specific octet in the internal
format, use @code{Bufbyte}.  If you want a Lisp-visible character, use a
@code{Lisp_Object} and @code{make_char}.  If you want a pointer to move
through the internal text, use @code{Bufbyte *}.  Also note that you
almost certainly do not need @code{Emchar *}.

@item Be careful not to confuse @code{Charcount}, @code{Bytecount}, and @code{Bufpos}.
The whole point of using different types is to avoid confusion about the
use of certain variables.  Lest this effect be nullified, you need to be
careful about using the right types.

@item Always convert external data
It is extremely important to always convert external data, because
XEmacs can crash if unexpected 8bit sequences are copied to its internal
buffers literally.

This means that when a system function, such as @code{readdir}, returns
a string, you need to convert it using one of the conversion macros
described in the previous chapter, before passing it further to Lisp.
In the case of @code{readdir}, you would use the
@code{GET_C_CHARPTR_INT_FILENAME_DATA_ALLOCA} macro.

Also note that many internal functions, such as @code{make_string},
accept Bufbytes, which removes the need for them to convert the data
they receive.  This increases efficiency because that way external data
needs to be decoded only once, when it is read.  After that, it is
passed around in internal format.
@end table

@node An Example of Mule-Aware Code
@subsection An Example of Mule-Aware Code

As an example of Mule-aware code, we shall will analyze the
@code{string} function, which conses up a Lisp string from the character
arguments it receives.  Here is the definition, pasted from
@code{alloc.c}:

@example
@group
DEFUN ("string", Fstring, 0, MANY, 0, /*
Concatenate all the argument characters and make the result a string.
*/
       (int nargs, Lisp_Object *args))
@{
  Bufbyte *storage = alloca_array (Bufbyte, nargs * MAX_EMCHAR_LEN);
  Bufbyte *p = storage;

  for (; nargs; nargs--, args++)
    @{
      Lisp_Object lisp_char = *args;
      CHECK_CHAR_COERCE_INT (lisp_char);
      p += set_charptr_emchar (p, XCHAR (lisp_char));
    @}
  return make_string (storage, p - storage);
@}
@end group
@end example

Now we can analyze the source line by line.

Obviously, string will be as long as there are arguments to the
function.  This is why we allocate @code{MAX_EMCHAR_LEN} * @var{nargs}
bytes on the stack, i.e. the worst-case number of bytes for @var{nargs}
@code{Emchar}s to fit in the string.

Then, the loop checks that each element is a character, converting
integers in the process.  Like many other functions in XEmacs, this
function silently accepts integers where characters are expected, for
historical and compatibility reasons.  Unless you know what you are
doing, @code{CHECK_CHAR} will also suffice.  @code{XCHAR (lisp_char)}
extracts the @code{Emchar} from the @code{Lisp_Object}, and
@code{set_charptr_emchar} stores it to storage, increasing @code{p} in
the process.

Other instructive examples of correct coding under Mule can be found all
over the XEmacs code.  For starters, I recommend
@code{Fnormalize_menu_item_name} in @file{menubar.c}.  After you have
understood this section of the manual and studied the examples, you can
proceed writing new Mule-aware code.

@node Techniques for XEmacs Developers
@section Techniques for XEmacs Developers

To make a quantified XEmacs, do: @code{make quantmacs}.

You simply can't dump Quantified and Purified images.  Run the image
like so:  @code{quantmacs -batch -l loadup.el run-temacs @var{xemacs-args...}}.

Before you go through the trouble, are you compiling with all
debugging and error-checking off?  If not try that first.  Be warned
that while Quantify is directly responsible for quite a few
optimizations which have been made to XEmacs, doing a run which
generates results which can be acted upon is not necessarily a trivial
task.

Also, if you're still willing to do some runs make sure you configure
with the @samp{--quantify} flag.  That will keep Quantify from starting
to record data until after the loadup is completed and will shut off
recording right before it shuts down (which generates enough bogus data
to throw most results off).  It also enables three additional elisp
commands: @code{quantify-start-recording-data},
@code{quantify-stop-recording-data} and @code{quantify-clear-data}.

If you want to make XEmacs faster, target your favorite slow benchmark,
run a profiler like Quantify, @code{gprof}, or @code{tcov}, and figure
out where the cycles are going.  Specific projects:

@itemize @bullet
@item
Make the garbage collector faster.  Figure out how to write an
incremental garbage collector.
@item
Write a compiler that takes bytecode and spits out C code.
Unfortunately, you will then need a C compiler and a more fully
developed module system.
@item
Speed up redisplay.
@item
Speed up syntax highlighting.  Maybe moving some of the syntax
highlighting capabilities into C would make a difference.
@item
Implement tail recursion in Emacs Lisp (hard!).
@end itemize

Unfortunately, Emacs Lisp is slow, and is going to stay slow.  Function
calls in elisp are especially expensive.  Iterating over a long list is
going to be 30 times faster implemented in C than in Elisp.

To get started debugging XEmacs, take a look at the @file{gdbinit} and
@file{dbxrc} files in the @file{src} directory.
@xref{Q2.1.15 - How to Debug an XEmacs problem with a debugger,,,
xemacs-faq, XEmacs FAQ}.

After making source code changes, run @code{make check} to ensure that
you haven't introduced any regressions.  If you're feeling ambitious,
you can try to improve the test suite in @file{tests/automated}.

Here are things to know when you create a new source file:

@itemize @bullet
@item
All @file{.c} files should @code{#include <config.h>} first.  Almost all
@file{.c} files should @code{#include "lisp.h"} second.

@item
Generated header files should be included using the @code{#include <...>} syntax,
not the @code{#include "..."} syntax.  The generated headers are:

@file{config.h puresize-adjust.h sheap-adjust.h paths.h Emacs.ad.h}

The basic rule is that you should assume builds using @code{--srcdir}
and the @code{#include <...>} syntax needs to be used when the
to-be-included generated file is in a potentially different directory
@emph{at compile time}.  The non-obvious C rule is that @code{#include "..."}
means to search for the included file in the same directory as the
including file, @emph{not} in the current directory.

@item
Header files should @emph{not} include @code{<config.h>} and
@code{"lisp.h"}.  It is the responsibility of the @file{.c} files that
use it to do so.

@item
If the header uses @code{INLINE}, either directly or through
@code{DECLARE_LRECORD}, then it must be added to @file{inline.c}'s
includes.

@item
Try compiling at least once with

@example
gcc --with-mule --with-union-type --error-checking=all
@end example

@item
Did I mention that you should run the test suite?
@example
make check
@end example
@end itemize


@node A Summary of the Various XEmacs Modules, Allocation of Objects in XEmacs Lisp, Rules When Writing New C Code, Top
@chapter A Summary of the Various XEmacs Modules

  This is accurate as of XEmacs 20.0.

@menu
* Low-Level Modules::
* Basic Lisp Modules::
* Modules for Standard Editing Operations::
* Editor-Level Control Flow Modules::
* Modules for the Basic Displayable Lisp Objects::
* Modules for other Display-Related Lisp Objects::
* Modules for the Redisplay Mechanism::
* Modules for Interfacing with the File System::
* Modules for Other Aspects of the Lisp Interpreter and Object System::
* Modules for Interfacing with the Operating System::
* Modules for Interfacing with X Windows::
* Modules for Internationalization::
@end menu

@node Low-Level Modules
@section Low-Level Modules

@example
config.h
@end example

This is automatically generated from @file{config.h.in} based on the
results of configure tests and user-selected optional features and
contains preprocessor definitions specifying the nature of the
environment in which XEmacs is being compiled.



@example
paths.h
@end example

This is automatically generated from @file{paths.h.in} based on supplied
configure values, and allows for non-standard installed configurations
of the XEmacs directories.  It's currently broken, though.



@example
emacs.c
signal.c
@end example

@file{emacs.c} contains @code{main()} and other code that performs the most
basic environment initializations and handles shutting down the XEmacs
process (this includes @code{kill-emacs}, the normal way that XEmacs is
exited; @code{dump-emacs}, which is used during the build process to
write out the XEmacs executable; @code{run-emacs-from-temacs}, which can
be used to start XEmacs directly when temacs has finished loading all
the Lisp code; and emergency code to handle crashes [XEmacs tries to
auto-save all files before it crashes]).

Low-level code that directly interacts with the Unix signal mechanism,
however, is in @file{signal.c}.  Note that this code does not handle system
dependencies in interfacing to signals; that is handled using the
@file{syssignal.h} header file, described in section J below.



@example
unexaix.c
unexalpha.c
unexapollo.c
unexconvex.c
unexec.c
unexelf.c
unexelfsgi.c
unexencap.c
unexenix.c
unexfreebsd.c
unexfx2800.c
unexhp9k3.c
unexhp9k800.c
unexmips.c
unexnext.c
unexsol2.c
unexsunos4.c
@end example

These modules contain code dumping out the XEmacs executable on various
different systems. (This process is highly machine-specific and
requires intimate knowledge of the executable format and the memory map
of the process.) Only one of these modules is actually used; this is
chosen by @file{configure}.



@example
crt0.c
lastfile.c
pre-crt0.c
@end example

These modules are used in conjunction with the dump mechanism.  On some
systems, an alternative version of the C startup code (the actual code
that receives control from the operating system when the process is
started, and which calls @code{main()}) is required so that the dumping
process works properly; @file{crt0.c} provides this.

@file{pre-crt0.c} and @file{lastfile.c} should be the very first and
very last file linked, respectively. (Actually, this is not really true.
@file{lastfile.c} should be after all Emacs modules whose initialized
data should be made constant, and before all other Emacs files and all
libraries.  In particular, the allocation modules @file{gmalloc.c},
@file{alloca.c}, etc. are normally placed past @file{lastfile.c}, and
all of the files that implement Xt widget classes @emph{must} be placed
after @file{lastfile.c} because they contain various structures that
must be statically initialized and into which Xt writes at various
times.) @file{pre-crt0.c} and @file{lastfile.c} contain exported symbols
that are used to determine the start and end of XEmacs' initialized
data space when dumping.



@example
alloca.c
free-hook.c
getpagesize.h
gmalloc.c
malloc.c
mem-limits.h
ralloc.c
vm-limit.c
@end example

These handle basic C allocation of memory.  @file{alloca.c} is an emulation of
the stack allocation function @code{alloca()} on machines that lack
this. (XEmacs makes extensive use of @code{alloca()} in its code.)

@file{gmalloc.c} and @file{malloc.c} are two implementations of the standard C
functions @code{malloc()}, @code{realloc()} and @code{free()}.  They are
often used in place of the standard system-provided @code{malloc()}
because they usually provide a much faster implementation, at the
expense of additional memory use.  @file{gmalloc.c} is a newer implementation
that is much more memory-efficient for large allocations than @file{malloc.c},
and should always be preferred if it works. (At one point, @file{gmalloc.c}
didn't work on some systems where @file{malloc.c} worked; but this should be
fixed now.)

@cindex relocating allocator
@file{ralloc.c} is the @dfn{relocating allocator}.  It provides
functions similar to @code{malloc()}, @code{realloc()} and @code{free()}
that allocate memory that can be dynamically relocated in memory.  The
advantage of this is that allocated memory can be shuffled around to
place all the free memory at the end of the heap, and the heap can then
be shrunk, releasing the memory back to the operating system.  The use
of this can be controlled with the configure option @code{--rel-alloc};
if enabled, memory allocated for buffers will be relocatable, so that if
a very large file is visited and the buffer is later killed, the memory
can be released to the operating system.  (The disadvantage of this
mechanism is that it can be very slow.  On systems with the
@code{mmap()} system call, the XEmacs version of @file{ralloc.c} uses
this to move memory around without actually having to block-copy it,
which can speed things up; but it can still cause noticeable performance
degradation.)

@file{free-hook.c} contains some debugging functions for checking for invalid
arguments to @code{free()}.

@file{vm-limit.c} contains some functions that warn the user when memory is
getting low.  These are callback functions that are called by @file{gmalloc.c}
and @file{malloc.c} at appropriate times.

@file{getpagesize.h} provides a uniform interface for retrieving the size of a
page in virtual memory.  @file{mem-limits.h} provides a uniform interface for
retrieving the total amount of available virtual memory.  Both are
similar in spirit to the @file{sys*.h} files described in section J, below.



@example
blocktype.c
blocktype.h
dynarr.c
@end example

These implement a couple of basic C data types to facilitate memory
allocation.  The @code{Blocktype} type efficiently manages the
allocation of fixed-size blocks by minimizing the number of times that
@code{malloc()} and @code{free()} are called.  It allocates memory in
large chunks, subdivides the chunks into blocks of the proper size, and
returns the blocks as requested.  When blocks are freed, they are placed
onto a linked list, so they can be efficiently reused.  This data type
is not much used in XEmacs currently, because it's a fairly new
addition.

@cindex dynamic array
The @code{Dynarr} type implements a @dfn{dynamic array}, which is
similar to a standard C array but has no fixed limit on the number of
elements it can contain.  Dynamic arrays can hold elements of any type,
and when you add a new element, the array automatically resizes itself
if it isn't big enough.  Dynarrs are extensively used in the redisplay
mechanism.



@example
inline.c
@end example

This module is used in connection with inline functions (available in
some compilers).  Often, inline functions need to have a corresponding
non-inline function that does the same thing.  This module is where they
reside.  It contains no actual code, but defines some special flags that
cause inline functions defined in header files to be rendered as actual
functions.  It then includes all header files that contain any inline
function definitions, so that each one gets a real function equivalent.



@example
debug.c
debug.h
@end example

These functions provide a system for doing internal consistency checks
during code development.  This system is not currently used; instead the
simpler @code{assert()} macro is used along with the various checks
provided by the @samp{--error-check-*} configuration options.



@example
prefix-args.c
@end example

This is actually the source for a small, self-contained program
used during building.


@example
universe.h
@end example

This is not currently used.



@node Basic Lisp Modules
@section Basic Lisp Modules

@example
emacsfns.h
lisp-disunion.h
lisp-union.h
lisp.h
lrecord.h
symsinit.h
@end example

These are the basic header files for all XEmacs modules.  Each module
includes @file{lisp.h}, which brings the other header files in.
@file{lisp.h} contains the definitions of the structures and extractor
and constructor macros for the basic Lisp objects and various other
basic definitions for the Lisp environment, as well as some
general-purpose definitions (e.g. @code{min()} and @code{max()}).
@file{lisp.h} includes either @file{lisp-disunion.h} or
@file{lisp-union.h}, depending on whether @code{USE_UNION_TYPE} is
defined.  These files define the typedef of the Lisp object itself (as
described above) and the low-level macros that hide the actual
implementation of the Lisp object.  All extractor and constructor macros
for particular types of Lisp objects are defined in terms of these
low-level macros.

As a general rule, all typedefs should go into the typedefs section of
@file{lisp.h} rather than into a module-specific header file even if the
structure is defined elsewhere.  This allows function prototypes that
use the typedef to be placed into other header files.  Forward structure
declarations (i.e. a simple declaration like @code{struct foo;} where
the structure itself is defined elsewhere) should be placed into the
typedefs section as necessary.

@file{lrecord.h} contains the basic structures and macros that implement
all record-type Lisp objects -- i.e. all objects whose type is a field
in their C structure, which includes all objects except the few most
basic ones.

@file{lisp.h} contains prototypes for most of the exported functions in
the various modules.  Lisp primitives defined using @code{DEFUN} that
need to be called by C code should be declared using @code{EXFUN}.
Other function prototypes should be placed either into the appropriate
section of @code{lisp.h}, or into a module-specific header file,
depending on how general-purpose the function is and whether it has
special-purpose argument types requiring definitions not in
@file{lisp.h}.)  All initialization functions are prototyped in
@file{symsinit.h}.



@example
alloc.c
pure.c
puresize.h
@end example

The large module @file{alloc.c} implements all of the basic allocation and
garbage collection for Lisp objects.  The most commonly used Lisp
objects are allocated in chunks, similar to the Blocktype data type
described above; others are allocated in individually @code{malloc()}ed
blocks.  This module provides the foundation on which all other aspects
of the Lisp environment sit, and is the first module initialized at
startup.

Note that @file{alloc.c} provides a series of generic functions that are
not dependent on any particular object type, and interfaces to
particular types of objects using a standardized interface of
type-specific methods.  This scheme is a fundamental principle of
object-oriented programming and is heavily used throughout XEmacs.  The
great advantage of this is that it allows for a clean separation of
functionality into different modules -- new classes of Lisp objects, new
event interfaces, new device types, new stream interfaces, etc. can be
added transparently without affecting code anywhere else in XEmacs.
Because the different subsystems are divided into general and specific
code, adding a new subtype within a subsystem will in general not
require changes to the generic subsystem code or affect any of the other
subtypes in the subsystem; this provides a great deal of robustness to
the XEmacs code.

@cindex pure space
@file{pure.c} contains the declaration of the @dfn{purespace} array.
Pure space is a hack used to place some constant Lisp data into the code
segment of the XEmacs executable, even though the data needs to be
initialized through function calls.  (See above in section VIII for more
info about this.)  During startup, certain sorts of data is
automatically copied into pure space, and other data is copied manually
in some of the basic Lisp files by calling the function @code{purecopy},
which copies the object if possible (this only works in temacs, of
course) and returns the new object.  In particular, while temacs is
executing, the Lisp reader automatically copies all compiled-function
objects that it reads into pure space.  Since compiled-function objects
are large, are never modified, and typically comprise the majority of
the contents of a compiled-Lisp file, this works well.  While XEmacs is
running, any attempt to modify an object that resides in pure space
causes an error.  Objects in pure space are never garbage collected --
almost all of the time, they're intended to be permanent, and in any
case you can't write into pure space to set the mark bits.

@file{puresize.h} contains the declaration of the size of the pure space
array.  This depends on the optional features that are compiled in, any
extra purespace requested by the user at compile time, and certain other
factors (e.g. 64-bit machines need more pure space because their Lisp
objects are larger).  The smallest size that suffices should be used, so
that there's no wasted space.  If there's not enough pure space, you
will get an error during the build process, specifying how much more
pure space is needed.



@example
eval.c
backtrace.h
@end example

This module contains all of the functions to handle the flow of control.
This includes the mechanisms of defining functions, calling functions,
traversing stack frames, and binding variables; the control primitives
and other special forms such as @code{while}, @code{if}, @code{eval},
@code{let}, @code{and}, @code{or}, @code{progn}, etc.; handling of
non-local exits, unwind-protects, and exception handlers; entering the
debugger; methods for the subr Lisp object type; etc.  It does
@emph{not} include the @code{read} function, the @code{print} function,
or the handling of symbols and obarrays.

@file{backtrace.h} contains some structures related to stack frames and the
flow of control.



@example
lread.c
@end example

This module implements the Lisp reader and the @code{read} function,
which converts text into Lisp objects, according to the read syntax of
the objects, as described above.  This is similar to the parser that is
a part of all compilers.



@example
print.c
@end example

This module implements the Lisp print mechanism and the @code{print}
function and related functions.  This is the inverse of the Lisp reader
-- it converts Lisp objects to a printed, textual representation.
(Hopefully something that can be read back in using @code{read} to get
an equivalent object.)



@example
general.c
symbols.c
symeval.h
@end example

@file{symbols.c} implements the handling of symbols, obarrays, and
retrieving the values of symbols.  Much of the code is devoted to
handling the special @dfn{symbol-value-magic} objects that define
special types of variables -- this includes buffer-local variables,
variable aliases, variables that forward into C variables, etc.  This
module is initialized extremely early (right after @file{alloc.c}),
because it is here that the basic symbols @code{t} and @code{nil} are
created, and those symbols are used everywhere throughout XEmacs.

@file{symeval.h} contains the definitions of symbol structures and the
@code{DEFVAR_LISP()} and related macros for declaring variables.



@example
data.c
floatfns.c
fns.c
@end example

These modules implement the methods and standard Lisp primitives for all
the basic Lisp object types other than symbols (which are described
above).  @file{data.c} contains all the predicates (primitives that return
whether an object is of a particular type); the integer arithmetic
functions; and the basic accessor and mutator primitives for the various
object types.  @file{fns.c} contains all the standard predicates for working
with sequences (where, abstractly speaking, a sequence is an ordered set
of objects, and can be represented by a list, string, vector, or
bit-vector); it also contains @code{equal}, perhaps on the grounds that
bulk of the operation of @code{equal} is comparing sequences.
@file{floatfns.c} contains methods and primitives for floats and floating-point
arithmetic.



@example
bytecode.c
bytecode.h
@end example

@file{bytecode.c} implements the byte-code interpreter and
compiled-function objects, and @file{bytecode.h} contains associated
structures.  Note that the byte-code @emph{compiler} is written in Lisp.




@node Modules for Standard Editing Operations
@section Modules for Standard Editing Operations

@example
buffer.c
buffer.h
bufslots.h
@end example

@file{buffer.c} implements the @dfn{buffer} Lisp object type.  This
includes functions that create and destroy buffers; retrieve buffers by
name or by other properties; manipulate lists of buffers (remember that
buffers are permanent objects and stored in various ordered lists);
retrieve or change buffer properties; etc.  It also contains the
definitions of all the built-in buffer-local variables (which can be
viewed as buffer properties).  It does @emph{not} contain code to
manipulate buffer-local variables (that's in @file{symbols.c}, described
above); or code to manipulate the text in a buffer.

@file{buffer.h} defines the structures associated with a buffer and the various
macros for retrieving text from a buffer and special buffer positions
(e.g. @code{point}, the default location for text insertion).  It also
contains macros for working with buffer positions and converting between
their representations as character offsets and as byte offsets (under
MULE, they are different, because characters can be multi-byte).  It is
one of the largest header files.

@file{bufslots.h} defines the fields in the buffer structure that correspond to
the built-in buffer-local variables.  It is its own header file because
it is included many times in @file{buffer.c}, as a way of iterating over all
the built-in buffer-local variables.



@example
insdel.c
insdel.h
@end example

@file{insdel.c} contains low-level functions for inserting and deleting text in
a buffer, keeping track of changed regions for use by redisplay, and
calling any before-change and after-change functions that may have been
registered for the buffer.  It also contains the actual functions that
convert between byte offsets and character offsets.

@file{insdel.h} contains associated headers.



@example
marker.c
@end example

This module implements the @dfn{marker} Lisp object type, which
conceptually is a pointer to a text position in a buffer that moves
around as text is inserted and deleted, so as to remain in the same
relative position.  This module doesn't actually move the markers around
-- that's handled in @file{insdel.c}.  This module just creates them and
implements the primitives for working with them.  As markers are simple
objects, this does not entail much.

Note that the standard arithmetic primitives (e.g. @code{+}) accept
markers in place of integers and automatically substitute the value of
@code{marker-position} for the marker, i.e. an integer describing the
current buffer position of the marker.



@example
extents.c
extents.h
@end example

This module implements the @dfn{extent} Lisp object type, which is like
a marker that works over a range of text rather than a single position.
Extents are also much more complex and powerful than markers and have a
more efficient (and more algorithmically complex) implementation.  The
implementation is described in detail in comments in @file{extents.c}.

The code in @file{extents.c} works closely with @file{insdel.c} so that
extents are properly moved around as text is inserted and deleted.
There is also code in @file{extents.c} that provides information needed
by the redisplay mechanism for efficient operation. (Remember that
extents can have display properties that affect [sometimes drastically,
as in the @code{invisible} property] the display of the text they
cover.)



@example
editfns.c
@end example

@file{editfns.c} contains the standard Lisp primitives for working with
a buffer's text, and calls the low-level functions in @file{insdel.c}.
It also contains primitives for working with @code{point} (the default
buffer insertion location).

@file{editfns.c} also contains functions for retrieving various
characteristics from the external environment: the current time, the
process ID of the running XEmacs process, the name of the user who ran
this XEmacs process, etc.  It's not clear why this code is in
@file{editfns.c}.



@example
callint.c
cmds.c
commands.h
@end example

@cindex interactive
These modules implement the basic @dfn{interactive} commands,
i.e. user-callable functions.  Commands, as opposed to other functions,
have special ways of getting their parameters interactively (by querying
the user), as opposed to having them passed in a normal function
invocation.  Many commands are not really meant to be called from other
Lisp functions, because they modify global state in a way that's often
undesired as part of other Lisp functions.

@file{callint.c} implements the mechanism for querying the user for
parameters and calling interactive commands.  The bulk of this module is
code that parses the interactive spec that is supplied with an
interactive command.

@file{cmds.c} implements the basic, most commonly used editing commands:
commands to move around the current buffer and insert and delete
characters.  These commands are implemented using the Lisp primitives
defined in @file{editfns.c}.

@file{commands.h} contains associated structure definitions and prototypes.



@example
regex.c
regex.h
search.c
@end example

@file{search.c} implements the Lisp primitives for searching for text in
a buffer, and some of the low-level algorithms for doing this.  In
particular, the fast fixed-string Boyer-Moore search algorithm is
implemented in @file{search.c}.  The low-level algorithms for doing
regular-expression searching, however, are implemented in @file{regex.c}
and @file{regex.h}.  These two modules are largely independent of
XEmacs, and are similar to (and based upon) the regular-expression
routines used in @file{grep} and other GNU utilities.



@example
doprnt.c
@end example

@file{doprnt.c} implements formatted-string processing, similar to
@code{printf()} command in C.



@example
undo.c
@end example

This module implements the undo mechanism for tracking buffer changes.
Most of this could be implemented in Lisp.



@node Editor-Level Control Flow Modules
@section Editor-Level Control Flow Modules

@example
event-Xt.c
event-stream.c
event-tty.c
events.c
events.h
@end example

These implement the handling of events (user input and other system
notifications).

@file{events.c} and @file{events.h} define the @dfn{event} Lisp object
type and primitives for manipulating it.

@file{event-stream.c} implements the basic functions for working with
event queues, dispatching an event by looking it up in relevant keymaps
and such, and handling timeouts; this includes the primitives
@code{next-event} and @code{dispatch-event}, as well as related
primitives such as @code{sit-for}, @code{sleep-for}, and
@code{accept-process-output}. (@file{event-stream.c} is one of the
hairiest and trickiest modules in XEmacs.  Beware!  You can easily mess
things up here.)

@file{event-Xt.c} and @file{event-tty.c} implement the low-level
interfaces onto retrieving events from Xt (the X toolkit) and from TTY's
(using @code{read()} and @code{select()}), respectively.  The event
interface enforces a clean separation between the specific code for
interfacing with the operating system and the generic code for working
with events, by defining an API of basic, low-level event methods;
@file{event-Xt.c} and @file{event-tty.c} are two different
implementations of this API.  To add support for a new operating system
(e.g. NeXTstep), one merely needs to provide another implementation of
those API functions.

Note that the choice of whether to use @file{event-Xt.c} or
@file{event-tty.c} is made at compile time!  Or at the very latest, it
is made at startup time.  @file{event-Xt.c} handles events for
@emph{both} X and TTY frames; @file{event-tty.c} is only used when X
support is not compiled into XEmacs.  The reason for this is that there
is only one event loop in XEmacs: thus, it needs to be able to receive
events from all different kinds of frames.



@example
keymap.c
keymap.h
@end example

@file{keymap.c} and @file{keymap.h} define the @dfn{keymap} Lisp object
type and associated methods and primitives. (Remember that keymaps are
objects that associate event descriptions with functions to be called to
``execute'' those events; @code{dispatch-event} looks up events in the
relevant keymaps.)



@example
keyboard.c
@end example

@file{keyboard.c} contains functions that implement the actual editor
command loop -- i.e. the event loop that cyclically retrieves and
dispatches events.  This code is also rather tricky, just like
@file{event-stream.c}.



@example
macros.c
macros.h
@end example

These two modules contain the basic code for defining keyboard macros.
These functions don't actually do much; most of the code that handles keyboard
macros is mixed in with the event-handling code in @file{event-stream.c}.



@example
minibuf.c
@end example

This contains some miscellaneous code related to the minibuffer (most of
the minibuffer code was moved into Lisp by Richard Mlynarik).  This
includes the primitives for completion (although filename completion is
in @file{dired.c}), the lowest-level interface to the minibuffer (if the
command loop were cleaned up, this too could be in Lisp), and code for
dealing with the echo area (this, too, was mostly moved into Lisp, and
the only code remaining is code to call out to Lisp or provide simple
bootstrapping implementations early in temacs, before the echo-area Lisp
code is loaded).



@node Modules for the Basic Displayable Lisp Objects
@section Modules for the Basic Displayable Lisp Objects

@example
device-ns.h
device-stream.c
device-stream.h
device-tty.c
device-tty.h
device-x.c
device-x.h
device.c
device.h
@end example

These modules implement the @dfn{device} Lisp object type.  This
abstracts a particular screen or connection on which frames are
displayed.  As with Lisp objects, event interfaces, and other
subsystems, the device code is separated into a generic component that
contains a standardized interface (in the form of a set of methods) onto
particular device types.

The device subsystem defines all the methods and provides method
services for not only device