1 \input texinfo @c -*-texinfo-*-
3 @setfilename ../../info/internals.info
4 @settitle XEmacs Internals Manual
8 @dircategory XEmacs Editor
10 * Internals: (internals). XEmacs Internals Manual.
13 Copyright @copyright{} 1992 - 1996 Ben Wing.
14 Copyright @copyright{} 1996, 1997 Sun Microsystems.
15 Copyright @copyright{} 1994 - 1998 Free Software Foundation.
16 Copyright @copyright{} 1994, 1995 Board of Trustees, University of Illinois.
19 Permission is granted to make and distribute verbatim copies of this
20 manual provided the copyright notice and this permission notice are
21 preserved on all copies.
24 Permission is granted to process this file through TeX and print the
25 results, provided the printed document carries copying permission notice
26 identical to this one except for the removal of this paragraph (this
27 paragraph not being relevant to the printed manual).
30 Permission is granted to copy and distribute modified versions of this
31 manual under the conditions for verbatim copying, provided that the
32 entire resulting derived work is distributed under the terms of a
33 permission notice identical to this one.
35 Permission is granted to copy and distribute translations of this manual
36 into another language, under the above conditions for modified versions,
37 except that this permission notice may be stated in a translation
38 approved by the Foundation.
40 Permission is granted to copy and distribute modified versions of this
41 manual under the conditions for verbatim copying, provided also that the
42 section entitled ``GNU General Public License'' is included exactly as
43 in the original, and provided that the entire resulting derived work is
44 distributed under the terms of a permission notice identical to this
47 Permission is granted to copy and distribute translations of this manual
48 into another language, under the above conditions for modified versions,
49 except that the section entitled ``GNU General Public License'' may be
50 included in a translation approved by the Free Software Foundation
51 instead of in the original English.
61 @setchapternewpage odd
65 @title XEmacs Internals Manual
66 @subtitle Version 1.3, August 1999
69 @author Martin Buchholz
71 @author Matthias Neubauer
72 @author Olivier Galibert
77 Copyright @copyright{} 1992 - 1996 Ben Wing. @*
78 Copyright @copyright{} 1996, 1997 Sun Microsystems, Inc. @*
79 Copyright @copyright{} 1994 - 1998 Free Software Foundation. @*
80 Copyright @copyright{} 1994, 1995 Board of Trustees, University of Illinois.
86 Permission is granted to make and distribute verbatim copies of this
87 manual provided the copyright notice and this permission notice are
88 preserved on all copies.
90 Permission is granted to copy and distribute modified versions of this
91 manual under the conditions for verbatim copying, provided also that the
92 section entitled ``GNU General Public License'' is included
93 exactly as in the original, and provided that the entire resulting
94 derived work is distributed under the terms of a permission notice
95 identical to this one.
97 Permission is granted to copy and distribute translations of this manual
98 into another language, under the above conditions for modified versions,
99 except that the section entitled ``GNU General Public License'' may be
100 included in a translation approved by the Free Software Foundation
101 instead of in the original English.
105 @node Top, A History of Emacs, (dir), (dir)
108 This Info file contains v1.0 of the XEmacs Internals Manual.
112 * A History of Emacs:: Times, dates, important events.
113 * XEmacs From the Outside:: A broad conceptual overview.
114 * The Lisp Language:: An overview.
115 * XEmacs From the Perspective of Building::
116 * XEmacs From the Inside::
117 * The XEmacs Object System (Abstractly Speaking)::
118 * How Lisp Objects Are Represented in C::
119 * Rules When Writing New C Code::
120 * A Summary of the Various XEmacs Modules::
121 * Allocation of Objects in XEmacs Lisp::
123 * Events and the Event Loop::
124 * Evaluation; Stack Frames; Bindings::
125 * Symbols and Variables::
126 * Buffers and Textual Representation::
127 * MULE Character Sets and Encodings::
128 * The Lisp Reader and Compiler::
130 * Consoles; Devices; Frames; Windows::
131 * The Redisplay Mechanism::
138 * Interface to X Windows::
143 --- The Detailed Node Listing ---
147 * Through Version 18:: Unification prevails.
148 * Lucid Emacs:: One version 19 Emacs.
149 * GNU Emacs 19:: The other version 19 Emacs.
150 * GNU Emacs 20:: The other version 20 Emacs.
151 * XEmacs:: The continuation of Lucid Emacs.
153 Rules When Writing New C Code
155 * General Coding Rules::
156 * Writing Lisp Primitives::
157 * Adding Global Lisp Variables::
159 * Techniques for XEmacs Developers::
163 * Character-Related Data Types::
164 * Working With Character and Byte Positions::
165 * Conversion to and from External Data::
166 * General Guidelines for Writing Mule-Aware Code::
167 * An Example of Mule-Aware Code::
169 A Summary of the Various XEmacs Modules
171 * Low-Level Modules::
172 * Basic Lisp Modules::
173 * Modules for Standard Editing Operations::
174 * Editor-Level Control Flow Modules::
175 * Modules for the Basic Displayable Lisp Objects::
176 * Modules for other Display-Related Lisp Objects::
177 * Modules for the Redisplay Mechanism::
178 * Modules for Interfacing with the File System::
179 * Modules for Other Aspects of the Lisp Interpreter and Object System::
180 * Modules for Interfacing with the Operating System::
181 * Modules for Interfacing with X Windows::
182 * Modules for Internationalization::
184 Allocation of Objects in XEmacs Lisp
186 * Introduction to Allocation::
187 * Garbage Collection::
189 * Garbage Collection - Step by Step::
190 * Integers and Characters::
191 * Allocation from Frob Blocks::
193 * Low-level allocation::
200 * Compiled Function::
202 Garbage Collection - Step by Step
205 * garbage_collect_1::
208 * sweep_lcrecords_1::
209 * compact_string_chars::
211 * sweep_bit_vectors_1::
216 * Data descriptions::
223 * Address allocation::
228 Events and the Event Loop
230 * Introduction to Events::
232 * Specifics of the Event Gathering Mechanism::
233 * Specifics About the Emacs Event::
234 * The Event Stream Callback Routines::
235 * Other Event Loop Functions::
236 * Converting Events::
237 * Dispatching Events; The Command Builder::
239 Evaluation; Stack Frames; Bindings
242 * Dynamic Binding; The specbinding Stack; Unwind-Protects::
243 * Simple Special Forms::
246 Symbols and Variables
248 * Introduction to Symbols::
252 Buffers and Textual Representation
254 * Introduction to Buffers:: A buffer holds a block of text such as a file.
255 * The Text in a Buffer:: Representation of the text in a buffer.
256 * Buffer Lists:: Keeping track of all buffers.
257 * Markers and Extents:: Tagging locations within a buffer.
258 * Bufbytes and Emchars:: Representation of individual characters.
259 * The Buffer Object:: The Lisp object corresponding to a buffer.
261 MULE Character Sets and Encodings
265 * Internal Mule Encodings::
270 * Japanese EUC (Extended Unix Code)::
273 Internal Mule Encodings
275 * Internal String Encoding::
276 * Internal Character Encoding::
280 * Creating an Lstream:: Creating an lstream object.
281 * Lstream Types:: Different sorts of things that are streamed.
282 * Lstream Functions:: Functions for working with lstreams.
283 * Lstream Methods:: Creating new lstream types.
285 Consoles; Devices; Frames; Windows
287 * Introduction to Consoles; Devices; Frames; Windows::
290 * The Window Object::
292 The Redisplay Mechanism
294 * Critical Redisplay Sections::
296 * Redisplay Piece by Piece::
300 * Introduction to Extents:: Extents are ranges over text, with properties.
301 * Extent Ordering:: How extents are ordered internally.
302 * Format of the Extent Info:: The extent information in a buffer or string.
303 * Zero-Length Extents:: A weird special case.
304 * Mathematics of Extent Ordering:: A rigorous foundation.
305 * Extent Fragments:: Cached information useful for redisplay.
310 @node A History of Emacs, XEmacs From the Outside, Top, Top
311 @chapter A History of Emacs
312 @cindex history of Emacs
313 @cindex Hackers (Steven Levy)
315 @cindex ITS (Incompatible Timesharing System)
316 @cindex Stallman, Richard
321 @cindex Free Software Foundation
323 XEmacs is a powerful, customizable text editor and development
324 environment. It began as Lucid Emacs, which was in turn derived from
325 GNU Emacs, a program written by Richard Stallman of the Free Software
326 Foundation. GNU Emacs dates back to the 1970's, and was modelled
327 after a package called ``Emacs'', written in 1976, that was a set of
328 macros on top of TECO, an old, old text editor written at MIT on the
329 DEC PDP 10 under one of the earliest time-sharing operating systems,
330 ITS (Incompatible Timesharing System). (ITS dates back well before
331 Unix.) ITS, TECO, and Emacs were products of a group of people at MIT
332 who called themselves ``hackers'', who shared an idealistic belief
333 system about the free exchange of information and were fanatical in
334 their devotion to and time spent with computers. (The hacker
335 subculture dates back to the late 1950's at MIT and is described in
336 detail in Steven Levy's book @cite{Hackers}. This book also includes
337 a lot of information about Stallman himself and the development of
338 Lisp, a programming language developed at MIT that underlies Emacs.)
341 * Through Version 18:: Unification prevails.
342 * Lucid Emacs:: One version 19 Emacs.
343 * GNU Emacs 19:: The other version 19 Emacs.
344 * GNU Emacs 20:: The other version 20 Emacs.
345 * XEmacs:: The continuation of Lucid Emacs.
348 @node Through Version 18, Lucid Emacs, A History of Emacs, A History of Emacs
349 @section Through Version 18
350 @cindex Gosling, James
351 @cindex Great Usenet Renaming
353 Although the history of the early versions of GNU Emacs is unclear,
354 the history is well-known from the middle of 1985. A time line is:
358 GNU Emacs version 15 (15.34) was released sometime in 1984 or 1985 and
359 shared some code with a version of Emacs written by James Gosling (the
360 same James Gosling who later created the Java language).
362 GNU Emacs version 16 (first released version was 16.56) was released on
363 July 15, 1985. All Gosling code was removed due to potential copyright
364 problems with the code.
366 version 16.57: released on September 16, 1985.
368 versions 16.58, 16.59: released on September 17, 1985.
370 version 16.60: released on September 19, 1985. These later version 16's
371 incorporated patches from the net, esp. for getting Emacs to work under
374 version 17.36 (first official v17 release) released on December 20,
375 1985. Included a TeX-able user manual. First official unpatched
376 version that worked on vanilla System V machines.
378 version 17.43 (second official v17 release) released on January 25,
381 version 17.45 released on January 30, 1986.
383 version 17.46 released on February 4, 1986.
385 version 17.48 released on February 10, 1986.
387 version 17.49 released on February 12, 1986.
389 version 17.55 released on March 18, 1986.
391 version 17.57 released on March 27, 1986.
393 version 17.58 released on April 4, 1986.
395 version 17.61 released on April 12, 1986.
397 version 17.63 released on May 7, 1986.
399 version 17.64 released on May 12, 1986.
401 version 18.24 (a beta version) released on October 2, 1986.
403 version 18.30 (a beta version) released on November 15, 1986.
405 version 18.31 (a beta version) released on November 23, 1986.
407 version 18.32 (a beta version) released on December 7, 1986.
409 version 18.33 (a beta version) released on December 12, 1986.
411 version 18.35 (a beta version) released on January 5, 1987.
413 version 18.36 (a beta version) released on January 21, 1987.
415 January 27, 1987: The Great Usenet Renaming. net.emacs is now
418 version 18.37 (a beta version) released on February 12, 1987.
420 version 18.38 (a beta version) released on March 3, 1987.
422 version 18.39 (a beta version) released on March 14, 1987.
424 version 18.40 (a beta version) released on March 18, 1987.
426 version 18.41 (the first ``official'' release) released on March 22,
429 version 18.45 released on June 2, 1987.
431 version 18.46 released on June 9, 1987.
433 version 18.47 released on June 18, 1987.
435 version 18.48 released on September 3, 1987.
437 version 18.49 released on September 18, 1987.
439 version 18.50 released on February 13, 1988.
441 version 18.51 released on May 7, 1988.
443 version 18.52 released on September 1, 1988.
445 version 18.53 released on February 24, 1989.
447 version 18.54 released on April 26, 1989.
449 version 18.55 released on August 23, 1989. This is the earliest version
450 that is still available by FTP.
452 version 18.56 released on January 17, 1991.
454 version 18.57 released late January, 1991.
456 version 18.58 released ?????.
458 version 18.59 released October 31, 1992.
461 @node Lucid Emacs, GNU Emacs 19, Through Version 18, A History of Emacs
468 Lucid Emacs was developed by the (now-defunct) Lucid Inc., a maker of
469 C++ and Lisp development environments. It began when Lucid decided they
470 wanted to use Emacs as the editor and cornerstone of their C++
471 development environment (called ``Energize''). They needed many features
472 that were not available in the existing version of GNU Emacs (version
473 18.5something), in particular good and integrated support for GUI
474 elements such as mouse support, multiple fonts, multiple window-system
475 windows, etc. A branch of GNU Emacs called Epoch, written at the
476 University of Illinois, existed that supplied many of these features;
477 however, Lucid needed more than what existed in Epoch. At the time, the
478 Free Software Foundation was working on version 19 of Emacs (this was
479 sometime around 1991), which was planned to have similar features, and
480 so Lucid decided to work with the Free Software Foundation. Their plan
481 was to add features that they needed, and coordinate with the FSF so
482 that the features would get included back into Emacs version 19.
484 Delays in the release of version 19 occurred, however (resulting in it
485 finally being released more than a year after what was initially
486 planned), and Lucid encountered unexpected technical resistance in
487 getting their changes merged back into version 19, so they decided to
488 release their own version of Emacs, which became Lucid Emacs 19.0.
490 @cindex Zawinski, Jamie
491 @cindex Sexton, Harlan
493 @cindex Devin, Matthieu
494 The initial authors of Lucid Emacs were Matthieu Devin, Harlan Sexton,
495 and Eric Benson, and the work was later taken over by Jamie Zawinski,
496 who became ``Mr. Lucid Emacs'' for many releases.
498 A time line for Lucid Emacs/XEmacs is
502 version 19.0 shipped with Energize 1.0, April 1992.
504 version 19.1 released June 4, 1992.
506 version 19.2 released June 19, 1992.
508 version 19.3 released September 9, 1992.
510 version 19.4 released January 21, 1993.
512 version 19.5 was a repackaging of 19.4 with a few bug fixes and
513 shipped with Energize 2.0. Never released to the net.
515 version 19.6 released April 9, 1993.
517 version 19.7 was a repackaging of 19.6 with a few bug fixes and
518 shipped with Energize 2.1. Never released to the net.
520 version 19.8 released September 6, 1993.
522 version 19.9 released January 12, 1994.
524 version 19.10 released May 27, 1994.
526 version 19.11 (first XEmacs) released September 13, 1994.
528 version 19.12 released June 23, 1995.
530 version 19.13 released September 1, 1995.
532 version 19.14 released June 23, 1996.
534 version 20.0 released February 9, 1997.
536 version 19.15 released March 28, 1997.
538 version 20.1 (not released to the net) April 15, 1997.
540 version 20.2 released May 16, 1997.
542 version 19.16 released October 31, 1997.
544 version 20.3 (the first stable version of XEmacs 20.x) released November 30,
546 version 20.4 released February 28, 1998.
549 @node GNU Emacs 19, GNU Emacs 20, Lucid Emacs, A History of Emacs
550 @section GNU Emacs 19
554 About a year after the initial release of Lucid Emacs, the FSF
555 released a beta of their version of Emacs 19 (referred to here as ``GNU
556 Emacs''). By this time, the current version of Lucid Emacs was
557 19.6. (Strangely, the first released beta from the FSF was GNU Emacs
558 19.7.) A time line for GNU Emacs version 19 is
562 version 19.8 (beta) released May 27, 1993.
564 version 19.9 (beta) released May 27, 1993.
566 version 19.10 (beta) released May 30, 1993.
568 version 19.11 (beta) released June 1, 1993.
570 version 19.12 (beta) released June 2, 1993.
572 version 19.13 (beta) released June 8, 1993.
574 version 19.14 (beta) released June 17, 1993.
576 version 19.15 (beta) released June 19, 1993.
578 version 19.16 (beta) released July 6, 1993.
580 version 19.17 (beta) released late July, 1993.
582 version 19.18 (beta) released August 9, 1993.
584 version 19.19 (beta) released August 15, 1993.
586 version 19.20 (beta) released November 17, 1993.
588 version 19.21 (beta) released November 17, 1993.
590 version 19.22 (beta) released November 28, 1993.
592 version 19.23 (beta) released May 17, 1994.
594 version 19.24 (beta) released May 16, 1994.
596 version 19.25 (beta) released June 3, 1994.
598 version 19.26 (beta) released September 11, 1994.
600 version 19.27 (beta) released September 14, 1994.
602 version 19.28 (first ``official'' release) released November 1, 1994.
604 version 19.29 released June 21, 1995.
606 version 19.30 released November 24, 1995.
608 version 19.31 released May 25, 1996.
610 version 19.32 released July 31, 1996.
612 version 19.33 released August 11, 1996.
614 version 19.34 released August 21, 1996.
616 version 19.34b released September 6, 1996.
619 @cindex Mlynarik, Richard
620 In some ways, GNU Emacs 19 was better than Lucid Emacs; in some ways,
621 worse. Lucid soon began incorporating features from GNU Emacs 19 into
622 Lucid Emacs; the work was mostly done by Richard Mlynarik, who had been
623 working on and using GNU Emacs for a long time (back as far as version
626 @node GNU Emacs 20, XEmacs, GNU Emacs 19, A History of Emacs
627 @section GNU Emacs 20
631 On February 2, 1997 work began on GNU Emacs to integrate Mule. The first
632 release was made in September of that year.
634 A timeline for Emacs 20 is
638 version 20.1 released September 17, 1997.
640 version 20.2 released September 20, 1997.
642 version 20.3 released August 19, 1998.
645 @node XEmacs, , GNU Emacs 20, A History of Emacs
649 @cindex Sun Microsystems
650 @cindex University of Illinois
651 @cindex Illinois, University of
653 @cindex Andreessen, Marc
655 @cindex Buchholz, Martin
656 @cindex Kaplan, Simon
658 @cindex Thompson, Chuck
661 @cindex Amdahl Corporation
662 Around the time that Lucid was developing Energize, Sun Microsystems
663 was developing their own development environment (called ``SPARCWorks'')
664 and also decided to use Emacs. They joined forces with the Epoch team
665 at the University of Illinois and later with Lucid. The maintainer of
666 the last-released version of Epoch was Marc Andreessen, but he dropped
667 out and the Epoch project, headed by Simon Kaplan, lured Chuck Thompson
668 away from a system administration job to become the primary Lucid Emacs
669 author for Epoch and Sun. Chuck's area of specialty became the
670 redisplay engine (he replaced the old Lucid Emacs redisplay engine with
671 a ported version from Epoch and then later rewrote it from scratch).
672 Sun also hired Ben Wing (the author of Win-Emacs, a port of Lucid Emacs
673 to Microsoft Windows 3.1) in 1993, for what was initially a one-month
674 contract to fix some event problems but later became a many-year
675 involvement, punctuated by a six-month contract with Amdahl Corporation.
677 @cindex rename to XEmacs
678 In 1994, Sun and Lucid agreed to rename Lucid Emacs to XEmacs (a name
679 not favorable to either company); the first release called XEmacs was
680 version 19.11. In June 1994, Lucid folded and Jamie quit to work for
681 the newly formed Mosaic Communications Corp., later Netscape
682 Communications Corp. (co-founded by the same Marc Andreessen, who had
683 quit his Epoch job to work on a graphical browser for the World Wide
684 Web). Chuck then become the primary maintainer of XEmacs, and put out
685 versions 19.11 through 19.14 in conjunction with Ben. For 19.12 and
686 19.13, Chuck added the new redisplay and many other display improvements
687 and Ben added MULE support (support for Asian and other languages) and
688 redesigned most of the internal Lisp subsystems to better support the
689 MULE work and the various other features being added to XEmacs. After
690 19.14 Chuck retired as primary maintainer and Steve Baur stepped in.
692 @cindex MULE merged XEmacs appears
693 Soon after 19.13 was released, work began in earnest on the MULE
694 internationalization code and the source tree was divided into two
695 development paths. The MULE version was initially called 19.20, but was
696 soon renamed to 20.0. In 1996 Martin Buchholz of Sun Microsystems took
697 over the care and feeding of it and worked on it in parallel with the
698 19.14 development that was occurring at the same time. After much work
699 by Martin, it was decided to release 20.0 ahead of 19.15 in February
700 1997. The source tree remained divided until 20.2 when the version 19
701 source was finally retired at version 19.16.
704 @cindex Buchholz, Martin
706 @cindex Niksic, Hrvoje
707 @cindex XEmacs goes it alone
708 In 1997, Sun finally dropped all pretense of support for XEmacs and
709 Martin Buchholz left the company in November. Since then, and mostly
710 for the previous year, because Steve Baur was never paid to work on
711 XEmacs, XEmacs has existed solely on the contributions of volunteers
712 from the Free Software Community. Starting from 1997, Hrvoje Niksic and
713 Kyle Jones have figured prominently in XEmacs development.
715 @cindex merging attempts
716 Many attempts have been made to merge XEmacs and GNU Emacs, but they
717 have consistently failed.
719 A more detailed history is contained in the XEmacs About page.
721 @node XEmacs From the Outside, The Lisp Language, A History of Emacs, Top
722 @chapter XEmacs From the Outside
723 @cindex read-eval-print
725 XEmacs appears to the outside world as an editor, but it is really a
726 Lisp environment. At its heart is a Lisp interpreter; it also
727 ``happens'' to contain many specialized object types (e.g. buffers,
728 windows, frames, events) that are useful for implementing an editor.
729 Some of these objects (in particular windows and frames) have
730 displayable representations, and XEmacs provides a function
731 @code{redisplay()} that ensures that the display of all such objects
732 matches their internal state. Most of the time, a standard Lisp
733 environment is in a @dfn{read-eval-print} loop---i.e. ``read some Lisp
734 code, execute it, and print the results''. XEmacs has a similar loop:
740 dispatch the event (i.e. ``do it'')
745 Reading an event is done using the Lisp function @code{next-event},
746 which waits for something to happen (typically, the user presses a key
747 or moves the mouse) and returns an event object describing this.
748 Dispatching an event is done using the Lisp function
749 @code{dispatch-event}, which looks up the event in a keymap object (a
750 particular kind of object that associates an event with a Lisp function)
751 and calls that function. The function ``does'' what the user has
752 requested by changing the state of particular frame objects, buffer
753 objects, etc. Finally, @code{redisplay()} is called, which updates the
754 display to reflect those changes just made. Thus is an ``editor'' born.
756 @cindex bridge, playing
758 @cindex pi, calculating
759 Note that you do not have to use XEmacs as an editor; you could just
760 as well make it do your taxes, compute pi, play bridge, etc. You'd just
761 have to write functions to do those operations in Lisp.
763 @node The Lisp Language, XEmacs From the Perspective of Building, XEmacs From the Outside, Top
764 @chapter The Lisp Language
767 @cindex Lisp vs. Java
768 @cindex Java vs. Lisp
769 @cindex dynamic scoping
770 @cindex scoping, dynamic
771 @cindex dynamic types
772 @cindex types, dynamic
775 @cindex Gosling, James
777 Lisp is a general-purpose language that is higher-level than C and in
778 many ways more powerful than C. Powerful dialects of Lisp such as
779 Common Lisp are probably much better languages for writing very large
780 applications than is C. (Unfortunately, for many non-technical
781 reasons C and its successor C++ have become the dominant languages for
782 application development. These languages are both inadequate for
783 extremely large applications, which is evidenced by the fact that newer,
784 larger programs are becoming ever harder to write and are requiring ever
785 more programmers despite great increases in C development environments;
786 and by the fact that, although hardware speeds and reliability have been
787 growing at an exponential rate, most software is still generally
788 considered to be slow and buggy.)
790 The new Java language holds promise as a better general-purpose
791 development language than C. Java has many features in common with
792 Lisp that are not shared by C (this is not a coincidence, since
793 Java was designed by James Gosling, a former Lisp hacker). This
794 will be discussed more later.
796 For those used to C, here is a summary of the basic differences between
801 Lisp has an extremely regular syntax. Every function, expression,
802 and control statement is written in the form
805 (@var{func} @var{arg1} @var{arg2} ...)
808 This is as opposed to C, which writes functions as
811 func(@var{arg1}, @var{arg2}, ...)
814 but writes expressions involving operators as (e.g.)
817 @var{arg1} + @var{arg2}
820 and writes control statements as (e.g.)
823 while (@var{expr}) @{ @var{statement1}; @var{statement2}; ... @}
826 Lisp equivalents of the latter two would be
829 (+ @var{arg1} @var{arg2} ...)
835 (while @var{expr} @var{statement1} @var{statement2} ...)
839 Lisp is a safe language. Assuming there are no bugs in the Lisp
840 interpreter/compiler, it is impossible to write a program that ``core
841 dumps'' or otherwise causes the machine to execute an illegal
842 instruction. This is very different from C, where perhaps the most
843 common outcome of a bug is exactly such a crash. A corollary of this is that
844 the C operation of casting a pointer is impossible (and unnecessary) in
845 Lisp, and that it is impossible to access memory outside the bounds of
849 Programs and data are written in the same form. The
850 parenthesis-enclosing form described above for statements is the same
851 form used for the most common data type in Lisp, the list. Thus, it is
852 possible to represent any Lisp program using Lisp data types, and for
853 one program to construct Lisp statements and then dynamically
854 @dfn{evaluate} them, or cause them to execute.
857 All objects are @dfn{dynamically typed}. This means that part of every
858 object is an indication of what type it is. A Lisp program can
859 manipulate an object without knowing what type it is, and can query an
860 object to determine its type. This means that, correspondingly,
861 variables and function parameters can hold objects of any type and are
862 not normally declared as being of any particular type. This is opposed
863 to the @dfn{static typing} of C, where variables can hold exactly one
864 type of object and must be declared as such, and objects do not contain
865 an indication of their type because it's implicit in the variables they
866 are stored in. It is possible in C to have a variable hold different
867 types of objects (e.g. through the use of @code{void *} pointers or
868 variable-argument functions), but the type information must then be
869 passed explicitly in some other fashion, leading to additional program
873 Allocated memory is automatically reclaimed when it is no longer in use.
874 This operation is called @dfn{garbage collection} and involves looking
875 through all variables to see what memory is being pointed to, and
876 reclaiming any memory that is not pointed to and is thus
877 ``inaccessible'' and out of use. This is as opposed to C, in which
878 allocated memory must be explicitly reclaimed using @code{free()}. If
879 you simply drop all pointers to memory without freeing it, it becomes
880 ``leaked'' memory that still takes up space. Over a long period of
881 time, this can cause your program to grow and grow until it runs out of
885 Lisp has built-in facilities for handling errors and exceptions. In C,
886 when an error occurs, usually either the program exits entirely or the
887 routine in which the error occurs returns a value indicating this. If
888 an error occurs in a deeply-nested routine, then every routine currently
889 called must unwind itself normally and return an error value back up to
890 the next routine. This means that every routine must explicitly check
891 for an error in all the routines it calls; if it does not do so,
892 unexpected and often random behavior results. This is an extremely
893 common source of bugs in C programs. An alternative would be to do a
894 non-local exit using @code{longjmp()}, but that is often very dangerous
895 because the routines that were exited past had no opportunity to clean
896 up after themselves and may leave things in an inconsistent state,
897 causing a crash shortly afterwards.
899 Lisp provides mechanisms to make such non-local exits safe. When an
900 error occurs, a routine simply signals that an error of a particular
901 class has occurred, and a non-local exit takes place. Any routine can
902 trap errors occurring in routines it calls by registering an error
903 handler for some or all classes of errors. (If no handler is registered,
904 a default handler, generally installed by the top-level event loop, is
905 executed; this prints out the error and continues.) Routines can also
906 specify cleanup code (called an @dfn{unwind-protect}) that will be
907 called when control exits from a block of code, no matter how that exit
908 occurs---i.e. even if a function deeply nested below it causes a
909 non-local exit back to the top level.
911 Note that this facility has appeared in some recent vintages of C, in
912 particular Visual C++ and other PC compilers written for the Microsoft
916 In Emacs Lisp, local variables are @dfn{dynamically scoped}. This means
917 that if you declare a local variable in a particular function, and then
918 call another function, that subfunction can ``see'' the local variable
919 you declared. This is actually considered a bug in Emacs Lisp and in
920 all other early dialects of Lisp, and was corrected in Common Lisp. (In
921 Common Lisp, you can still declare dynamically scoped variables if you
922 want to---they are sometimes useful---but variables by default are
923 @dfn{lexically scoped} as in C.)
926 For those familiar with Lisp, Emacs Lisp is modelled after MacLisp, an
927 early dialect of Lisp developed at MIT (no relation to the Macintosh
928 computer). There is a Common Lisp compatibility package available for
929 Emacs that provides many of the features of Common Lisp.
931 The Java language is derived in many ways from C, and shares a similar
932 syntax, but has the following features in common with Lisp (and different
937 Java is a safe language, like Lisp.
939 Java provides garbage collection, like Lisp.
941 Java has built-in facilities for handling errors and exceptions, like
944 Java has a type system that combines the best advantages of both static
945 and dynamic typing. Objects (except very simple types) are explicitly
946 marked with their type, as in dynamic typing; but there is a hierarchy
947 of types and functions are declared to accept only certain types, thus
948 providing the increased compile-time error-checking of static typing.
951 The Java language also has some negative attributes:
955 Java uses the edit/compile/run model of software development. This
956 makes it hard to use interactively. For example, to use Java like
957 @code{bc} it is necessary to write a special purpose, albeit tiny,
958 application. In Emacs Lisp, a calculator comes built-in without any
959 effort - one can always just type an expression in the @code{*scratch*}
962 Java tries too hard to enforce, not merely enable, portability, making
963 ordinary access to standard OS facilities painful. Java has an
964 @dfn{agenda}. I think this is why @code{chdir} is not part of standard
965 Java, which is inexcusable.
968 Unfortunately, there is no perfect language. Static typing allows a
969 compiler to catch programmer errors and produce more efficient code, but
970 makes programming more tedious and less fun. For the foreseeable future,
971 an Ideal Editing and Programming Environment (and that is what XEmacs
972 aspires to) will be programmable in multiple languages: high level ones
973 like Lisp for user customization and prototyping, and lower level ones
974 for infrastructure and industrial strength applications. If I had my
975 way, XEmacs would be friendly towards the Python, Scheme, C++, ML,
976 etc... communities. But there are serious technical difficulties to
979 The word @dfn{application} in the previous paragraph was used
980 intentionally. XEmacs implements an API for programs written in Lisp
981 that makes it a full-fledged application platform, very much like an OS
984 @node XEmacs From the Perspective of Building, XEmacs From the Inside, The Lisp Language, Top
985 @chapter XEmacs From the Perspective of Building
987 The heart of XEmacs is the Lisp environment, which is written in C.
988 This is contained in the @file{src/} subdirectory. Underneath
989 @file{src/} are two subdirectories of header files: @file{s/} (header
990 files for particular operating systems) and @file{m/} (header files for
991 particular machine types). In practice the distinction between the two
992 types of header files is blurred. These header files define or undefine
993 certain preprocessor constants and macros to indicate particular
994 characteristics of the associated machine or operating system. As part
995 of the configure process, one @file{s/} file and one @file{m/} file is
996 identified for the particular environment in which XEmacs is being
999 XEmacs also contains a great deal of Lisp code. This implements the
1000 operations that make XEmacs useful as an editor as well as just a Lisp
1001 environment, and also contains many add-on packages that allow XEmacs to
1002 browse directories, act as a mail and Usenet news reader, compile Lisp
1003 code, etc. There is actually more Lisp code than C code associated with
1004 XEmacs, but much of the Lisp code is peripheral to the actual operation
1005 of the editor. The Lisp code all lies in subdirectories underneath the
1006 @file{lisp/} directory.
1008 The @file{lwlib/} directory contains C code that implements a
1009 generalized interface onto different X widget toolkits and also
1010 implements some widgets of its own that behave like Motif widgets but
1011 are faster, free, and in some cases more powerful. The code in this
1012 directory compiles into a library and is mostly independent from XEmacs.
1014 The @file{etc/} directory contains various data files associated with
1015 XEmacs. Some of them are actually read by XEmacs at startup; others
1016 merely contain useful information of various sorts.
1018 The @file{lib-src/} directory contains C code for various auxiliary
1019 programs that are used in connection with XEmacs. Some of them are used
1020 during the build process; others are used to perform certain functions
1021 that cannot conveniently be placed in the XEmacs executable (e.g. the
1022 @file{movemail} program for fetching mail out of @file{/var/spool/mail},
1023 which must be setgid to @file{mail} on many systems; and the
1024 @file{gnuclient} program, which allows an external script to communicate
1025 with a running XEmacs process).
1027 The @file{man/} directory contains the sources for the XEmacs
1028 documentation. It is mostly in a form called Texinfo, which can be
1029 converted into either a printed document (by passing it through @TeX{})
1030 or into on-line documentation called @dfn{info files}.
1032 The @file{info/} directory contains the results of formatting the XEmacs
1033 documentation as @dfn{info files}, for on-line use. These files are
1034 used when you enter the Info system using @kbd{C-h i} or through the
1037 The @file{dynodump/} directory contains auxiliary code used to build
1038 XEmacs on Solaris platforms.
1040 The other directories contain various miscellaneous code and information
1041 that is not normally used or needed.
1043 The first step of building involves running the @file{configure} program
1044 and passing it various parameters to specify any optional features you
1045 want and compiler arguments and such, as described in the @file{INSTALL}
1046 file. This determines what the build environment is, chooses the
1047 appropriate @file{s/} and @file{m/} file, and runs a series of tests to
1048 determine many details about your environment, such as which library
1049 functions are available and exactly how they work. The reason for
1050 running these tests is that it allows XEmacs to be compiled on a much
1051 wider variety of platforms than those that the XEmacs developers happen
1052 to be familiar with, including various sorts of hybrid platforms. This
1053 is especially important now that many operating systems give you a great
1054 deal of control over exactly what features you want installed, and allow
1055 for easy upgrading of parts of a system without upgrading the rest. It
1056 would be impossible to pre-determine and pre-specify the information for
1057 all possible configurations.
1059 In fact, the @file{s/} and @file{m/} files are basically @emph{evil},
1060 since they contain unmaintainable platform-specific hard-coded
1061 information. XEmacs has been moving in the direction of having all
1062 system-specific information be determined dynamically by
1063 @file{configure}. Perhaps someday we can @code{rm -rf src/s src/m}.
1065 When configure is done running, it generates @file{Makefile}s and
1066 @file{GNUmakefile}s and the file @file{src/config.h} (which describes
1067 the features of your system) from template files. You then run
1068 @file{make}, which compiles the auxiliary code and programs in
1069 @file{lib-src/} and @file{lwlib/} and the main XEmacs executable in
1070 @file{src/}. The result of compiling and linking is an executable
1071 called @file{temacs}, which is @emph{not} the final XEmacs executable.
1072 @file{temacs} by itself is not intended to function as an editor or even
1073 display any windows on the screen, and if you simply run it, it will
1074 exit immediately. The @file{Makefile} runs @file{temacs} with certain
1075 options that cause it to initialize itself, read in a number of basic
1076 Lisp files, and then dump itself out into a new executable called
1077 @file{xemacs}. This new executable has been pre-initialized and
1078 contains pre-digested Lisp code that is necessary for the editor to
1079 function (this includes most basic editing functions,
1080 e.g. @code{kill-line}, that can be defined in terms of other Lisp
1081 primitives; some initialization code that is called when certain
1082 objects, such as frames, are created; and all of the standard
1083 keybindings and code for the actions they result in). This executable,
1084 @file{xemacs}, is the executable that you run to use the XEmacs editor.
1086 Although @file{temacs} is not intended to be run as an editor, it can,
1087 by using the incantation @code{temacs -batch -l loadup.el run-temacs}.
1088 This is useful when the dumping procedure described above is broken, or
1089 when using certain program debugging tools such as Purify. These tools
1090 get mighty confused by the tricks played by the XEmacs build process,
1091 such as allocation memory in one process, and freeing it in the next.
1093 @node XEmacs From the Inside, The XEmacs Object System (Abstractly Speaking), XEmacs From the Perspective of Building, Top
1094 @chapter XEmacs From the Inside
1096 Internally, XEmacs is quite complex, and can be very confusing. To
1097 simplify things, it can be useful to think of XEmacs as containing an
1098 event loop that ``drives'' everything, and a number of other subsystems,
1099 such as a Lisp engine and a redisplay mechanism. Each of these other
1100 subsystems exists simultaneously in XEmacs, and each has a certain
1101 state. The flow of control continually passes in and out of these
1102 different subsystems in the course of normal operation of the editor.
1104 It is important to keep in mind that, most of the time, the editor is
1105 ``driven'' by the event loop. Except during initialization and batch
1106 mode, all subsystems are entered directly or indirectly through the
1107 event loop, and ultimately, control exits out of all subsystems back up
1108 to the event loop. This cycle of entering a subsystem, exiting back out
1109 to the event loop, and starting another iteration of the event loop
1110 occurs once each keystroke, mouse motion, etc.
1112 If you're trying to understand a particular subsystem (other than the
1113 event loop), think of it as a ``daemon'' process or ``servant'' that is
1114 responsible for one particular aspect of a larger system, and
1115 periodically receives commands or environment changes that cause it to
1116 do something. Ultimately, these commands and environment changes are
1117 always triggered by the event loop. For example:
1121 The window and frame mechanism is responsible for keeping track of what
1122 windows and frames exist, what buffers are in them, etc. It is
1123 periodically given commands (usually from the user) to make a change to
1124 the current window/frame state: i.e. create a new frame, delete a
1128 The buffer mechanism is responsible for keeping track of what buffers
1129 exist and what text is in them. It is periodically given commands
1130 (usually from the user) to insert or delete text, create a buffer, etc.
1131 When it receives a text-change command, it notifies the redisplay
1135 The redisplay mechanism is responsible for making sure that windows and
1136 frames are displayed correctly. It is periodically told (by the event
1137 loop) to actually ``do its job'', i.e. snoop around and see what the
1138 current state of the environment (mostly of the currently-existing
1139 windows, frames, and buffers) is, and make sure that that state matches
1140 what's actually displayed. It keeps lots and lots of information around
1141 (such as what is actually being displayed currently, and what the
1142 environment was last time it checked) so that it can minimize the work
1143 it has to do. It is also helped along in that whenever a relevant
1144 change to the environment occurs, the redisplay mechanism is told about
1145 this, so it has a pretty good idea of where it has to look to find
1146 possible changes and doesn't have to look everywhere.
1149 The Lisp engine is responsible for executing the Lisp code in which most
1150 user commands are written. It is entered through a call to @code{eval}
1151 or @code{funcall}, which occurs as a result of dispatching an event from
1152 the event loop. The functions it calls issue commands to the buffer
1153 mechanism, the window/frame subsystem, etc.
1156 The Lisp allocation subsystem is responsible for keeping track of Lisp
1157 objects. It is given commands from the Lisp engine to allocate objects,
1158 garbage collect, etc.
1163 The important idea here is that there are a number of independent
1164 subsystems each with its own responsibility and persistent state, just
1165 like different employees in a company, and each subsystem is
1166 periodically given commands from other subsystems. Commands can flow
1167 from any one subsystem to any other, but there is usually some sort of
1168 hierarchy, with all commands originating from the event subsystem.
1170 XEmacs is entered in @code{main()}, which is in @file{emacs.c}. When
1171 this is called the first time (in a properly-invoked @file{temacs}), it
1176 It does some very basic environment initializations, such as determining
1177 where it and its directories (e.g. @file{lisp/} and @file{etc/}) reside
1178 and setting up signal handlers.
1180 It initializes the entire Lisp interpreter.
1182 It sets the initial values of many built-in variables (including many
1183 variables that are visible to Lisp programs), such as the global keymap
1184 object and the built-in faces (a face is an object that describes the
1185 display characteristics of text). This involves creating Lisp objects
1186 and thus is dependent on step (2).
1188 It performs various other initializations that are relevant to the
1189 particular environment it is running in, such as retrieving environment
1190 variables, determining the current date and the user who is running the
1191 program, examining its standard input, creating any necessary file
1194 At this point, the C initialization is complete. A Lisp program that
1195 was specified on the command line (usually @file{loadup.el}) is called
1196 (temacs is normally invoked as @code{temacs -batch -l loadup.el dump}).
1197 @file{loadup.el} loads all of the other Lisp files that are needed for
1198 the operation of the editor, calls the @code{dump-emacs} function to
1199 write out @file{xemacs}, and then kills the temacs process.
1202 When @file{xemacs} is then run, it only redoes steps (1) and (4)
1203 above; all variables already contain the values they were set to when
1204 the executable was dumped, and all memory that was allocated with
1205 @code{malloc()} is still around. (XEmacs knows whether it is being run
1206 as @file{xemacs} or @file{temacs} because it sets the global variable
1207 @code{initialized} to 1 after step (4) above.) At this point,
1208 @file{xemacs} calls a Lisp function to do any further initialization,
1209 which includes parsing the command-line (the C code can only do limited
1210 command-line parsing, which includes looking for the @samp{-batch} and
1211 @samp{-l} flags and a few other flags that it needs to know about before
1212 initialization is complete), creating the first frame (or @dfn{window}
1213 in standard window-system parlance), running the user's init file
1214 (usually the file @file{.emacs} in the user's home directory), etc. The
1215 function to do this is usually called @code{normal-top-level};
1216 @file{loadup.el} tells the C code about this function by setting its
1217 name as the value of the Lisp variable @code{top-level}.
1219 When the Lisp initialization code is done, the C code enters the event
1220 loop, and stays there for the duration of the XEmacs process. The code
1221 for the event loop is contained in @file{keyboard.c}, and is called
1222 @code{Fcommand_loop_1()}. Note that this event loop could very well be
1223 written in Lisp, and in fact a Lisp version exists; but apparently,
1224 doing this makes XEmacs run noticeably slower.
1226 Notice how much of the initialization is done in Lisp, not in C.
1227 In general, XEmacs tries to move as much code as is possible
1228 into Lisp. Code that remains in C is code that implements the
1229 Lisp interpreter itself, or code that needs to be very fast, or
1230 code that needs to do system calls or other such stuff that
1231 needs to be done in C, or code that needs to have access to
1232 ``forbidden'' structures. (One conscious aspect of the design of
1233 Lisp under XEmacs is a clean separation between the external
1234 interface to a Lisp object's functionality and its internal
1235 implementation. Part of this design is that Lisp programs
1236 are forbidden from accessing the contents of the object other
1237 than through using a standard API. In this respect, XEmacs Lisp
1238 is similar to modern Lisp dialects but differs from GNU Emacs,
1239 which tends to expose the implementation and allow Lisp
1240 programs to look at it directly. The major advantage of
1241 hiding the implementation is that it allows the implementation
1242 to be redesigned without affecting any Lisp programs, including
1243 those that might want to be ``clever'' by looking directly at
1244 the object's contents and possibly manipulating them.)
1246 Moving code into Lisp makes the code easier to debug and maintain and
1247 makes it much easier for people who are not XEmacs developers to
1248 customize XEmacs, because they can make a change with much less chance
1249 of obscure and unwanted interactions occurring than if they were to
1252 @node The XEmacs Object System (Abstractly Speaking), How Lisp Objects Are Represented in C, XEmacs From the Inside, Top
1253 @chapter The XEmacs Object System (Abstractly Speaking)
1255 At the heart of the Lisp interpreter is its management of objects.
1256 XEmacs Lisp contains many built-in objects, some of which are
1257 simple and others of which can be very complex; and some of which
1258 are very common, and others of which are rarely used or are only
1259 used internally. (Since the Lisp allocation system, with its
1260 automatic reclamation of unused storage, is so much more convenient
1261 than @code{malloc()} and @code{free()}, the C code makes extensive use of it
1262 in its internal operations.)
1264 The basic Lisp objects are
1268 28 or 31 bits of precision, or 60 or 63 bits on 64-bit machines; the
1269 reason for this is described below when the internal Lisp object
1270 representation is described.
1272 Same precision as a double in C.
1274 A simple container for two Lisp objects, used to implement lists and
1275 most other data structures in Lisp.
1277 An object representing a single character of text; chars behave like
1278 integers in many ways but are logically considered text rather than
1279 numbers and have a different read syntax. (the read syntax for a char
1280 contains the char itself or some textual encoding of it---for example,
1281 a Japanese Kanji character might be encoded as @samp{^[$(B#&^[(B} using the
1282 ISO-2022 encoding standard---rather than the numerical representation
1283 of the char; this way, if the mapping between chars and integers
1284 changes, which is quite possible for Kanji characters and other extended
1285 characters, the same character will still be created. Note that some
1286 primitives confuse chars and integers. The worst culprit is @code{eq},
1287 which makes a special exception and considers a char to be @code{eq} to
1288 its integer equivalent, even though in no other case are objects of two
1289 different types @code{eq}. The reason for this monstrosity is
1290 compatibility with existing code; the separation of char from integer
1291 came fairly recently.)
1293 An object that contains Lisp objects and is referred to by name;
1294 symbols are used to implement variables and named functions
1295 and to provide the equivalent of preprocessor constants in C.
1297 A one-dimensional array of Lisp objects providing constant-time access
1298 to any of the objects; access to an arbitrary object in a vector is
1299 faster than for lists, but the operations that can be done on a vector
1302 Self-explanatory; behaves much like a vector of chars
1303 but has a different read syntax and is stored and manipulated
1306 A vector of bits; similar to a string in spirit.
1307 @item compiled-function
1308 An object containing compiled Lisp code, known as @dfn{byte code}.
1310 A Lisp primitive, i.e. a Lisp-callable function implemented in C.
1314 Note that there is no basic ``function'' type, as in more powerful
1315 versions of Lisp (where it's called a @dfn{closure}). XEmacs Lisp does
1316 not provide the closure semantics implemented by Common Lisp and Scheme.
1317 The guts of a function in XEmacs Lisp are represented in one of four
1318 ways: a symbol specifying another function (when one function is an
1319 alias for another), a list (whose first element must be the symbol
1320 @code{lambda}) containing the function's source code, a
1321 compiled-function object, or a subr object. (In other words, given a
1322 symbol specifying the name of a function, calling @code{symbol-function}
1323 to retrieve the contents of the symbol's function cell will return one
1324 of these types of objects.)
1326 XEmacs Lisp also contains numerous specialized objects used to implement
1331 Stores text like a string, but is optimized for insertion and deletion
1332 and has certain other properties that can be set.
1334 An object with various properties whose displayable representation is a
1335 @dfn{window} in window-system parlance.
1337 A section of a frame that displays the contents of a buffer;
1338 often called a @dfn{pane} in window-system parlance.
1339 @item window-configuration
1340 An object that represents a saved configuration of windows in a frame.
1342 An object representing a screen on which frames can be displayed;
1343 equivalent to a @dfn{display} in the X Window System and a @dfn{TTY} in
1346 An object specifying the appearance of text or graphics; it has
1347 properties such as font, foreground color, and background color.
1349 An object that refers to a particular position in a buffer and moves
1350 around as text is inserted and deleted to stay in the same relative
1351 position to the text around it.
1353 Similar to a marker but covers a range of text in a buffer; can also
1354 specify properties of the text, such as a face in which the text is to
1355 be displayed, whether the text is invisible or unmodifiable, etc.
1357 Generated by calling @code{next-event} and contains information
1358 describing a particular event happening in the system, such as the user
1359 pressing a key or a process terminating.
1361 An object that maps from events (described using lists, vectors, and
1362 symbols rather than with an event object because the mapping is for
1363 classes of events, rather than individual events) to functions to
1364 execute or other events to recursively look up; the functions are
1365 described by name, using a symbol, or using lists to specify the
1368 An object that describes the appearance of an image (e.g. pixmap) on
1369 the screen; glyphs can be attached to the beginning or end of extents
1370 and in some future version of XEmacs will be able to be inserted
1371 directly into a buffer.
1373 An object that describes a connection to an externally-running process.
1376 There are some other, less-commonly-encountered general objects:
1380 An object that maps from an arbitrary Lisp object to another arbitrary
1381 Lisp object, using hashing for fast lookup.
1383 A limited form of hash-table that maps from strings to symbols; obarrays
1384 are used to look up a symbol given its name and are not actually their
1385 own object type but are kludgily represented using vectors with hidden
1386 fields (this representation derives from GNU Emacs).
1388 A complex object used to specify the value of a display property; a
1389 default value is given and different values can be specified for
1390 particular frames, buffers, windows, devices, or classes of device.
1392 An object that maps from chars or classes of chars to arbitrary Lisp
1393 objects; internally char tables use a complex nested-vector
1394 representation that is optimized to the way characters are represented
1397 An object that maps from ranges of integers to arbitrary Lisp objects.
1400 And some strange special-purpose objects:
1404 @itemx coding-system
1405 Objects used when MULE, or multi-lingual/Asian-language, support is
1407 @item color-instance
1408 @itemx font-instance
1409 @itemx image-instance
1410 An object that encapsulates a window-system resource; instances are
1411 mostly used internally but are exposed on the Lisp level for cleanness
1412 of the specifier model and because it's occasionally useful for Lisp
1413 program to create or query the properties of instances.
1415 An object that encapsulate a @dfn{subwindow} resource, i.e. a
1416 window-system child window that is drawn into by an external process;
1417 this object should be integrated into the glyph system but isn't yet,
1418 and may change form when this is done.
1419 @item tooltalk-message
1420 @itemx tooltalk-pattern
1421 Objects that represent resources used in the ToolTalk interprocess
1422 communication protocol.
1423 @item toolbar-button
1424 An object used in conjunction with the toolbar.
1427 And objects that are only used internally:
1431 A generic object for encapsulating arbitrary memory; this allows you the
1432 generality of @code{malloc()} and the convenience of the Lisp object
1435 A buffering I/O stream, used to provide a unified interface to anything
1436 that can accept output or provide input, such as a file descriptor, a
1437 stdio stream, a chunk of memory, a Lisp buffer, a Lisp string, etc.;
1438 it's a Lisp object to make its memory management more convenient.
1439 @item char-table-entry
1440 Subsidiary objects in the internal char-table representation.
1441 @item extent-auxiliary
1444 Various special-purpose objects that are basically just used to
1445 encapsulate memory for particular subsystems, similar to the more
1446 general ``opaque'' object.
1447 @item symbol-value-forward
1448 @itemx symbol-value-buffer-local
1449 @itemx symbol-value-varalias
1450 @itemx symbol-value-lisp-magic
1451 Special internal-only objects that are placed in the value cell of a
1452 symbol to indicate that there is something special with this variable --
1453 e.g. it has no value, it mirrors another variable, or it mirrors some C
1454 variable; there is really only one kind of object, called a
1455 @dfn{symbol-value-magic}, but it is sort-of halfway kludged into
1456 semi-different object types.
1459 @cindex permanent objects
1460 @cindex temporary objects
1461 Some types of objects are @dfn{permanent}, meaning that once created,
1462 they do not disappear until explicitly destroyed, using a function such
1463 as @code{delete-buffer}, @code{delete-window}, @code{delete-frame}, etc.
1464 Others will disappear once they are not longer used, through the garbage
1465 collection mechanism. Buffers, frames, windows, devices, and processes
1466 are among the objects that are permanent. Note that some objects can go
1467 both ways: Faces can be created either way; extents are normally
1468 permanent, but detached extents (extents not referring to any text, as
1469 happens to some extents when the text they are referring to is deleted)
1470 are temporary. Note that some permanent objects, such as faces and
1471 coding systems, cannot be deleted. Note also that windows are unique in
1472 that they can be @emph{undeleted} after having previously been
1473 deleted. (This happens as a result of restoring a window configuration.)
1476 Note that many types of objects have a @dfn{read syntax}, i.e. a way of
1477 specifying an object of that type in Lisp code. When you load a Lisp
1478 file, or type in code to be evaluated, what really happens is that the
1479 function @code{read} is called, which reads some text and creates an object
1480 based on the syntax of that text; then @code{eval} is called, which
1481 possibly does something special; then this loop repeats until there's
1482 no more text to read. (@code{eval} only actually does something special
1483 with symbols, which causes the symbol's value to be returned,
1484 similar to referencing a variable; and with conses [i.e. lists],
1485 which cause a function invocation. All other values are returned
1494 converts to an integer whose value is 17297.
1500 converts to a float whose value is 1.983e-4, or .0001983.
1506 converts to a char that represents the lowercase letter b.
1512 (where @samp{^[} actually is an @samp{ESC} character) converts to a
1513 particular Kanji character when using an ISO2022-based coding system for
1514 input. (To decode this goo: @samp{ESC} begins an escape sequence;
1515 @samp{ESC $ (} is a class of escape sequences meaning ``switch to a
1516 94x94 character set''; @samp{ESC $ ( B} means ``switch to Japanese
1517 Kanji''; @samp{#} and @samp{&} collectively index into a 94-by-94 array
1518 of characters [subtract 33 from the ASCII value of each character to get
1519 the corresponding index]; @samp{ESC (} is a class of escape sequences
1520 meaning ``switch to a 94 character set''; @samp{ESC (B} means ``switch
1521 to US ASCII''. It is a coincidence that the letter @samp{B} is used to
1522 denote both Japanese Kanji and US ASCII. If the first @samp{B} were
1523 replaced with an @samp{A}, you'd be requesting a Chinese Hanzi character
1524 from the GB2312 character set.)
1530 converts to a string.
1536 converts to a symbol whose name is @code{"foobar"}. This is done by
1537 looking up the string equivalent in the global variable
1538 @code{obarray}, whose contents should be an obarray. If no symbol
1539 is found, a new symbol with the name @code{"foobar"} is automatically
1540 created and added to @code{obarray}; this process is called
1541 @dfn{interning} the symbol.
1548 converts to a cons cell containing the symbols @code{foo} and @code{bar}.
1554 converts to a three-element list containing the specified objects
1555 (note that a list is actually a set of nested conses; see the
1556 XEmacs Lisp Reference).
1562 converts to a three-element vector containing the specified objects.
1568 converts to a compiled-function object (the actual contents are not
1569 shown since they are not relevant here; look at a file that ends with
1570 @file{.elc} for examples).
1576 converts to a bit-vector.
1579 #s(hash-table ... ...)
1582 converts to a hash table (the actual contents are not shown).
1585 #s(range-table ... ...)
1588 converts to a range table (the actual contents are not shown).
1591 #s(char-table ... ...)
1594 converts to a char table (the actual contents are not shown).
1596 Note that the @code{#s()} syntax is the general syntax for structures,
1597 which are not really implemented in XEmacs Lisp but should be.
1599 When an object is printed out (using @code{print} or a related
1600 function), the read syntax is used, so that the same object can be read
1603 The other objects do not have read syntaxes, usually because it does not
1604 really make sense to create them in this fashion (i.e. processes, where
1605 it doesn't make sense to have a subprocess created as a side effect of
1606 reading some Lisp code), or because they can't be created at all
1607 (e.g. subrs). Permanent objects, as a rule, do not have a read syntax;
1608 nor do most complex objects, which contain too much state to be easily
1609 initialized through a read syntax.
1611 @node How Lisp Objects Are Represented in C, Rules When Writing New C Code, The XEmacs Object System (Abstractly Speaking), Top
1612 @chapter How Lisp Objects Are Represented in C
1614 Lisp objects are represented in C using a 32-bit or 64-bit machine word
1615 (depending on the processor; i.e. DEC Alphas use 64-bit Lisp objects and
1616 most other processors use 32-bit Lisp objects). The representation
1617 stuffs a pointer together with a tag, as follows:
1620 [ 3 3 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 ]
1621 [ 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 ]
1623 <---------------------------------------------------------> <->
1624 a pointer to a structure, or an integer tag
1627 A tag of 00 is used for all pointer object types, a tag of 10 is used
1628 for characters, and the other two tags 01 and 11 are joined together to
1629 form the integer object type. This representation gives us 31 bit
1630 integers and 30 bit characters, while pointers are represented directly
1631 without any bit masking or shifting. This representation, though,
1632 assumes that pointers to structs are always aligned to multiples of 4,
1633 so the lower 2 bits are always zero.
1635 Lisp objects use the typedef @code{Lisp_Object}, but the actual C type
1636 used for the Lisp object can vary. It can be either a simple type
1637 (@code{long} on the DEC Alpha, @code{int} on other machines) or a
1638 structure whose fields are bit fields that line up properly (actually, a
1639 union of structures is used). Generally the simple integral type is
1640 preferable because it ensures that the compiler will actually use a
1641 machine word to represent the object (some compilers will use more
1642 general and less efficient code for unions and structs even if they can
1643 fit in a machine word). The union type, however, has the advantage of
1644 stricter type checking. If you accidentally pass an integer where a Lisp
1645 object is desired, you get a compile error. The choice of which type
1646 to use is determined by the preprocessor constant @code{USE_UNION_TYPE}
1647 which is defined via the @code{--use-union-type} option to
1650 Various macros are used to convert between Lisp_Objects and the
1651 corresponding C type. Macros of the form @code{XINT()}, @code{XCHAR()},
1652 @code{XSTRING()}, @code{XSYMBOL()}, do any required bit shifting and/or
1653 masking and cast it to the appropriate type. @code{XINT()} needs to be
1654 a bit tricky so that negative numbers are properly sign-extended. Since
1655 integers are stored left-shifted, if the right-shift operator does an
1656 arithmetic shift (i.e. it leaves the most-significant bit as-is rather
1657 than shifting in a zero, so that it mimics a divide-by-two even for
1658 negative numbers) the shift to remove the tag bit is enough. This is
1659 the case on all the systems we support.
1661 Note that when @code{ERROR_CHECK_TYPECHECK} is defined, the converter
1662 macros become more complicated---they check the tag bits and/or the
1663 type field in the first four bytes of a record type to ensure that the
1664 object is really of the correct type. This is great for catching places
1665 where an incorrect type is being dereferenced---this typically results
1666 in a pointer being dereferenced as the wrong type of structure, with
1667 unpredictable (and sometimes not easily traceable) results.
1669 There are similar @code{XSET@var{TYPE}()} macros that construct a Lisp
1670 object. These macros are of the form @code{XSET@var{TYPE}
1671 (@var{lvalue}, @var{result})}, i.e. they have to be a statement rather
1672 than just used in an expression. The reason for this is that standard C
1673 doesn't let you ``construct'' a structure (but GCC does). Granted, this
1674 sometimes isn't too convenient; for the case of integers, at least, you
1675 can use the function @code{make_int()}, which constructs and
1676 @emph{returns} an integer Lisp object. Note that the
1677 @code{XSET@var{TYPE}()} macros are also affected by
1678 @code{ERROR_CHECK_TYPECHECK} and make sure that the structure is of the
1679 right type in the case of record types, where the type is contained in
1682 The C programmer is responsible for @strong{guaranteeing} that a
1683 Lisp_Object is the correct type before using the @code{X@var{TYPE}}
1684 macros. This is especially important in the case of lists. Use
1685 @code{XCAR} and @code{XCDR} if a Lisp_Object is certainly a cons cell,
1686 else use @code{Fcar()} and @code{Fcdr()}. Trust other C code, but not
1687 Lisp code. On the other hand, if XEmacs has an internal logic error,
1688 it's better to crash immediately, so sprinkle @code{assert()}s and
1689 ``unreachable'' @code{abort()}s liberally about the source code. Where
1690 performance is an issue, use @code{type_checking_assert},
1691 @code{bufpos_checking_assert}, and @code{gc_checking_assert}, which do
1692 nothing unless the corresponding configure error checking flag was
1695 @node Rules When Writing New C Code, A Summary of the Various XEmacs Modules, How Lisp Objects Are Represented in C, Top
1696 @chapter Rules When Writing New C Code
1698 The XEmacs C Code is extremely complex and intricate, and there are many
1699 rules that are more or less consistently followed throughout the code.
1700 Many of these rules are not obvious, so they are explained here. It is
1701 of the utmost importance that you follow them. If you don't, you may
1702 get something that appears to work, but which will crash in odd
1703 situations, often in code far away from where the actual breakage is.
1706 * General Coding Rules::
1707 * Writing Lisp Primitives::
1708 * Adding Global Lisp Variables::
1710 * Techniques for XEmacs Developers::
1713 @node General Coding Rules, Writing Lisp Primitives, Rules When Writing New C Code, Rules When Writing New C Code
1714 @section General Coding Rules
1716 The C code is actually written in a dialect of C called @dfn{Clean C},
1717 meaning that it can be compiled, mostly warning-free, with either a C or
1718 C++ compiler. Coding in Clean C has several advantages over plain C.
1719 C++ compilers are more nit-picking, and a number of coding errors have
1720 been found by compiling with C++. The ability to use both C and C++
1721 tools means that a greater variety of development tools are available to
1724 Almost every module contains a @code{syms_of_*()} function and a
1725 @code{vars_of_*()} function. The former declares any Lisp primitives
1726 you have defined and defines any symbols you will be using. The latter
1727 declares any global Lisp variables you have added and initializes global
1728 C variables in the module. For each such function, declare it in
1729 @file{symsinit.h} and make sure it's called in the appropriate place in
1730 @file{emacs.c}. @strong{Important}: There are stringent requirements on
1731 exactly what can go into these functions. See the comment in
1732 @file{emacs.c}. The reason for this is to avoid obscure unwanted
1733 interactions during initialization. If you don't follow these rules,
1734 you'll be sorry! If you want to do anything that isn't allowed, create
1735 a @code{complex_vars_of_*()} function for it. Doing this is tricky,
1736 though: You have to make sure your function is called at the right time
1737 so that all the initialization dependencies work out.
1739 Every module includes @file{<config.h>} (angle brackets so that
1740 @samp{--srcdir} works correctly; @file{config.h} may or may not be in
1741 the same directory as the C sources) and @file{lisp.h}. @file{config.h}
1742 must always be included before any other header files (including
1743 system header files) to ensure that certain tricks played by various
1744 @file{s/} and @file{m/} files work out correctly.
1746 When including header files, always use angle brackets, not double
1747 quotes, except when the file to be included is always in the same
1748 directory as the including file. If either file is a generated file,
1749 then that is not likely to be the case. In order to understand why we
1750 have this rule, imagine what happens when you do a build in the source
1751 directory using @samp{./configure} and another build in another
1752 directory using @samp{../work/configure}. There will be two different
1753 @file{config.h} files. Which one will be used if you @samp{#include
1756 @strong{All global and static variables that are to be modifiable must
1757 be declared uninitialized.} This means that you may not use the
1758 ``declare with initializer'' form for these variables, such as @code{int
1759 some_variable = 0;}. The reason for this has to do with some kludges
1760 done during the dumping process: If possible, the initialized data
1761 segment is re-mapped so that it becomes part of the (unmodifiable) code
1762 segment in the dumped executable. This allows this memory to be shared
1763 among multiple running XEmacs processes. XEmacs is careful to place as
1764 much constant data as possible into initialized variables during the
1765 @file{temacs} phase.
1767 @cindex copy-on-write
1768 @strong{Please note:} This kludge only works on a few systems nowadays,
1769 and is rapidly becoming irrelevant because most modern operating systems
1770 provide @dfn{copy-on-write} semantics. All data is initially shared
1771 between processes, and a private copy is automatically made (on a
1772 page-by-page basis) when a process first attempts to write to a page of
1775 Formerly, there was a requirement that static variables not be declared
1776 inside of functions. This had to do with another hack along the same
1777 vein as what was just described: old USG systems put statically-declared
1778 variables in the initialized data space, so those header files had a
1779 @code{#define static} declaration. (That way, the data-segment remapping
1780 described above could still work.) This fails badly on static variables
1781 inside of functions, which suddenly become automatic variables;
1782 therefore, you weren't supposed to have any of them. This awful kludge
1783 has been removed in XEmacs because
1787 almost all of the systems that used this kludge ended up having
1788 to disable the data-segment remapping anyway;
1790 the only systems that didn't were extremely outdated ones;
1792 this hack completely messed up inline functions.
1795 The C source code makes heavy use of C preprocessor macros. One popular
1799 #define FOO(var, value) do @{ \
1800 Lisp_Object FOO_value = (value); \
1801 ... /* compute using FOO_value */ \
1806 The @code{do @{...@} while (0)} is a standard trick to allow FOO to have
1807 statement semantics, so that it can safely be used within an @code{if}
1808 statement in C, for example. Multiple evaluation is prevented by
1809 copying a supplied argument into a local variable, so that
1810 @code{FOO(var,fun(1))} only calls @code{fun} once.
1812 Lisp lists are popular data structures in the C code as well as in
1813 Elisp. There are two sets of macros that iterate over lists.
1814 @code{EXTERNAL_LIST_LOOP_@var{n}} should be used when the list has been
1815 supplied by the user, and cannot be trusted to be acyclic and
1816 nil-terminated. A @code{malformed-list} or @code{circular-list} error
1817 will be generated if the list being iterated over is not entirely
1818 kosher. @code{LIST_LOOP_@var{n}}, on the other hand, is faster and less
1819 safe, and can be used only on trusted lists.
1821 Related macros are @code{GET_EXTERNAL_LIST_LENGTH} and
1822 @code{GET_LIST_LENGTH}, which calculate the length of a list, and in the
1823 case of @code{GET_EXTERNAL_LIST_LENGTH}, validating the properness of
1824 the list. The macros @code{EXTERNAL_LIST_LOOP_DELETE_IF} and
1825 @code{LIST_LOOP_DELETE_IF} delete elements from a lisp list satisfying some
1828 @node Writing Lisp Primitives, Adding Global Lisp Variables, General Coding Rules, Rules When Writing New C Code
1829 @section Writing Lisp Primitives
1831 Lisp primitives are Lisp functions implemented in C. The details of
1832 interfacing the C function so that Lisp can call it are handled by a few
1833 C macros. The only way to really understand how to write new C code is
1834 to read the source, but we can explain some things here.
1836 An example of a special form is the definition of @code{prog1}, from
1837 @file{eval.c}. (An ordinary function would have the same general
1840 @cindex garbage collection protection
1843 DEFUN ("prog1", Fprog1, 1, UNEVALLED, 0, /*
1844 Similar to `progn', but the value of the first form is returned.
1845 \(prog1 FIRST BODY...): All the arguments are evaluated sequentially.
1846 The value of FIRST is saved during evaluation of the remaining args,
1847 whose values are discarded.
1851 /* This function can GC */
1852 REGISTER Lisp_Object val, form, tail;
1853 struct gcpro gcpro1;
1855 val = Feval (XCAR (args));
1859 LIST_LOOP_3 (form, XCDR (args), tail)
1868 Let's start with a precise explanation of the arguments to the
1869 @code{DEFUN} macro. Here is a template for them:
1873 DEFUN (@var{lname}, @var{fname}, @var{min_args}, @var{max_args}, @var{interactive}, /*
1882 This string is the name of the Lisp symbol to define as the function
1883 name; in the example above, it is @code{"prog1"}.
1886 This is the C function name for this function. This is the name that is
1887 used in C code for calling the function. The name is, by convention,
1888 @samp{F} prepended to the Lisp name, with all dashes (@samp{-}) in the
1889 Lisp name changed to underscores. Thus, to call this function from C
1890 code, call @code{Fprog1}. Remember that the arguments are of type
1891 @code{Lisp_Object}; various macros and functions for creating values of
1892 type @code{Lisp_Object} are declared in the file @file{lisp.h}.
1894 Primitives whose names are special characters (e.g. @code{+} or
1895 @code{<}) are named by spelling out, in some fashion, the special
1896 character: e.g. @code{Fplus()} or @code{Flss()}. Primitives whose names
1897 begin with normal alphanumeric characters but also contain special
1898 characters are spelled out in some creative way, e.g. @code{let*}
1899 becomes @code{FletX()}.
1901 Each function also has an associated structure that holds the data for
1902 the subr object that represents the function in Lisp. This structure
1903 conveys the Lisp symbol name to the initialization routine that will
1904 create the symbol and store the subr object as its definition. The C
1905 variable name of this structure is always @samp{S} prepended to the
1906 @var{fname}. You hardly ever need to be aware of the existence of this
1907 structure, since @code{DEFUN} plus @code{DEFSUBR} takes care of all the
1911 This is the minimum number of arguments that the function requires. The
1912 function @code{prog1} allows a minimum of one argument.
1915 This is the maximum number of arguments that the function accepts, if
1916 there is a fixed maximum. Alternatively, it can be @code{UNEVALLED},
1917 indicating a special form that receives unevaluated arguments, or
1918 @code{MANY}, indicating an unlimited number of evaluated arguments (the
1919 C equivalent of @code{&rest}). Both @code{UNEVALLED} and @code{MANY}
1920 are macros. If @var{max_args} is a number, it may not be less than
1921 @var{min_args} and it may not be greater than 8. (If you need to add a
1922 function with more than 8 arguments, use the @code{MANY} form. Resist
1923 the urge to edit the definition of @code{DEFUN} in @file{lisp.h}. If
1924 you do it anyways, make sure to also add another clause to the switch
1925 statement in @code{primitive_funcall().})
1928 This is an interactive specification, a string such as might be used as
1929 the argument of @code{interactive} in a Lisp function. In the case of
1930 @code{prog1}, it is 0 (a null pointer), indicating that @code{prog1}
1931 cannot be called interactively. A value of @code{""} indicates a
1932 function that should receive no arguments when called interactively.
1935 This is the documentation string. It is written just like a
1936 documentation string for a function defined in Lisp; in particular, the
1937 first line should be a single sentence. Note how the documentation
1938 string is enclosed in a comment, none of the documentation is placed on
1939 the same lines as the comment-start and comment-end characters, and the
1940 comment-start characters are on the same line as the interactive
1941 specification. @file{make-docfile}, which scans the C files for
1942 documentation strings, is very particular about what it looks for, and
1943 will not properly extract the doc string if it's not in this exact format.
1945 In order to make both @file{etags} and @file{make-docfile} happy, make
1946 sure that the @code{DEFUN} line contains the @var{lname} and
1947 @var{fname}, and that the comment-start characters for the doc string
1948 are on the same line as the interactive specification, and put a newline
1949 directly after them (and before the comment-end characters).
1952 This is the comma-separated list of arguments to the C function. For a
1953 function with a fixed maximum number of arguments, provide a C argument
1954 for each Lisp argument. In this case, unlike regular C functions, the
1955 types of the arguments are not declared; they are simply always of type
1958 The names of the C arguments will be used as the names of the arguments
1959 to the Lisp primitive as displayed in its documentation, modulo the same
1960 concerns described above for @code{F...} names (in particular,
1961 underscores in the C arguments become dashes in the Lisp arguments).
1963 There is one additional kludge: A trailing `_' on the C argument is
1964 discarded when forming the Lisp argument. This allows C language
1965 reserved words (like @code{default}) or global symbols (like
1966 @code{dirname}) to be used as argument names without compiler warnings
1969 A Lisp function with @w{@var{max_args} = @code{UNEVALLED}} is a
1970 @w{@dfn{special form}}; its arguments are not evaluated. Instead it
1971 receives one argument of type @code{Lisp_Object}, a (Lisp) list of the
1972 unevaluated arguments, conventionally named @code{(args)}.
1974 When a Lisp function has no upper limit on the number of arguments,
1975 specify @w{@var{max_args} = @code{MANY}}. In this case its implementation in
1976 C actually receives exactly two arguments: the number of Lisp arguments
1977 (an @code{int}) and the address of a block containing their values (a
1978 @w{@code{Lisp_Object *}}). In this case only are the C types specified
1979 in the @var{arglist}: @w{@code{(int nargs, Lisp_Object *args)}}.
1983 Within the function @code{Fprog1} itself, note the use of the macros
1984 @code{GCPRO1} and @code{UNGCPRO}. @code{GCPRO1} is used to ``protect''
1985 a variable from garbage collection---to inform the garbage collector
1986 that it must look in that variable and regard the object pointed at by
1987 its contents as an accessible object. This is necessary whenever you
1988 call @code{Feval} or anything that can directly or indirectly call
1989 @code{Feval} (this includes the @code{QUIT} macro!). At such a time,
1990 any Lisp object that you intend to refer to again must be protected
1991 somehow. @code{UNGCPRO} cancels the protection of the variables that
1992 are protected in the current function. It is necessary to do this
1995 The macro @code{GCPRO1} protects just one local variable. If you want
1996 to protect two, use @code{GCPRO2} instead; repeating @code{GCPRO1} will
1997 not work. Macros @code{GCPRO3} and @code{GCPRO4} also exist.
1999 These macros implicitly use local variables such as @code{gcpro1}; you
2000 must declare these explicitly, with type @code{struct gcpro}. Thus, if
2001 you use @code{GCPRO2}, you must declare @code{gcpro1} and @code{gcpro2}.
2003 @cindex caller-protects (@code{GCPRO} rule)
2004 Note also that the general rule is @dfn{caller-protects}; i.e. you are
2005 only responsible for protecting those Lisp objects that you create. Any
2006 objects passed to you as arguments should have been protected by whoever
2007 created them, so you don't in general have to protect them.
2009 In particular, the arguments to any Lisp primitive are always
2010 automatically @code{GCPRO}ed, when called ``normally'' from Lisp code or
2011 bytecode. So only a few Lisp primitives that are called frequently from
2012 C code, such as @code{Fprogn} protect their arguments as a service to
2013 their caller. You don't need to protect your arguments when writing a
2016 @code{GCPRO}ing is perhaps the trickiest and most error-prone part of
2017 XEmacs coding. It is @strong{extremely} important that you get this
2018 right and use a great deal of discipline when writing this code.
2019 @xref{GCPROing, ,@code{GCPRO}ing}, for full details on how to do this.
2021 What @code{DEFUN} actually does is declare a global structure of type
2022 @code{Lisp_Subr} whose name begins with capital @samp{SF} and which
2023 contains information about the primitive (e.g. a pointer to the
2024 function, its minimum and maximum allowed arguments, a string describing
2025 its Lisp name); @code{DEFUN} then begins a normal C function declaration
2026 using the @code{F...} name. The Lisp subr object that is the function
2027 definition of a primitive (i.e. the object in the function slot of the
2028 symbol that names the primitive) actually points to this @samp{SF}
2029 structure; when @code{Feval} encounters a subr, it looks in the
2030 structure to find out how to call the C function.
2032 Defining the C function is not enough to make a Lisp primitive
2033 available; you must also create the Lisp symbol for the primitive (the
2034 symbol is @dfn{interned}; @pxref{Obarrays}) and store a suitable subr
2035 object in its function cell. (If you don't do this, the primitive won't
2036 be seen by Lisp code.) The code looks like this:
2039 DEFSUBR (@var{fname});
2043 Here @var{fname} is the same name you used as the second argument to
2046 This call to @code{DEFSUBR} should go in the @code{syms_of_*()} function
2047 at the end of the module. If no such function exists, create it and
2048 make sure to also declare it in @file{symsinit.h} and call it from the
2049 appropriate spot in @code{main()}. @xref{General Coding Rules}.
2051 Note that C code cannot call functions by name unless they are defined
2052 in C. The way to call a function written in Lisp from C is to use
2053 @code{Ffuncall}, which embodies the Lisp function @code{funcall}. Since
2054 the Lisp function @code{funcall} accepts an unlimited number of
2055 arguments, in C it takes two: the number of Lisp-level arguments, and a
2056 one-dimensional array containing their values. The first Lisp-level
2057 argument is the Lisp function to call, and the rest are the arguments to
2058 pass to it. Since @code{Ffuncall} can call the evaluator, you must
2059 protect pointers from garbage collection around the call to
2060 @code{Ffuncall}. (However, @code{Ffuncall} explicitly protects all of
2061 its parameters, so you don't have to protect any pointers passed as
2064 The C functions @code{call0}, @code{call1}, @code{call2}, and so on,
2065 provide handy ways to call a Lisp function conveniently with a fixed
2066 number of arguments. They work by calling @code{Ffuncall}.
2068 @file{eval.c} is a very good file to look through for examples;
2069 @file{lisp.h} contains the definitions for important macros and
2072 @node Adding Global Lisp Variables, Coding for Mule, Writing Lisp Primitives, Rules When Writing New C Code
2073 @section Adding Global Lisp Variables
2075 Global variables whose names begin with @samp{Q} are constants whose
2076 value is a symbol of a particular name. The name of the variable should
2077 be derived from the name of the symbol using the same rules as for Lisp
2078 primitives. These variables are initialized using a call to
2079 @code{defsymbol()} in the @code{syms_of_*()} function. (This call
2080 interns a symbol, sets the C variable to the resulting Lisp object, and
2081 calls @code{staticpro()} on the C variable to tell the
2082 garbage-collection mechanism about this variable. What
2083 @code{staticpro()} does is add a pointer to the variable to a large
2084 global array; when garbage-collection happens, all pointers listed in
2085 the array are used as starting points for marking Lisp objects. This is
2086 important because it's quite possible that the only current reference to
2087 the object is the C variable. In the case of symbols, the
2088 @code{staticpro()} doesn't matter all that much because the symbol is
2089 contained in @code{obarray}, which is itself @code{staticpro()}ed.
2090 However, it's possible that a naughty user could do something like
2091 uninterning the symbol out of @code{obarray} or even setting
2092 @code{obarray} to a different value [although this is likely to make
2095 @strong{Please note:} It is potentially deadly if you declare a
2096 @samp{Q...} variable in two different modules. The two calls to
2097 @code{defsymbol()} are no problem, but some linkers will complain about
2098 multiply-defined symbols. The most insidious aspect of this is that
2099 often the link will succeed anyway, but then the resulting executable
2100 will sometimes crash in obscure ways during certain operations! To
2101 avoid this problem, declare any symbols with common names (such as
2102 @code{text}) that are not obviously associated with this particular
2103 module in the module @file{general.c}.
2105 Global variables whose names begin with @samp{V} are variables that
2106 contain Lisp objects. The convention here is that all global variables
2107 of type @code{Lisp_Object} begin with @samp{V}, and all others don't
2108 (including integer and boolean variables that have Lisp
2109 equivalents). Most of the time, these variables have equivalents in
2110 Lisp, but some don't. Those that do are declared this way by a call to
2111 @code{DEFVAR_LISP()} in the @code{vars_of_*()} initializer for the
2112 module. What this does is create a special @dfn{symbol-value-forward}
2113 Lisp object that contains a pointer to the C variable, intern a symbol
2114 whose name is as specified in the call to @code{DEFVAR_LISP()}, and set
2115 its value to the symbol-value-forward Lisp object; it also calls
2116 @code{staticpro()} on the C variable to tell the garbage-collection
2117 mechanism about the variable. When @code{eval} (or actually
2118 @code{symbol-value}) encounters this special object in the process of
2119 retrieving a variable's value, it follows the indirection to the C
2120 variable and gets its value. @code{setq} does similar things so that
2121 the C variable gets changed.
2123 Whether or not you @code{DEFVAR_LISP()} a variable, you need to
2124 initialize it in the @code{vars_of_*()} function; otherwise it will end
2125 up as all zeroes, which is the integer 0 (@emph{not} @code{nil}), and
2126 this is probably not what you want. Also, if the variable is not
2127 @code{DEFVAR_LISP()}ed, @strong{you must call} @code{staticpro()} on the
2128 C variable in the @code{vars_of_*()} function. Otherwise, the
2129 garbage-collection mechanism won't know that the object in this variable
2130 is in use, and will happily collect it and reuse its storage for another
2131 Lisp object, and you will be the one who's unhappy when you can't figure
2132 out how your variable got overwritten.
2134 @node Coding for Mule, Techniques for XEmacs Developers, Adding Global Lisp Variables, Rules When Writing New C Code
2135 @section Coding for Mule
2136 @cindex Coding for Mule
2138 Although Mule support is not compiled by default in XEmacs, many people
2139 are using it, and we consider it crucial that new code works correctly
2140 with multibyte characters. This is not hard; it is only a matter of
2141 following several simple user-interface guidelines. Even if you never
2142 compile with Mule, with a little practice you will find it quite easy
2143 to code Mule-correctly.
2145 Note that these guidelines are not necessarily tied to the current Mule
2146 implementation; they are also a good idea to follow on the grounds of
2147 code generalization for future I18N work.
2150 * Character-Related Data Types::
2151 * Working With Character and Byte Positions::
2152 * Conversion to and from External Data::
2153 * General Guidelines for Writing Mule-Aware Code::
2154 * An Example of Mule-Aware Code::
2157 @node Character-Related Data Types, Working With Character and Byte Positions, Coding for Mule, Coding for Mule
2158 @subsection Character-Related Data Types
2160 First, let's review the basic character-related datatypes used by
2161 XEmacs. Note that the separate @code{typedef}s are not mandatory in the
2162 current implementation (all of them boil down to @code{unsigned char} or
2163 @code{int}), but they improve clarity of code a great deal, because one
2164 glance at the declaration can tell the intended use of the variable.
2169 An @code{Emchar} holds a single Emacs character.
2171 Obviously, the equality between characters and bytes is lost in the Mule
2172 world. Characters can be represented by one or more bytes in the
2173 buffer, and @code{Emchar} is the C type large enough to hold any
2176 Without Mule support, an @code{Emchar} is equivalent to an
2177 @code{unsigned char}.
2181 The data representing the text in a buffer or string is logically a set
2184 XEmacs does not work with the same character formats all the time; when
2185 reading characters from the outside, it decodes them to an internal
2186 format, and likewise encodes them when writing. @code{Bufbyte} (in fact
2187 @code{unsigned char}) is the basic unit of XEmacs internal buffers and
2188 strings format. A @code{Bufbyte *} is the type that points at text
2189 encoded in the variable-width internal encoding.
2191 One character can correspond to one or more @code{Bufbyte}s. In the
2192 current Mule implementation, an ASCII character is represented by the
2193 same @code{Bufbyte}, and other characters are represented by a sequence
2194 of two or more @code{Bufbyte}s.
2196 Without Mule support, there are exactly 256 characters, implicitly
2197 Latin-1, and each character is represented using one @code{Bufbyte}, and
2198 there is a one-to-one correspondence between @code{Bufbyte}s and
2205 A @code{Bufpos} represents a character position in a buffer or string.
2206 A @code{Charcount} represents a number (count) of characters.
2207 Logically, subtracting two @code{Bufpos} values yields a
2208 @code{Charcount} value. Although all of these are @code{typedef}ed to
2209 @code{EMACS_INT}, we use them in preference to @code{EMACS_INT} to make
2210 it clear what sort of position is being used.
2212 @code{Bufpos} and @code{Charcount} values are the only ones that are
2213 ever visible to Lisp.
2219 A @code{Bytind} represents a byte position in a buffer or string. A
2220 @code{Bytecount} represents the distance between two positions, in bytes.
2221 The relationship between @code{Bytind} and @code{Bytecount} is the same
2222 as the relationship between @code{Bufpos} and @code{Charcount}.
2228 When dealing with the outside world, XEmacs works with @code{Extbyte}s,
2229 which are equivalent to @code{unsigned char}. Obviously, an
2230 @code{Extcount} is the distance between two @code{Extbyte}s. Extbytes
2231 and Extcounts are not all that frequent in XEmacs code.
2234 @node Working With Character and Byte Positions, Conversion to and from External Data, Character-Related Data Types, Coding for Mule
2235 @subsection Working With Character and Byte Positions
2237 Now that we have defined the basic character-related types, we can look
2238 at the macros and functions designed for work with them and for
2239 conversion between them. Most of these macros are defined in
2240 @file{buffer.h}, and we don't discuss all of them here, but only the
2241 most important ones. Examining the existing code is the best way to
2245 @item MAX_EMCHAR_LEN
2246 @cindex MAX_EMCHAR_LEN
2247 This preprocessor constant is the maximum number of buffer bytes to
2248 represent an Emacs character in the variable width internal encoding.
2249 It is useful when allocating temporary strings to keep a known number of
2250 characters. For instance:
2258 /* Allocate place for @var{cclen} characters. */
2259 Bufbyte *buf = (Bufbyte *)alloca (cclen * MAX_EMCHAR_LEN);
2264 If you followed the previous section, you can guess that, logically,
2265 multiplying a @code{Charcount} value with @code{MAX_EMCHAR_LEN} produces
2266 a @code{Bytecount} value.
2268 In the current Mule implementation, @code{MAX_EMCHAR_LEN} equals 4.
2269 Without Mule, it is 1.
2271 @item charptr_emchar
2272 @itemx set_charptr_emchar
2273 @cindex charptr_emchar
2274 @cindex set_charptr_emchar
2275 The @code{charptr_emchar} macro takes a @code{Bufbyte} pointer and
2276 returns the @code{Emchar} stored at that position. If it were a
2277 function, its prototype would be:
2280 Emchar charptr_emchar (Bufbyte *p);
2283 @code{set_charptr_emchar} stores an @code{Emchar} to the specified byte
2284 position. It returns the number of bytes stored:
2287 Bytecount set_charptr_emchar (Bufbyte *p, Emchar c);
2290 It is important to note that @code{set_charptr_emchar} is safe only for
2291 appending a character at the end of a buffer, not for overwriting a
2292 character in the middle. This is because the width of characters
2293 varies, and @code{set_charptr_emchar} cannot resize the string if it
2294 writes, say, a two-byte character where a single-byte character used to
2297 A typical use of @code{set_charptr_emchar} can be demonstrated by this
2298 example, which copies characters from buffer @var{buf} to a temporary
2305 for (pos = beg; pos < end; pos++)
2307 Emchar c = BUF_FETCH_CHAR (buf, pos);
2308 p += set_charptr_emchar (buf, c);
2314 Note how @code{set_charptr_emchar} is used to store the @code{Emchar}
2315 and increment the counter, at the same time.
2321 These two macros increment and decrement a @code{Bufbyte} pointer,
2322 respectively. They will adjust the pointer by the appropriate number of
2323 bytes according to the byte length of the character stored there. Both
2324 macros assume that the memory address is located at the beginning of a
2327 Without Mule support, @code{INC_CHARPTR (p)} and @code{DEC_CHARPTR (p)}
2328 simply expand to @code{p++} and @code{p--}, respectively.
2330 @item bytecount_to_charcount
2331 @cindex bytecount_to_charcount
2332 Given a pointer to a text string and a length in bytes, return the
2333 equivalent length in characters.
2336 Charcount bytecount_to_charcount (Bufbyte *p, Bytecount bc);
2339 @item charcount_to_bytecount
2340 @cindex charcount_to_bytecount
2341 Given a pointer to a text string and a length in characters, return the
2342 equivalent length in bytes.
2345 Bytecount charcount_to_bytecount (Bufbyte *p, Charcount cc);
2348 @item charptr_n_addr
2349 @cindex charptr_n_addr
2350 Return a pointer to the beginning of the character offset @var{cc} (in
2351 characters) from @var{p}.
2354 Bufbyte *charptr_n_addr (Bufbyte *p, Charcount cc);
2358 @node Conversion to and from External Data, General Guidelines for Writing Mule-Aware Code, Working With Character and Byte Positions, Coding for Mule
2359 @subsection Conversion to and from External Data
2361 When an external function, such as a C library function, returns a
2362 @code{char} pointer, you should almost never treat it as @code{Bufbyte}.
2363 This is because these returned strings may contain 8bit characters which
2364 can be misinterpreted by XEmacs, and cause a crash. Likewise, when
2365 exporting a piece of internal text to the outside world, you should
2366 always convert it to an appropriate external encoding, lest the internal
2367 stuff (such as the infamous \201 characters) leak out.
2369 The interface to conversion between the internal and external
2370 representations of text are the numerous conversion macros defined in
2371 @file{buffer.h}. There used to be a fixed set of external formats
2372 supported by these macros, but now any coding system can be used with
2373 these macros. The coding system alias mechanism is used to create the
2374 following logical coding systems, which replace the fixed external
2375 formats. The (dontusethis-set-symbol-value-handler) mechanism was
2376 enhanced to make this possible (more work on that is needed - like
2377 remove the @code{dontusethis-} prefix).
2381 This is the simplest format and is what we use in the absence of a more
2382 appropriate format. This converts according to the @code{binary} coding
2387 On input, bytes 0--255 are converted into (implicitly Latin-1)
2388 characters 0--255. A non-Mule xemacs doesn't really know about
2389 different character sets and the fonts to display them, so the bytes can
2390 be treated as text in different 1-byte encodings by simply setting the
2391 appropriate fonts. So in a sense, non-Mule xemacs is a multi-lingual
2392 editor if, for example, different fonts are used to display text in
2393 different buffers, faces, or windows. The specifier mechanism gives the
2394 user complete control over this kind of behavior.
2396 On output, characters 0--255 are converted into bytes 0--255 and other
2397 characters are converted into `~'.
2401 Format used for filenames. This is user-definable via either the
2402 @code{file-name-coding-system} or @code{pathname-coding-system} (now
2403 obsolete) variables.
2406 Format used for the external Unix environment---@code{argv[]}, stuff
2407 from @code{getenv()}, stuff from the @file{/etc/passwd} file, etc.
2408 Currently this is the same as Qfile_name. The two should be
2409 distinguished for clarity and possible future separation.
2412 Compound--text format. This is the standard X11 format used for data
2413 stored in properties, selections, and the like. This is an 8-bit
2414 no-lock-shift ISO2022 coding system. This is a real coding system,
2415 unlike Qfile_name, which is user-definable.
2418 There are two fundamental macros to convert between external and
2421 @code{TO_INTERNAL_FORMAT} converts external data to internal format, and
2422 @code{TO_EXTERNAL_FORMAT} converts the other way around. The arguments
2423 each of these receives are a source type, a source, a sink type, a sink,
2424 and a coding system (or a symbol naming a coding system).
2426 A typical call looks like
2428 TO_EXTERNAL_FORMAT (LISP_STRING, str, C_STRING_MALLOC, ptr, Qfile_name);
2431 which means that the contents of the lisp string @code{str} are written
2432 to a malloc'ed memory area which will be pointed to by @code{ptr}, after
2433 the function returns. The conversion will be done using the
2434 @code{file-name} coding system, which will be controlled by the user
2435 indirectly by setting or binding the variable
2436 @code{file-name-coding-system}.
2438 Some sources and sinks require two C variables to specify. We use some
2439 preprocessor magic to allow different source and sink types, and even
2440 different numbers of arguments to specify different types of sources and
2443 So we can have a call that looks like
2445 TO_INTERNAL_FORMAT (DATA, (ptr, len),
2450 The parenthesized argument pairs are required to make the preprocessor
2453 Here are the different source and sink types:
2456 @item @code{DATA, (ptr, len),}
2457 input data is a fixed buffer of size @var{len} at address @var{ptr}
2458 @item @code{ALLOCA, (ptr, len),}
2459 output data is placed in an alloca()ed buffer of size @var{len} pointed to by @var{ptr}
2460 @item @code{MALLOC, (ptr, len),}
2461 output data is in a malloc()ed buffer of size @var{len} pointed to by @var{ptr}
2462 @item @code{C_STRING_ALLOCA, ptr,}
2463 equivalent to @code{ALLOCA (ptr, len_ignored)} on output.
2464 @item @code{C_STRING_MALLOC, ptr,}
2465 equivalent to @code{MALLOC (ptr, len_ignored)} on output
2466 @item @code{C_STRING, ptr,}
2467 equivalent to @code{DATA, (ptr, strlen (ptr) + 1)} on input
2468 @item @code{LISP_STRING, string,}
2469 input or output is a Lisp_Object of type string
2470 @item @code{LISP_BUFFER, buffer,}
2471 output is written to @code{(point)} in lisp buffer @var{buffer}
2472 @item @code{LISP_LSTREAM, lstream,}
2473 input or output is a Lisp_Object of type lstream
2474 @item @code{LISP_OPAQUE, object,}
2475 input or output is a Lisp_Object of type opaque
2478 Often, the data is being converted to a '\0'-byte-terminated string,
2479 which is the format required by many external system C APIs. For these
2480 purposes, a source type of @code{C_STRING} or a sink type of
2481 @code{C_STRING_ALLOCA} or @code{C_STRING_MALLOC} is appropriate.
2482 Otherwise, we should try to keep XEmacs '\0'-byte-clean, which means
2483 using (ptr, len) pairs.
2485 The sinks to be specified must be lvalues, unless they are the lisp
2486 object types @code{LISP_LSTREAM} or @code{LISP_BUFFER}.
2488 For the sink types @code{ALLOCA} and @code{C_STRING_ALLOCA}, the
2489 resulting text is stored in a stack-allocated buffer, which is
2490 automatically freed on returning from the function. However, the sink
2491 types @code{MALLOC} and @code{C_STRING_MALLOC} return @code{xmalloc()}ed
2492 memory. The caller is responsible for freeing this memory using
2495 Note that it doesn't make sense for @code{LISP_STRING} to be a source
2496 for @code{TO_INTERNAL_FORMAT} or a sink for @code{TO_EXTERNAL_FORMAT}.
2497 You'll get an assertion failure if you try.
2500 @node General Guidelines for Writing Mule-Aware Code, An Example of Mule-Aware Code, Conversion to and from External Data, Coding for Mule
2501 @subsection General Guidelines for Writing Mule-Aware Code
2503 This section contains some general guidance on how to write Mule-aware
2504 code, as well as some pitfalls you should avoid.
2507 @item Never use @code{char} and @code{char *}.
2508 In XEmacs, the use of @code{char} and @code{char *} is almost always a
2509 mistake. If you want to manipulate an Emacs character from ``C'', use
2510 @code{Emchar}. If you want to examine a specific octet in the internal
2511 format, use @code{Bufbyte}. If you want a Lisp-visible character, use a
2512 @code{Lisp_Object} and @code{make_char}. If you want a pointer to move
2513 through the internal text, use @code{Bufbyte *}. Also note that you
2514 almost certainly do not need @code{Emchar *}.
2516 @item Be careful not to confuse @code{Charcount}, @code{Bytecount}, and @code{Bufpos}.
2517 The whole point of using different types is to avoid confusion about the
2518 use of certain variables. Lest this effect be nullified, you need to be
2519 careful about using the right types.
2521 @item Always convert external data
2522 It is extremely important to always convert external data, because
2523 XEmacs can crash if unexpected 8bit sequences are copied to its internal
2526 This means that when a system function, such as @code{readdir}, returns
2527 a string, you may need to convert it using one of the conversion macros
2528 described in the previous chapter, before passing it further to Lisp.
2530 Actually, most of the basic system functions that accept '\0'-terminated
2531 string arguments, like @code{stat()} and @code{open()}, have been
2532 @strong{encapsulated} so that they are they @code{always} do internal to
2533 external conversion themselves. This means you must pass internally
2534 encoded data, typically the @code{XSTRING_DATA} of a Lisp_String to
2535 these functions. This is actually a design bug, since it unexpectedly
2536 changes the semantics of the system functions. A better design would be
2537 to provide separate versions of these system functions that accepted
2538 Lisp_Objects which were lisp strings in place of their current
2539 @code{char *} arguments.
2542 int stat_lisp (Lisp_Object path, struct stat *buf); /* Implement me */
2545 Also note that many internal functions, such as @code{make_string},
2546 accept Bufbytes, which removes the need for them to convert the data
2547 they receive. This increases efficiency because that way external data
2548 needs to be decoded only once, when it is read. After that, it is
2549 passed around in internal format.
2552 @node An Example of Mule-Aware Code, , General Guidelines for Writing Mule-Aware Code, Coding for Mule
2553 @subsection An Example of Mule-Aware Code
2555 As an example of Mule-aware code, we will analyze the @code{string}
2556 function, which conses up a Lisp string from the character arguments it
2557 receives. Here is the definition, pasted from @code{alloc.c}:
2561 DEFUN ("string", Fstring, 0, MANY, 0, /*
2562 Concatenate all the argument characters and make the result a string.
2564 (int nargs, Lisp_Object *args))
2566 Bufbyte *storage = alloca_array (Bufbyte, nargs * MAX_EMCHAR_LEN);
2567 Bufbyte *p = storage;
2569 for (; nargs; nargs--, args++)
2571 Lisp_Object lisp_char = *args;
2572 CHECK_CHAR_COERCE_INT (lisp_char);
2573 p += set_charptr_emchar (p, XCHAR (lisp_char));
2575 return make_string (storage, p - storage);
2580 Now we can analyze the source line by line.
2582 Obviously, string will be as long as there are arguments to the
2583 function. This is why we allocate @code{MAX_EMCHAR_LEN} * @var{nargs}
2584 bytes on the stack, i.e. the worst-case number of bytes for @var{nargs}
2585 @code{Emchar}s to fit in the string.
2587 Then, the loop checks that each element is a character, converting
2588 integers in the process. Like many other functions in XEmacs, this
2589 function silently accepts integers where characters are expected, for
2590 historical and compatibility reasons. Unless you know what you are
2591 doing, @code{CHECK_CHAR} will also suffice. @code{XCHAR (lisp_char)}
2592 extracts the @code{Emchar} from the @code{Lisp_Object}, and
2593 @code{set_charptr_emchar} stores it to storage, increasing @code{p} in
2596 Other instructive examples of correct coding under Mule can be found all
2597 over the XEmacs code. For starters, I recommend
2598 @code{Fnormalize_menu_item_name} in @file{menubar.c}. After you have
2599 understood this section of the manual and studied the examples, you can
2600 proceed writing new Mule-aware code.
2602 @node Techniques for XEmacs Developers, , Coding for Mule, Rules When Writing New C Code
2603 @section Techniques for XEmacs Developers
2605 To make a purified XEmacs, do: @code{make puremacs}.
2606 To make a quantified XEmacs, do: @code{make quantmacs}.
2608 You simply can't dump Quantified and Purified images (unless using the
2609 portable dumper). Purify gets confused when xemacs frees memory in one
2610 process that was allocated in a @emph{different} process on a different
2611 machine!. Run it like so:
2613 temacs -batch -l loadup.el run-temacs @var{xemacs-args...}
2616 Before you go through the trouble, are you compiling with all
2617 debugging and error-checking off? If not, try that first. Be warned
2618 that while Quantify is directly responsible for quite a few
2619 optimizations which have been made to XEmacs, doing a run which
2620 generates results which can be acted upon is not necessarily a trivial
2623 Also, if you're still willing to do some runs make sure you configure
2624 with the @samp{--quantify} flag. That will keep Quantify from starting
2625 to record data until after the loadup is completed and will shut off
2626 recording right before it shuts down (which generates enough bogus data
2627 to throw most results off). It also enables three additional elisp
2628 commands: @code{quantify-start-recording-data},
2629 @code{quantify-stop-recording-data} and @code{quantify-clear-data}.
2631 If you want to make XEmacs faster, target your favorite slow benchmark,
2632 run a profiler like Quantify, @code{gprof}, or @code{tcov}, and figure
2633 out where the cycles are going. Specific projects:
2637 Make the garbage collector faster. Figure out how to write an
2638 incremental garbage collector.
2640 Write a compiler that takes bytecode and spits out C code.
2641 Unfortunately, you will then need a C compiler and a more fully
2642 developed module system.
2646 Speed up syntax highlighting. Maybe moving some of the syntax
2647 highlighting capabilities into C would make a difference.
2649 Implement tail recursion in Emacs Lisp (hard!).
2652 Unfortunately, Emacs Lisp is slow, and is going to stay slow. Function
2653 calls in elisp are especially expensive. Iterating over a long list is
2654 going to be 30 times faster implemented in C than in Elisp.
2656 Heavily used small code fragments need to be fast. The traditional way
2657 to implement such code fragments in C is with macros. But macros in C
2658 are known to be broken.
2660 Macro arguments that are repeatedly evaluated may suffer from repeated
2661 side effects or suboptimal performance.
2663 Variable names used in macros may collide with caller's variables,
2664 causing (at least) unwanted compiler warnings.
2666 In order to solve these problems, and maintain statement semantics, one
2667 should use the @code{do @{ ... @} while (0)} trick while trying to
2668 reference macro arguments exactly once using local variables.
2670 Let's take a look at this poor macro definition:
2673 #define MARK_OBJECT(obj) \
2674 if (!marked_p (obj)) mark_object (obj), did_mark = 1
2677 This macro evaluates its argument twice, and also fails if used like this:
2679 if (flag) MARK_OBJECT (obj); else do_something();
2682 A much better definition is
2685 #define MARK_OBJECT(obj) do @{ \
2686 Lisp_Object mo_obj = (obj); \
2687 if (!marked_p (mo_obj)) \
2689 mark_object (mo_obj); \
2695 Notice the elimination of double evaluation by using the local variable
2696 with the obscure name. Writing safe and efficient macros requires great
2697 care. The one problem with macros that cannot be portably worked around
2698 is, since a C block has no value, a macro used as an expression rather
2699 than a statement cannot use the techniques just described to avoid
2700 multiple evaluation.
2702 In most cases where a macro has function semantics, an inline function
2703 is a better implementation technique. Modern compiler optimizers tend
2704 to inline functions even if they have no @code{inline} keyword, and
2705 configure magic ensures that the @code{inline} keyword can be safely
2706 used as an additional compiler hint. Inline functions used in a single
2707 .c files are easy. The function must already be defined to be
2708 @code{static}. Just add another @code{inline} keyword to the
2713 heavily_used_small_function (int arg)
2719 Inline functions in header files are trickier, because we would like to
2720 make the following optimization if the function is @emph{not} inlined
2721 (for example, because we're compiling for debugging). We would like the
2722 function to be defined externally exactly once, and each calling
2723 translation unit would create an external reference to the function,
2724 instead of including a definition of the inline function in the object
2725 code of every translation unit that uses it. This optimization is
2726 currently only available for gcc. But you don't have to worry about the
2727 trickiness; just define your inline functions in header files using this
2732 i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg);
2734 i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg)
2740 The declaration right before the definition is to prevent warnings when
2741 compiling with @code{gcc -Wmissing-declarations}. I consider issuing
2742 this warning for inline functions a gcc bug, but the gcc maintainers disagree.
2744 Every header which contains inline functions, either directly by using
2745 @code{INLINE_HEADER} or indirectly by using @code{DECLARE_LRECORD} must
2746 be added to @file{inline.c}'s includes to make the optimization
2747 described above work. (Optimization note: if all INLINE_HEADER
2748 functions are in fact inlined in all translation units, then the linker
2749 can just discard @code{inline.o}, since it contains only unreferenced code).
2751 To get started debugging XEmacs, take a look at the @file{.gdbinit} and
2752 @file{.dbxrc} files in the @file{src} directory. See the section in the
2753 XEmacs FAQ on How to Debug an XEmacs problem with a debugger.
2755 After making source code changes, run @code{make check} to ensure that
2756 you haven't introduced any regressions. If you want to make xemacs more
2757 reliable, please improve the test suite in @file{tests/automated}.
2759 Did you make sure you didn't introduce any new compiler warnings?
2761 Before submitting a patch, please try compiling at least once with
2764 configure --with-mule --with-union-type --error-checking=all
2767 Here are things to know when you create a new source file:
2771 All @file{.c} files should @code{#include <config.h>} first. Almost all
2772 @file{.c} files should @code{#include "lisp.h"} second.
2775 Generated header files should be included using the @code{#include <...>} syntax,
2776 not the @code{#include "..."} syntax. The generated headers are:
2778 @file{config.h sheap-adjust.h paths.h Emacs.ad.h}
2780 The basic rule is that you should assume builds using @code{--srcdir}
2781 and the @code{#include <...>} syntax needs to be used when the
2782 to-be-included generated file is in a potentially different directory
2783 @emph{at compile time}. The non-obvious C rule is that @code{#include "..."}
2784 means to search for the included file in the same directory as the
2785 including file, @emph{not} in the current directory.
2788 Header files should @emph{not} include @code{<config.h>} and
2789 @code{"lisp.h"}. It is the responsibility of the @file{.c} files that
2794 Here is a checklist of things to do when creating a new lisp object type
2803 add definitions of @code{syms_of_@var{foo}}, etc. to @file{@var{foo}.c}
2805 add declarations of @code{syms_of_@var{foo}}, etc. to @file{symsinit.h}
2807 add calls to @code{syms_of_@var{foo}}, etc. to @file{emacs.c}
2809 add definitions of macros like @code{CHECK_@var{FOO}} and
2810 @code{@var{FOO}P} to @file{@var{foo}.h}
2812 add the new type index to @code{enum lrecord_type}
2814 add a DEFINE_LRECORD_IMPLEMENTATION call to @file{@var{foo}.c}
2816 add an INIT_LRECORD_IMPLEMENTATION call to @code{syms_of_@var{foo}.c}
2819 @node A Summary of the Various XEmacs Modules, Allocation of Objects in XEmacs Lisp, Rules When Writing New C Code, Top
2820 @chapter A Summary of the Various XEmacs Modules
2822 This is accurate as of XEmacs 20.0.
2825 * Low-Level Modules::
2826 * Basic Lisp Modules::
2827 * Modules for Standard Editing Operations::
2828 * Editor-Level Control Flow Modules::
2829 * Modules for the Basic Displayable Lisp Objects::
2830 * Modules for other Display-Related Lisp Objects::
2831 * Modules for the Redisplay Mechanism::
2832 * Modules for Interfacing with the File System::
2833 * Modules for Other Aspects of the Lisp Interpreter and Object System::
2834 * Modules for Interfacing with the Operating System::
2835 * Modules for Interfacing with X Windows::
2836 * Modules for Internationalization::
2839 @node Low-Level Modules, Basic Lisp Modules, A Summary of the Various XEmacs Modules, A Summary of the Various XEmacs Modules
2840 @section Low-Level Modules
2846 This is automatically generated from @file{config.h.in} based on the
2847 results of configure tests and user-selected optional features and
2848 contains preprocessor definitions specifying the nature of the
2849 environment in which XEmacs is being compiled.
2857 This is automatically generated from @file{paths.h.in} based on supplied
2858 configure values, and allows for non-standard installed configurations
2859 of the XEmacs directories. It's currently broken, though.
2868 @file{emacs.c} contains @code{main()} and other code that performs the most
2869 basic environment initializations and handles shutting down the XEmacs
2870 process (this includes @code{kill-emacs}, the normal way that XEmacs is
2871 exited; @code{dump-emacs}, which is used during the build process to
2872 write out the XEmacs executable; @code{run-emacs-from-temacs}, which can
2873 be used to start XEmacs directly when temacs has finished loading all
2874 the Lisp code; and emergency code to handle crashes [XEmacs tries to
2875 auto-save all files before it crashes]).
2877 Low-level code that directly interacts with the Unix signal mechanism,
2878 however, is in @file{signal.c}. Note that this code does not handle system
2879 dependencies in interfacing to signals; that is handled using the
2880 @file{syssignal.h} header file, described in section J below.
2904 These modules contain code dumping out the XEmacs executable on various
2905 different systems. (This process is highly machine-specific and
2906 requires intimate knowledge of the executable format and the memory map
2907 of the process.) Only one of these modules is actually used; this is
2908 chosen by @file{configure}.
2918 These modules are used in conjunction with the dump mechanism. On some
2919 systems, an alternative version of the C startup code (the actual code
2920 that receives control from the operating system when the process is
2921 started, and which calls @code{main()}) is required so that the dumping
2922 process works properly; @file{crt0.c} provides this.
2924 @file{pre-crt0.c} and @file{lastfile.c} should be the very first and
2925 very last file linked, respectively. (Actually, this is not really true.
2926 @file{lastfile.c} should be after all Emacs modules whose initialized
2927 data should be made constant, and before all other Emacs files and all
2928 libraries. In particular, the allocation modules @file{gmalloc.c},
2929 @file{alloca.c}, etc. are normally placed past @file{lastfile.c}, and
2930 all of the files that implement Xt widget classes @emph{must} be placed
2931 after @file{lastfile.c} because they contain various structures that
2932 must be statically initialized and into which Xt writes at various
2933 times.) @file{pre-crt0.c} and @file{lastfile.c} contain exported symbols
2934 that are used to determine the start and end of XEmacs' initialized
2935 data space when dumping.
2950 These handle basic C allocation of memory. @file{alloca.c} is an emulation of
2951 the stack allocation function @code{alloca()} on machines that lack
2952 this. (XEmacs makes extensive use of @code{alloca()} in its code.)
2954 @file{gmalloc.c} and @file{malloc.c} are two implementations of the standard C
2955 functions @code{malloc()}, @code{realloc()} and @code{free()}. They are
2956 often used in place of the standard system-provided @code{malloc()}
2957 because they usually provide a much faster implementation, at the
2958 expense of additional memory use. @file{gmalloc.c} is a newer implementation
2959 that is much more memory-efficient for large allocations than @file{malloc.c},
2960 and should always be preferred if it works. (At one point, @file{gmalloc.c}
2961 didn't work on some systems where @file{malloc.c} worked; but this should be
2964 @cindex relocating allocator
2965 @file{ralloc.c} is the @dfn{relocating allocator}. It provides
2966 functions similar to @code{malloc()}, @code{realloc()} and @code{free()}
2967 that allocate memory that can be dynamically relocated in memory. The
2968 advantage of this is that allocated memory can be shuffled around to
2969 place all the free memory at the end of the heap, and the heap can then
2970 be shrunk, releasing the memory back to the operating system. The use
2971 of this can be controlled with the configure option @code{--rel-alloc};
2972 if enabled, memory allocated for buffers will be relocatable, so that if
2973 a very large file is visited and the buffer is later killed, the memory
2974 can be released to the operating system. (The disadvantage of this
2975 mechanism is that it can be very slow. On systems with the
2976 @code{mmap()} system call, the XEmacs version of @file{ralloc.c} uses
2977 this to move memory around without actually having to block-copy it,
2978 which can speed things up; but it can still cause noticeable performance
2981 @file{free-hook.c} contains some debugging functions for checking for invalid
2982 arguments to @code{free()}.
2984 @file{vm-limit.c} contains some functions that warn the user when memory is
2985 getting low. These are callback functions that are called by @file{gmalloc.c}
2986 and @file{malloc.c} at appropriate times.
2988 @file{getpagesize.h} provides a uniform interface for retrieving the size of a
2989 page in virtual memory. @file{mem-limits.h} provides a uniform interface for
2990 retrieving the total amount of available virtual memory. Both are
2991 similar in spirit to the @file{sys*.h} files described in section J, below.
3001 These implement a couple of basic C data types to facilitate memory
3002 allocation. The @code{Blocktype} type efficiently manages the
3003 allocation of fixed-size blocks by minimizing the number of times that
3004 @code{malloc()} and @code{free()} are called. It allocates memory in
3005 large chunks, subdivides the chunks into blocks of the proper size, and
3006 returns the blocks as requested. When blocks are freed, they are placed
3007 onto a linked list, so they can be efficiently reused. This data type
3008 is not much used in XEmacs currently, because it's a fairly new
3011 @cindex dynamic array
3012 The @code{Dynarr} type implements a @dfn{dynamic array}, which is
3013 similar to a standard C array but has no fixed limit on the number of
3014 elements it can contain. Dynamic arrays can hold elements of any type,
3015 and when you add a new element, the array automatically resizes itself
3016 if it isn't big enough. Dynarrs are extensively used in the redisplay
3025 This module is used in connection with inline functions (available in
3026 some compilers). Often, inline functions need to have a corresponding
3027 non-inline function that does the same thing. This module is where they
3028 reside. It contains no actual code, but defines some special flags that
3029 cause inline functions defined in header files to be rendered as actual
3030 functions. It then includes all header files that contain any inline
3031 function definitions, so that each one gets a real function equivalent.
3040 These functions provide a system for doing internal consistency checks
3041 during code development. This system is not currently used; instead the
3042 simpler @code{assert()} macro is used along with the various checks
3043 provided by the @samp{--error-check-*} configuration options.
3051 This is actually the source for a small, self-contained program
3052 used during building.
3059 This is not currently used.
3063 @node Basic Lisp Modules, Modules for Standard Editing Operations, Low-Level Modules, A Summary of the Various XEmacs Modules
3064 @section Basic Lisp Modules
3075 These are the basic header files for all XEmacs modules. Each module
3076 includes @file{lisp.h}, which brings the other header files in.
3077 @file{lisp.h} contains the definitions of the structures and extractor
3078 and constructor macros for the basic Lisp objects and various other
3079 basic definitions for the Lisp environment, as well as some
3080 general-purpose definitions (e.g. @code{min()} and @code{max()}).
3081 @file{lisp.h} includes either @file{lisp-disunion.h} or
3082 @file{lisp-union.h}, depending on whether @code{USE_UNION_TYPE} is
3083 defined. These files define the typedef of the Lisp object itself (as
3084 described above) and the low-level macros that hide the actual
3085 implementation of the Lisp object. All extractor and constructor macros
3086 for particular types of Lisp objects are defined in terms of these
3089 As a general rule, all typedefs should go into the typedefs section of
3090 @file{lisp.h} rather than into a module-specific header file even if the
3091 structure is defined elsewhere. This allows function prototypes that
3092 use the typedef to be placed into other header files. Forward structure
3093 declarations (i.e. a simple declaration like @code{struct foo;} where
3094 the structure itself is defined elsewhere) should be placed into the
3095 typedefs section as necessary.
3097 @file{lrecord.h} contains the basic structures and macros that implement
3098 all record-type Lisp objects---i.e. all objects whose type is a field
3099 in their C structure, which includes all objects except the few most
3102 @file{lisp.h} contains prototypes for most of the exported functions in
3103 the various modules. Lisp primitives defined using @code{DEFUN} that
3104 need to be called by C code should be declared using @code{EXFUN}.
3105 Other function prototypes should be placed either into the appropriate
3106 section of @code{lisp.h}, or into a module-specific header file,
3107 depending on how general-purpose the function is and whether it has
3108 special-purpose argument types requiring definitions not in
3109 @file{lisp.h}.) All initialization functions are prototyped in
3118 The large module @file{alloc.c} implements all of the basic allocation and
3119 garbage collection for Lisp objects. The most commonly used Lisp
3120 objects are allocated in chunks, similar to the Blocktype data type
3121 described above; others are allocated in individually @code{malloc()}ed
3122 blocks. This module provides the foundation on which all other aspects
3123 of the Lisp environment sit, and is the first module initialized at
3126 Note that @file{alloc.c} provides a series of generic functions that are
3127 not dependent on any particular object type, and interfaces to
3128 particular types of objects using a standardized interface of
3129 type-specific methods. This scheme is a fundamental principle of
3130 object-oriented programming and is heavily used throughout XEmacs. The
3131 great advantage of this is that it allows for a clean separation of
3132 functionality into different modules---new classes of Lisp objects, new
3133 event interfaces, new device types, new stream interfaces, etc. can be
3134 added transparently without affecting code anywhere else in XEmacs.
3135 Because the different subsystems are divided into general and specific
3136 code, adding a new subtype within a subsystem will in general not
3137 require changes to the generic subsystem code or affect any of the other
3138 subtypes in the subsystem; this provides a great deal of robustness to
3147 This module contains all of the functions to handle the flow of control.
3148 This includes the mechanisms of defining functions, calling functions,
3149 traversing stack frames, and binding variables; the control primitives
3150 and other special forms such as @code{while}, @code{if}, @code{eval},
3151 @code{let}, @code{and}, @code{or}, @code{progn}, etc.; handling of
3152 non-local exits, unwind-protects, and exception handlers; entering the
3153 debugger; methods for the subr Lisp object type; etc. It does
3154 @emph{not} include the @code{read} function, the @code{print} function,
3155 or the handling of symbols and obarrays.
3157 @file{backtrace.h} contains some structures related to stack frames and the
3166 This module implements the Lisp reader and the @code{read} function,
3167 which converts text into Lisp objects, according to the read syntax of
3168 the objects, as described above. This is similar to the parser that is
3169 a part of all compilers.
3177 This module implements the Lisp print mechanism and the @code{print}
3178 function and related functions. This is the inverse of the Lisp reader
3179 -- it converts Lisp objects to a printed, textual representation.
3180 (Hopefully something that can be read back in using @code{read} to get
3181 an equivalent object.)
3191 @file{symbols.c} implements the handling of symbols, obarrays, and
3192 retrieving the values of symbols. Much of the code is devoted to
3193 handling the special @dfn{symbol-value-magic} objects that define
3194 special types of variables---this includes buffer-local variables,
3195 variable aliases, variables that forward into C variables, etc. This
3196 module is initialized extremely early (right after @file{alloc.c}),
3197 because it is here that the basic symbols @code{t} and @code{nil} are
3198 created, and those symbols are used everywhere throughout XEmacs.
3200 @file{symeval.h} contains the definitions of symbol structures and the
3201 @code{DEFVAR_LISP()} and related macros for declaring variables.
3211 These modules implement the methods and standard Lisp primitives for all
3212 the basic Lisp object types other than symbols (which are described
3213 above). @file{data.c} contains all the predicates (primitives that return
3214 whether an object is of a particular type); the integer arithmetic
3215 functions; and the basic accessor and mutator primitives for the various
3216 object types. @file{fns.c} contains all the standard predicates for working
3217 with sequences (where, abstractly speaking, a sequence is an ordered set
3218 of objects, and can be represented by a list, string, vector, or
3219 bit-vector); it also contains @code{equal}, perhaps on the grounds that
3220 bulk of the operation of @code{equal} is comparing sequences.
3221 @file{floatfns.c} contains methods and primitives for floats and floating-point
3231 @file{bytecode.c} implements the byte-code interpreter and
3232 compiled-function objects, and @file{bytecode.h} contains associated
3233 structures. Note that the byte-code @emph{compiler} is written in Lisp.
3238 @node Modules for Standard Editing Operations, Editor-Level Control Flow Modules, Basic Lisp Modules, A Summary of the Various XEmacs Modules
3239 @section Modules for Standard Editing Operations
3247 @file{buffer.c} implements the @dfn{buffer} Lisp object type. This
3248 includes functions that create and destroy buffers; retrieve buffers by
3249 name or by other properties; manipulate lists of buffers (remember that
3250 buffers are permanent objects and stored in various ordered lists);
3251 retrieve or change buffer properties; etc. It also contains the
3252 definitions of all the built-in buffer-local variables (which can be
3253 viewed as buffer properties). It does @emph{not} contain code to
3254 manipulate buffer-local variables (that's in @file{symbols.c}, described
3255 above); or code to manipulate the text in a buffer.
3257 @file{buffer.h} defines the structures associated with a buffer and the various
3258 macros for retrieving text from a buffer and special buffer positions
3259 (e.g. @code{point}, the default location for text insertion). It also
3260 contains macros for working with buffer positions and converting between
3261 their representations as character offsets and as byte offsets (under
3262 MULE, they are different, because characters can be multi-byte). It is
3263 one of the largest header files.
3265 @file{bufslots.h} defines the fields in the buffer structure that correspond to
3266 the built-in buffer-local variables. It is its own header file because
3267 it is included many times in @file{buffer.c}, as a way of iterating over all
3268 the built-in buffer-local variables.
3277 @file{insdel.c} contains low-level functions for inserting and deleting text in
3278 a buffer, keeping track of changed regions for use by redisplay, and
3279 calling any before-change and after-change functions that may have been
3280 registered for the buffer. It also contains the actual functions that
3281 convert between byte offsets and character offsets.
3283 @file{insdel.h} contains associated headers.
3291 This module implements the @dfn{marker} Lisp object type, which
3292 conceptually is a pointer to a text position in a buffer that moves
3293 around as text is inserted and deleted, so as to remain in the same
3294 relative position. This module doesn't actually move the markers around
3295 -- that's handled in @file{insdel.c}. This module just creates them and
3296 implements the primitives for working with them. As markers are simple
3297 objects, this does not entail much.
3299 Note that the standard arithmetic primitives (e.g. @code{+}) accept
3300 markers in place of integers and automatically substitute the value of
3301 @code{marker-position} for the marker, i.e. an integer describing the
3302 current buffer position of the marker.
3311 This module implements the @dfn{extent} Lisp object type, which is like
3312 a marker that works over a range of text rather than a single position.
3313 Extents are also much more complex and powerful than markers and have a
3314 more efficient (and more algorithmically complex) implementation. The
3315 implementation is described in detail in comments in @file{extents.c}.
3317 The code in @file{extents.c} works closely with @file{insdel.c} so that
3318 extents are properly moved around as text is inserted and deleted.
3319 There is also code in @file{extents.c} that provides information needed
3320 by the redisplay mechanism for efficient operation. (Remember that
3321 extents can have display properties that affect [sometimes drastically,
3322 as in the @code{invisible} property] the display of the text they
3331 @file{editfns.c} contains the standard Lisp primitives for working with
3332 a buffer's text, and calls the low-level functions in @file{insdel.c}.
3333 It also contains primitives for working with @code{point} (the default
3334 buffer insertion location).
3336 @file{editfns.c} also contains functions for retrieving various
3337 characteristics from the external environment: the current time, the
3338 process ID of the running XEmacs process, the name of the user who ran
3339 this XEmacs process, etc. It's not clear why this code is in
3351 These modules implement the basic @dfn{interactive} commands,
3352 i.e. user-callable functions. Commands, as opposed to other functions,
3353 have special ways of getting their parameters interactively (by querying
3354 the user), as opposed to having them passed in a normal function
3355 invocation. Many commands are not really meant to be called from other
3356 Lisp functions, because they modify global state in a way that's often
3357 undesired as part of other Lisp functions.
3359 @file{callint.c} implements the mechanism for querying the user for
3360 parameters and calling interactive commands. The bulk of this module is
3361 code that parses the interactive spec that is supplied with an
3362 interactive command.
3364 @file{cmds.c} implements the basic, most commonly used editing commands:
3365 commands to move around the current buffer and insert and delete
3366 characters. These commands are implemented using the Lisp primitives
3367 defined in @file{editfns.c}.
3369 @file{commands.h} contains associated structure definitions and prototypes.
3379 @file{search.c} implements the Lisp primitives for searching for text in
3380 a buffer, and some of the low-level algorithms for doing this. In
3381 particular, the fast fixed-string Boyer-Moore search algorithm is
3382 implemented in @file{search.c}. The low-level algorithms for doing
3383 regular-expression searching, however, are implemented in @file{regex.c}
3384 and @file{regex.h}. These two modules are largely independent of
3385 XEmacs, and are similar to (and based upon) the regular-expression
3386 routines used in @file{grep} and other GNU utilities.
3394 @file{doprnt.c} implements formatted-string processing, similar to
3395 @code{printf()} command in C.
3403 This module implements the undo mechanism for tracking buffer changes.
3404 Most of this could be implemented in Lisp.
3408 @node Editor-Level Control Flow Modules, Modules for the Basic Displayable Lisp Objects, Modules for Standard Editing Operations, A Summary of the Various XEmacs Modules
3409 @section Editor-Level Control Flow Modules
3419 These implement the handling of events (user input and other system
3422 @file{events.c} and @file{events.h} define the @dfn{event} Lisp object
3423 type and primitives for manipulating it.
3425 @file{event-stream.c} implements the basic functions for working with
3426 event queues, dispatching an event by looking it up in relevant keymaps
3427 and such, and handling timeouts; this includes the primitives
3428 @code{next-event} and @code{dispatch-event}, as well as related
3429 primitives such as @code{sit-for}, @code{sleep-for}, and
3430 @code{accept-process-output}. (@file{event-stream.c} is one of the
3431 hairiest and trickiest modules in XEmacs. Beware! You can easily mess
3434 @file{event-Xt.c} and @file{event-tty.c} implement the low-level
3435 interfaces onto retrieving events from Xt (the X toolkit) and from TTY's
3436 (using @code{read()} and @code{select()}), respectively. The event
3437 interface enforces a clean separation between the specific code for
3438 interfacing with the operating system and the generic code for working
3439 with events, by defining an API of basic, low-level event methods;
3440 @file{event-Xt.c} and @file{event-tty.c} are two different
3441 implementations of this API. To add support for a new operating system
3442 (e.g. NeXTstep), one merely needs to provide another implementation of
3443 those API functions.
3445 Note that the choice of whether to use @file{event-Xt.c} or
3446 @file{event-tty.c} is made at compile time! Or at the very latest, it
3447 is made at startup time. @file{event-Xt.c} handles events for
3448 @emph{both} X and TTY frames; @file{event-tty.c} is only used when X
3449 support is not compiled into XEmacs. The reason for this is that there
3450 is only one event loop in XEmacs: thus, it needs to be able to receive
3451 events from all different kinds of frames.
3460 @file{keymap.c} and @file{keymap.h} define the @dfn{keymap} Lisp object
3461 type and associated methods and primitives. (Remember that keymaps are
3462 objects that associate event descriptions with functions to be called to
3463 ``execute'' those events; @code{dispatch-event} looks up events in the
3472 @file{keyboard.c} contains functions that implement the actual editor
3473 command loop---i.e. the event loop that cyclically retrieves and
3474 dispatches events. This code is also rather tricky, just like
3475 @file{event-stream.c}.
3484 These two modules contain the basic code for defining keyboard macros.
3485 These functions don't actually do much; most of the code that handles keyboard
3486 macros is mixed in with the event-handling code in @file{event-stream.c}.
3494 This contains some miscellaneous code related to the minibuffer (most of
3495 the minibuffer code was moved into Lisp by Richard Mlynarik). This
3496 includes the primitives for completion (although filename completion is
3497 in @file{dired.c}), the lowest-level interface to the minibuffer (if the
3498 command loop were cleaned up, this too could be in Lisp), and code for
3499 dealing with the echo area (this, too, was mostly moved into Lisp, and
3500 the only code remaining is code to call out to Lisp or provide simple
3501 bootstrapping implementations early in temacs, before the echo-area Lisp
3506 @node Modules for the Basic Displayable Lisp Objects, Modules for other Display-Related Lisp Objects, Editor-Level Control Flow Modules, A Summary of the Various XEmacs Modules
3507 @section Modules for the Basic Displayable Lisp Objects
3521 These modules implement the @dfn{device} Lisp object type. This
3522 abstracts a particular screen or connection on which frames are
3523 displayed. As with Lisp objects, event interfaces, and other
3524 subsystems, the device code is separated into a generic component that
3525 contains a standardized interface (in the form of a set of methods) onto
3526 particular device types.
3528 The device subsystem defines all the methods and provides method
3529 services for not only device operations but also for the frame, window,
3530 menubar, scrollbar, toolbar, and other displayable-object subsystems.
3531 The reason for this is that all of these subsystems have the same
3532 subtypes (X, TTY, NeXTstep, Microsoft Windows, etc.) as devices do.
3545 Each device contains one or more frames in which objects (e.g. text) are
3546 displayed. A frame corresponds to a window in the window system;
3547 usually this is a top-level window but it could potentially be one of a
3548 number of overlapping child windows within a top-level window, using the
3549 MDI (Multiple Document Interface) protocol in Microsoft Windows or a
3552 The @file{frame-*} files implement the @dfn{frame} Lisp object type and
3553 provide the generic and device-type-specific operations on frames
3554 (e.g. raising, lowering, resizing, moving, etc.).
3563 @cindex window (in Emacs)
3565 Each frame consists of one or more non-overlapping @dfn{windows} (better
3566 known as @dfn{panes} in standard window-system terminology) in which a
3567 buffer's text can be displayed. Windows can also have scrollbars
3568 displayed around their edges.
3570 @file{window.c} and @file{window.h} implement the @dfn{window} Lisp
3571 object type and provide code to manage windows. Since windows have no
3572 associated resources in the window system (the window system knows only
3573 about the frame; no child windows or anything are used for XEmacs
3574 windows), there is no device-type-specific code here; all of that code
3575 is part of the redisplay mechanism or the code for particular object
3576 types such as scrollbars.
3580 @node Modules for other Display-Related Lisp Objects, Modules for the Redisplay Mechanism, Modules for the Basic Displayable Lisp Objects, A Summary of the Various XEmacs Modules
3581 @section Modules for other Display-Related Lisp Objects
3641 This file provides C support for syntax highlighting---i.e.
3642 highlighting different syntactic constructs of a source file in
3643 different colors, for easy reading. The C support is provided so that
3655 These modules decode GIF-format image files, for use with glyphs.
3659 @node Modules for the Redisplay Mechanism, Modules for Interfacing with the File System, Modules for other Display-Related Lisp Objects, A Summary of the Various XEmacs Modules
3660 @section Modules for the Redisplay Mechanism
3670 These files provide the redisplay mechanism. As with many other
3671 subsystems in XEmacs, there is a clean separation between the general
3672 and device-specific support.
3674 @file{redisplay.c} contains the bulk of the redisplay engine. These
3675 functions update the redisplay structures (which describe how the screen
3676 is to appear) to reflect any changes made to the state of any
3677 displayable objects (buffer, frame, window, etc.) since the last time
3678 that redisplay was called. These functions are highly optimized to
3679 avoid doing more work than necessary (since redisplay is called
3680 extremely often and is potentially a huge time sink), and depend heavily
3681 on notifications from the objects themselves that changes have occurred,
3682 so that redisplay doesn't explicitly have to check each possible object.
3683 The redisplay mechanism also contains a great deal of caching to further
3684 speed things up; some of this caching is contained within the various
3685 displayable objects.
3687 @file{redisplay-output.c} goes through the redisplay structures and converts
3688 them into calls to device-specific methods to actually output the screen
3691 @file{redisplay-x.c} and @file{redisplay-tty.c} are two implementations
3692 of these redisplay output methods, for X frames and TTY frames,
3701 This module contains various functions and Lisp primitives for
3702 converting between buffer positions and screen positions. These
3703 functions call the redisplay mechanism to do most of the work, and then
3704 examine the redisplay structures to get the necessary information. This
3715 These files contain functions for working with the termcap (BSD-style)
3716 and terminfo (System V style) databases of terminal capabilities and
3717 escape sequences, used when XEmacs is displaying in a TTY.
3726 These files provide some miscellaneous TTY-output functions and should
3727 probably be merged into @file{redisplay-tty.c}.
3731 @node Modules for Interfacing with the File System, Modules for Other Aspects of the Lisp Interpreter and Object System, Modules for the Redisplay Mechanism, A Summary of the Various XEmacs Modules
3732 @section Modules for Interfacing with the File System
3739 These modules implement the @dfn{stream} Lisp object type. This is an
3740 internal-only Lisp object that implements a generic buffering stream.
3741 The idea is to provide a uniform interface onto all sources and sinks of
3742 data, including file descriptors, stdio streams, chunks of memory, Lisp
3743 buffers, Lisp strings, etc. That way, I/O functions can be written to
3744 the stream interface and can transparently handle all possible sources
3745 and sinks. (For example, the @code{read} function can read data from a
3746 file, a string, a buffer, or even a function that is called repeatedly
3747 to return data, without worrying about where the data is coming from or
3748 what-size chunks it is returned in.)
3751 Note that in the C code, streams are called @dfn{lstreams} (for ``Lisp
3752 streams'') to distinguish them from other kinds of streams, e.g. stdio
3753 streams and C++ I/O streams.
3755 Similar to other subsystems in XEmacs, lstreams are separated into
3756 generic functions and a set of methods for the different types of
3757 lstreams. @file{lstream.c} provides implementations of many different
3758 types of streams; others are provided, e.g., in @file{mule-coding.c}.
3766 This implements the basic primitives for interfacing with the file
3767 system. This includes primitives for reading files into buffers,
3768 writing buffers into files, checking for the presence or accessibility
3769 of files, canonicalizing file names, etc. Note that these primitives
3770 are usually not invoked directly by the user: There is a great deal of
3771 higher-level Lisp code that implements the user commands such as
3772 @code{find-file} and @code{save-buffer}. This is similar to the
3773 distinction between the lower-level primitives in @file{editfns.c} and
3774 the higher-level user commands in @file{commands.c} and
3783 This file provides functions for detecting clashes between different
3784 processes (e.g. XEmacs and some external process, or two different
3785 XEmacs processes) modifying the same file. (XEmacs can optionally use
3786 the @file{lock/} subdirectory to provide a form of ``locking'' between
3787 different XEmacs processes.) This module is also used by the low-level
3788 functions in @file{insdel.c} to ensure that, if the first modification
3789 is being made to a buffer whose corresponding file has been externally
3790 modified, the user is made aware of this so that the buffer can be
3791 synched up with the external changes if necessary.
3798 This file provides some miscellaneous functions that construct a
3799 @samp{rwxr-xr-x}-type permissions string (as might appear in an
3800 @file{ls}-style directory listing) given the information returned by the
3801 @code{stat()} system call.
3810 These files implement the XEmacs interface to directory searching. This
3811 includes a number of primitives for determining the files in a directory
3812 and for doing filename completion. (Remember that generic completion is
3813 handled by a different mechanism, in @file{minibuf.c}.)
3815 @file{ndir.h} is a header file used for the directory-searching
3816 emulation functions provided in @file{sysdep.c} (see section J below),
3817 for systems that don't provide any directory-searching functions. (On
3818 those systems, directories can be read directly as files, and parsed.)
3826 This file provides an implementation of the @code{realpath()} function
3827 for expanding symbolic links, on systems that don't implement it or have
3828 a broken implementation.
3832 @node Modules for Other Aspects of the Lisp Interpreter and Object System, Modules for Interfacing with the Operating System, Modules for Interfacing with the File System, A Summary of the Various XEmacs Modules
3833 @section Modules for Other Aspects of the Lisp Interpreter and Object System
3842 These files provide two implementations of hash tables. Files
3843 @file{hash.c} and @file{hash.h} provide a generic C implementation of
3844 hash tables which can stand independently of XEmacs. Files
3845 @file{elhash.c} and @file{elhash.h} provide a separate implementation of
3846 hash tables that can store only Lisp objects, and knows about Lispy
3847 things like garbage collection, and implement the @dfn{hash-table} Lisp
3856 This module implements the @dfn{specifier} Lisp object type. This is
3857 primarily used for displayable properties, and allows for values that
3858 are specific to a particular buffer, window, frame, device, or device
3859 class, as well as a default value existing. This is used, for example,
3860 to control the height of the horizontal scrollbar or the appearance of
3861 the @code{default}, @code{bold}, or other faces. The specifier object
3862 consists of a number of specifications, each of which maps from a
3863 buffer, window, etc. to a value. The function @code{specifier-instance}
3864 looks up a value given a window (from which a buffer, frame, and device
3874 @file{chartab.c} and @file{chartab.h} implement the @dfn{char table}
3875 Lisp object type, which maps from characters or certain sorts of
3876 character ranges to Lisp objects. The implementation of this object
3877 type is optimized for the internal representation of characters. Char
3878 tables come in different types, which affect the allowed object types to
3879 which a character can be mapped and also dictate certain other
3880 properties of the char table.
3883 @file{casetab.c} implements one sort of char table, the @dfn{case
3884 table}, which maps characters to other characters of possibly different
3885 case. These are used by XEmacs to implement case-changing primitives
3886 and to do case-insensitive searching.
3896 This module implements @dfn{syntax tables}, another sort of char table
3897 that maps characters into syntax classes that define the syntax of these
3898 characters (e.g. a parenthesis belongs to a class of @samp{open}
3899 characters that have corresponding @samp{close} characters and can be
3900 nested). This module also implements the Lisp @dfn{scanner}, a set of
3901 primitives for scanning over text based on syntax tables. This is used,
3902 for example, to find the matching parenthesis in a command such as
3903 @code{forward-sexp}, and by @file{font-lock.c} to locate quoted strings,
3912 This module implements various Lisp primitives for upcasing, downcasing
3913 and capitalizing strings or regions of buffers.
3921 This module implements the @dfn{range table} Lisp object type, which
3922 provides for a mapping from ranges of integers to arbitrary Lisp
3932 This module implements the @dfn{opaque} Lisp object type, an
3933 internal-only Lisp object that encapsulates an arbitrary block of memory
3934 so that it can be managed by the Lisp allocation system. To create an
3935 opaque object, you call @code{make_opaque()}, passing a pointer to a
3936 block of memory. An object is created that is big enough to hold the
3937 memory, which is copied into the object's storage. The object will then
3938 stick around as long as you keep pointers to it, after which it will be
3939 automatically reclaimed.
3942 Opaque objects can also have an arbitrary @dfn{mark method} associated
3943 with them, in case the block of memory contains other Lisp objects that
3944 need to be marked for garbage-collection purposes. (If you need other
3945 object methods, such as a finalize method, you should just go ahead and
3946 create a new Lisp object type---it's not hard.)
3954 This function provides a few primitives for doing dynamic abbreviation
3955 expansion. In XEmacs, most of the code for this has been moved into
3956 Lisp. Some C code remains for speed and because the primitive
3957 @code{self-insert-command} (which is executed for all self-inserting
3958 characters) hooks into the abbrev mechanism. (@code{self-insert-command}
3959 is itself in C only for speed.)
3967 This function provides primitives for retrieving the documentation
3968 strings of functions and variables. These documentation strings contain
3969 certain special markers that get dynamically expanded (e.g. a
3970 reverse-lookup is performed on some named functions to retrieve their
3971 current key bindings). Some documentation strings (in particular, for
3972 the built-in primitives and pre-loaded Lisp functions) are stored
3973 externally in a file @file{DOC} in the @file{lib-src/} directory and
3974 need to be fetched from that file. (Part of the build stage involves
3975 building this file, and another part involves constructing an index for
3976 this file and embedding it into the executable, so that the functions in
3977 @file{doc.c} do not have to search the entire @file{DOC} file to find
3978 the appropriate documentation string.)
3986 This function provides a Lisp primitive that implements the MD5 secure
3987 hashing scheme, used to create a large hash value of a string of data such that
3988 the data cannot be derived from the hash value. This is used for
3989 various security applications on the Internet.
3994 @node Modules for Interfacing with the Operating System, Modules for Interfacing with X Windows, Modules for Other Aspects of the Lisp Interpreter and Object System, A Summary of the Various XEmacs Modules
3995 @section Modules for Interfacing with the Operating System
4003 These modules allow XEmacs to spawn and communicate with subprocesses
4004 and network connections.
4006 @cindex synchronous subprocesses
4007 @cindex subprocesses, synchronous
4008 @file{callproc.c} implements (through the @code{call-process}
4009 primitive) what are called @dfn{synchronous subprocesses}. This means
4010 that XEmacs runs a program, waits till it's done, and retrieves its
4011 output. A typical example might be calling the @file{ls} program to get
4012 a directory listing.
4014 @cindex asynchronous subprocesses
4015 @cindex subprocesses, asynchronous
4016 @file{process.c} and @file{process.h} implement @dfn{asynchronous
4017 subprocesses}. This means that XEmacs starts a program and then
4018 continues normally, not waiting for the process to finish. Data can be
4019 sent to the process or retrieved from it as it's running. This is used
4020 for the @code{shell} command (which provides a front end onto a shell
4021 program such as @file{csh}), the mail and news readers implemented in
4022 XEmacs, etc. The result of calling @code{start-process} to start a
4023 subprocess is a process object, a particular kind of object used to
4024 communicate with the subprocess. You can send data to the process by
4025 passing the process object and the data to @code{send-process}, and you
4026 can specify what happens to data retrieved from the process by setting
4027 properties of the process object. (When the process sends data, XEmacs
4028 receives a process event, which says that there is data ready. When
4029 @code{dispatch-event} is called on this event, it reads the data from
4030 the process and does something with it, as specified by the process
4031 object's properties. Typically, this means inserting the data into a
4032 buffer or calling a function.) Another property of the process object is
4033 called the @dfn{sentinel}, which is a function that is called when the
4036 @cindex network connections
4037 Process objects are also used for network connections (connections to a
4038 process running on another machine). Network connections are started
4039 with @code{open-network-stream} but otherwise work just like
4049 These modules implement most of the low-level, messy operating-system
4050 interface code. This includes various device control (ioctl) operations
4051 for file descriptors, TTY's, pseudo-terminals, etc. (usually this stuff
4052 is fairly system-dependent; thus the name of this module), and emulation
4053 of standard library functions and system calls on systems that don't
4054 provide them or have broken versions.
4070 These header files provide consistent interfaces onto system-dependent
4071 header files and system calls. The idea is that, instead of including a
4072 standard header file like @file{<sys/param.h>} (which may or may not
4073 exist on various systems) or having to worry about whether all system
4074 provide a particular preprocessor constant, or having to deal with the
4075 four different paradigms for manipulating signals, you just include the
4076 appropriate @file{sys*.h} header file, which includes all the right
4077 system header files, defines and missing preprocessor constants,
4078 provides a uniform interface onto system calls, etc.
4080 @file{sysdir.h} provides a uniform interface onto directory-querying
4081 functions. (In some cases, this is in conjunction with emulation
4082 functions in @file{sysdep.c}.)
4084 @file{sysfile.h} includes all the necessary header files for standard
4085 system calls (e.g. @code{read()}), ensures that all necessary
4086 @code{open()} and @code{stat()} preprocessor constants are defined, and
4087 possibly (usually) substitutes sugared versions of @code{read()},
4088 @code{write()}, etc. that automatically restart interrupted I/O
4091 @file{sysfloat.h} includes the necessary header files for floating-point
4094 @file{sysproc.h} includes the necessary header files for calling
4095 @code{select()}, @code{fork()}, @code{execve()}, socket operations, and
4096 the like, and ensures that the @code{FD_*()} macros for descriptor-set
4097 manipulations are available.
4099 @file{syspwd.h} includes the necessary header files for obtaining
4100 information from @file{/etc/passwd} (the functions are emulated under
4103 @file{syssignal.h} includes the necessary header files for
4104 signal-handling and provides a uniform interface onto the different
4105 signal-handling and signal-blocking paradigms.
4107 @file{systime.h} includes the necessary header files and provides
4108 uniform interfaces for retrieving the time of day, setting file
4109 access/modification times, getting the amount of time used by the XEmacs
4112 @file{systty.h} buffers against the infinitude of different ways of
4115 @file{syswait.h} provides a uniform way of retrieving the exit status
4116 from a @code{wait()}ed-on process (some systems use a union, others use
4133 These files implement the ability to play various sounds on some types
4134 of computers. You have to configure your XEmacs with sound support in
4135 order to get this capability.
4137 @file{sound.c} provides the generic interface. It implements various
4138 Lisp primitives and variables that let you specify which sounds should
4139 be played in certain conditions. (The conditions are identified by
4140 symbols, which are passed to @code{ding} to make a sound. Various
4141 standard functions call this function at certain times; if sound support
4142 does not exist, a simple beep results.
4144 @cindex native sound
4145 @cindex sound, native
4146 @file{sgiplay.c}, @file{sunplay.c}, @file{hpplay.c}, and
4147 @file{linuxplay.c} interface to the machine's speaker for various
4148 different kind of machines. This is called @dfn{native} sound.
4150 @cindex sound, network
4151 @cindex network sound
4153 @file{nas.c} interfaces to a computer somewhere else on the network
4154 using the NAS (Network Audio Server) protocol, playing sounds on that
4155 machine. This allows you to run XEmacs on a remote machine, with its
4156 display set to your local machine, and have the sounds be made on your
4157 local machine, provided that you have a NAS server running on your local
4160 @file{libsst.c}, @file{libsst.h}, and @file{libst.h} provide some
4161 additional functions for playing sound on a Sun SPARC but are not
4171 These two modules implement an interface to the ToolTalk protocol, which
4172 is an interprocess communication protocol implemented on some versions
4173 of Unix. ToolTalk is a high-level protocol that allows processes to
4174 register themselves as providers of particular services; other processes
4175 can then request a service without knowing or caring exactly who is
4176 providing the service. It is similar in spirit to the DDE protocol
4177 provided under Microsoft Windows. ToolTalk is a part of the new CDE
4178 (Common Desktop Environment) specification and is used to connect the
4179 parts of the SPARCWorks development environment.
4187 This module provides the ability to retrieve the system's current load
4188 average. (The way to do this is highly system-specific, unfortunately,
4189 and requires a lot of special-case code.)
4197 This module provides a small amount of code used internally at Sun to
4198 keep statistics on the usage of XEmacs.
4209 These files provide replacement functions and prototypes to fix numerous
4210 bugs in early releases of SunOS 4.1.
4218 This module provides some terminal-control code necessary on versions of
4228 These modules are used for MS-DOS support, which does not work in
4233 @node Modules for Interfacing with X Windows, Modules for Internationalization, Modules for Interfacing with the Operating System, A Summary of the Various XEmacs Modules
4234 @section Modules for Interfacing with X Windows
4240 A file generated from @file{Emacs.ad}, which contains XEmacs-supplied
4241 fallback resources (so that XEmacs has pretty defaults).
4251 These modules implement an Xt widget class that encapsulates a frame.
4252 This is for ease in integrating with Xt. The EmacsFrame widget covers
4253 the entire X window except for the menubar; the scrollbars are
4254 positioned on top of the EmacsFrame widget.
4256 @strong{Warning:} Abandon hope, all ye who enter here. This code took
4257 an ungodly amount of time to get right, and is likely to fall apart
4258 mercilessly at the slightest change. Such is life under Xt.
4268 These modules implement a simple Xt manager (i.e. composite) widget
4269 class that simply lets its children set whatever geometry they want.
4270 It's amazing that Xt doesn't provide this standardly, but on second
4271 thought, it makes sense, considering how amazingly broken Xt is.
4281 These modules implement two Xt widget classes that are subclasses of
4282 the TopLevelShell and TransientShell classes. This is necessary to deal
4283 with more brokenness that Xt has sadistically thrust onto the backs of
4293 These modules provide functions for maintenance and caching of GC's
4294 (graphics contexts) under the X Window System. This code is junky and
4295 needs to be rewritten.
4304 This module provides an interface to the X Window System's concept of
4305 @dfn{selections}, the standard way for X applications to communicate
4317 These header files are similar in spirit to the @file{sys*.h} files and buffer
4318 against different implementations of Xt and Motif.
4322 @file{xintrinsic.h} should be included in place of @file{<Intrinsic.h>}.
4324 @file{xintrinsicp.h} should be included in place of @file{<IntrinsicP.h>}.
4326 @file{xmmanagerp.h} should be included in place of @file{<XmManagerP.h>}.
4328 @file{xmprimitivep.h} should be included in place of @file{<XmPrimitiveP.h>}.
4338 These files provide an emulation of the Xmu library for those systems
4339 (i.e. HPUX) that don't provide it as a standard part of X.
4344 ExternalClient-Xlib.c
4357 @cindex external widget
4358 These files provide the @dfn{external widget} interface, which allows an
4359 XEmacs frame to appear as a widget in another application. To do this,
4360 you have to configure with @samp{--external-widget}.
4362 @file{ExternalShell*} provides the server (XEmacs) side of the
4365 @file{ExternalClient*} provides the client (other application) side of
4366 the connection. These files are not compiled into XEmacs but are
4367 compiled into libraries that are then linked into your application.
4369 @file{extw-*} is common code that is used for both the client and server.
4371 Don't touch this code; something is liable to break if you do.
4375 @node Modules for Internationalization, , Modules for Interfacing with X Windows, A Summary of the Various XEmacs Modules
4376 @section Modules for Internationalization
4391 These files implement the MULE (Asian-language) support. Note that MULE
4392 actually provides a general interface for all sorts of languages, not
4393 just Asian languages (although they are generally the most complicated
4394 to support). This code is still in beta.
4396 @file{mule-charset.*} and @file{mule-coding.*} provide the heart of the
4397 XEmacs MULE support. @file{mule-charset.*} implements the @dfn{charset}
4398 Lisp object type, which encapsulates a character set (an ordered one- or
4399 two-dimensional set of characters, such as US ASCII or JISX0208 Japanese
4402 @file{mule-coding.*} implements the @dfn{coding-system} Lisp object
4403 type, which encapsulates a method of converting between different
4404 encodings. An encoding is a representation of a stream of characters,
4405 possibly from multiple character sets, using a stream of bytes or words,
4406 and defines (e.g.) which escape sequences are used to specify particular
4407 character sets, how the indices for a character are converted into bytes
4408 (sometimes this involves setting the high bit; sometimes complicated
4409 rearranging of the values takes place, as in the Shift-JIS encoding),
4412 @file{mule-ccl.c} provides the CCL (Code Conversion Language)
4413 interpreter. CCL is similar in spirit to Lisp byte code and is used to
4414 implement converters for custom encodings.
4416 @file{mule-canna.c} and @file{mule-wnnfns.c} implement interfaces to
4417 external programs used to implement the Canna and WNN input methods,
4418 respectively. This is currently in beta.
4420 @file{mule-mcpath.c} provides some functions to allow for pathnames
4421 containing extended characters. This code is fragmentary, obsolete, and
4422 completely non-working. Instead, @var{pathname-coding-system} is used
4423 to specify conversions of names of files and directories. The standard
4424 C I/O functions like @samp{open()} are wrapped so that conversion occurs
4427 @file{mule.c} provides a few miscellaneous things that should probably
4436 This provides some miscellaneous internationalization code for
4437 implementing message translation and interfacing to the Ximp input
4438 method. None of this code is currently working.
4446 This contains leftover code from an earlier implementation of
4447 Asian-language support, and is not currently used.
4452 @node Allocation of Objects in XEmacs Lisp, Dumping, A Summary of the Various XEmacs Modules, Top
4453 @chapter Allocation of Objects in XEmacs Lisp
4456 * Introduction to Allocation::
4457 * Garbage Collection::
4459 * Garbage Collection - Step by Step::
4460 * Integers and Characters::
4461 * Allocation from Frob Blocks::
4463 * Low-level allocation::
4470 * Compiled Function::
4473 @node Introduction to Allocation, Garbage Collection, Allocation of Objects in XEmacs Lisp, Allocation of Objects in XEmacs Lisp
4474 @section Introduction to Allocation
4476 Emacs Lisp, like all Lisps, has garbage collection. This means that
4477 the programmer never has to explicitly free (destroy) an object; it
4478 happens automatically when the object becomes inaccessible. Most
4479 experts agree that garbage collection is a necessity in a modern,
4480 high-level language. Its omission from C stems from the fact that C was
4481 originally designed to be a nice abstract layer on top of assembly
4482 language, for writing kernels and basic system utilities rather than
4485 Lisp objects can be created by any of a number of Lisp primitives.
4486 Most object types have one or a small number of basic primitives
4487 for creating objects. For conses, the basic primitive is @code{cons};
4488 for vectors, the primitives are @code{make-vector} and @code{vector}; for
4489 symbols, the primitives are @code{make-symbol} and @code{intern}; etc.
4490 Some Lisp objects, especially those that are primarily used internally,
4491 have no corresponding Lisp primitives. Every Lisp object, though,
4492 has at least one C primitive for creating it.
4494 Recall from section (VII) that a Lisp object, as stored in a 32-bit or
4495 64-bit word, has a few tag bits, and a ``value'' that occupies the
4496 remainder of the bits. We can separate the different Lisp object types
4497 into three broad categories:
4501 (a) Those for whom the value directly represents the contents of the
4502 Lisp object. Only two types are in this category: integers and
4503 characters. No special allocation or garbage collection is necessary
4504 for such objects. Lisp objects of these types do not need to be
4508 In the remaining two categories, the type is stored in the object
4509 itself. The tag for all such objects is the generic @dfn{lrecord}
4510 (Lisp_Type_Record) tag. The first bytes of the object's structure are an
4511 integer (actually a char) characterising the object's type and some
4512 flags, in particular the mark bit used for garbage collection. A
4513 structure describing the type is accessible thru the
4514 lrecord_implementation_table indexed with said integer. This structure
4515 includes the method pointers and a pointer to a string naming the type.
4519 (b) Those lrecords that are allocated in frob blocks (see above). This
4520 includes the objects that are most common and relatively small, and
4521 includes conses, strings, subrs, floats, compiled functions, symbols,
4522 extents, events, and markers. With the cleanup of frob blocks done in
4523 19.12, it's not terribly hard to add more objects to this category, but
4524 it's a bit trickier than adding an object type to type (c) (esp. if the
4525 object needs a finalization method), and is not likely to save much
4526 space unless the object is small and there are many of them. (In fact,
4527 if there are very few of them, it might actually waste space.)
4529 (c) Those lrecords that are individually @code{malloc()}ed. These are
4530 called @dfn{lcrecords}. All other types are in this category. Adding a
4531 new type to this category is comparatively easy, and all types added
4532 since 19.8 (when the current allocation scheme was devised, by Richard
4533 Mlynarik), with the exception of the character type, have been in this
4537 Note that bit vectors are a bit of a special case. They are
4538 simple lrecords as in category (b), but are individually @code{malloc()}ed
4539 like vectors. You can basically view them as exactly like vectors
4540 except that their type is stored in lrecord fashion rather than
4541 in directly-tagged fashion.
4544 @node Garbage Collection, GCPROing, Introduction to Allocation, Allocation of Objects in XEmacs Lisp
4545 @section Garbage Collection
4546 @cindex garbage collection
4548 @cindex mark and sweep
4549 Garbage collection is simple in theory but tricky to implement.
4550 Emacs Lisp uses the oldest garbage collection method, called
4551 @dfn{mark and sweep}. Garbage collection begins by starting with
4552 all accessible locations (i.e. all variables and other slots where
4553 Lisp objects might occur) and recursively traversing all objects
4554 accessible from those slots, marking each one that is found.
4555 We then go through all of memory and free each object that is
4556 not marked, and unmarking each object that is marked. Note
4557 that ``all of memory'' means all currently allocated objects.
4558 Traversing all these objects means traversing all frob blocks,
4559 all vectors (which are chained in one big list), and all
4560 lcrecords (which are likewise chained).
4562 Garbage collection can be invoked explicitly by calling
4563 @code{garbage-collect} but is also called automatically by @code{eval},
4564 once a certain amount of memory has been allocated since the last
4565 garbage collection (according to @code{gc-cons-threshold}).
4568 @node GCPROing, Garbage Collection - Step by Step, Garbage Collection, Allocation of Objects in XEmacs Lisp
4569 @section @code{GCPRO}ing
4571 @code{GCPRO}ing is one of the ugliest and trickiest parts of Emacs
4572 internals. The basic idea is that whenever garbage collection
4573 occurs, all in-use objects must be reachable somehow or
4574 other from one of the roots of accessibility. The roots
4575 of accessibility are:
4579 All objects that have been @code{staticpro()}d or
4580 @code{staticpro_nodump()}ed. This is used for any global C variables
4581 that hold Lisp objects. A call to @code{staticpro()} happens implicitly
4582 as a result of any symbols declared with @code{defsymbol()} and any
4583 variables declared with @code{DEFVAR_FOO()}. You need to explicitly
4584 call @code{staticpro()} (in the @code{vars_of_foo()} method of a module)
4585 for other global C variables holding Lisp objects. (This typically
4586 includes internal lists and such things.). Use
4587 @code{staticpro_nodump()} only in the rare cases when you do not want
4588 the pointed variable to be saved at dump time but rather recompute it at
4591 Note that @code{obarray} is one of the @code{staticpro()}d things.
4592 Therefore, all functions and variables get marked through this.
4594 Any shadowed bindings that are sitting on the @code{specpdl} stack.
4596 Any objects sitting in currently active (Lisp) stack frames,
4597 catches, and condition cases.
4599 A couple of special-case places where active objects are
4602 Anything currently marked with @code{GCPRO}.
4605 Marking with @code{GCPRO} is necessary because some C functions (quite
4606 a lot, in fact), allocate objects during their operation. Quite
4607 frequently, there will be no other pointer to the object while the
4608 function is running, and if a garbage collection occurs and the object
4609 needs to be referenced again, bad things will happen. The solution is
4610 to mark those objects with @code{GCPRO}. Unfortunately this is easy to
4611 forget, and there is basically no way around this problem. Here are
4616 For every @code{GCPRO@var{n}}, there have to be declarations of
4617 @code{struct gcpro gcpro1, gcpro2}, etc.
4620 You @emph{must} @code{UNGCPRO} anything that's @code{GCPRO}ed, and you
4621 @emph{must not} @code{UNGCPRO} if you haven't @code{GCPRO}ed. Getting
4622 either of these wrong will lead to crashes, often in completely random
4623 places unrelated to where the problem lies.
4626 The way this actually works is that all currently active @code{GCPRO}s
4627 are chained through the @code{struct gcpro} local variables, with the
4628 variable @samp{gcprolist} pointing to the head of the list and the nth
4629 local @code{gcpro} variable pointing to the first @code{gcpro} variable
4630 in the next enclosing stack frame. Each @code{GCPRO}ed thing is an
4631 lvalue, and the @code{struct gcpro} local variable contains a pointer to
4632 this lvalue. This is why things will mess up badly if you don't pair up
4633 the @code{GCPRO}s and @code{UNGCPRO}s---you will end up with
4634 @code{gcprolist}s containing pointers to @code{struct gcpro}s or local
4635 @code{Lisp_Object} variables in no-longer-active stack frames.
4638 It is actually possible for a single @code{struct gcpro} to
4639 protect a contiguous array of any number of values, rather than
4640 just a single lvalue. To effect this, call @code{GCPRO@var{n}} as usual on
4641 the first object in the array and then set @code{gcpro@var{n}.nvars}.
4644 @strong{Strings are relocated.} What this means in practice is that the
4645 pointer obtained using @code{XSTRING_DATA()} is liable to change at any
4646 time, and you should never keep it around past any function call, or
4647 pass it as an argument to any function that might cause a garbage
4648 collection. This is why a number of functions accept either a
4649 ``non-relocatable'' @code{char *} pointer or a relocatable Lisp string,
4650 and only access the Lisp string's data at the very last minute. In some
4651 cases, you may end up having to @code{alloca()} some space and copy the
4652 string's data into it.
4655 By convention, if you have to nest @code{GCPRO}'s, use @code{NGCPRO@var{n}}
4656 (along with @code{struct gcpro ngcpro1, ngcpro2}, etc.), @code{NNGCPRO@var{n}},
4657 etc. This avoids compiler warnings about shadowed locals.
4660 It is @emph{always} better to err on the side of extra @code{GCPRO}s
4661 rather than too few. The extra cycles spent on this are
4662 almost never going to make a whit of difference in the
4666 The general rule to follow is that caller, not callee, @code{GCPRO}s.
4667 That is, you should not have to explicitly @code{GCPRO} any Lisp objects
4668 that are passed in as parameters.
4670 One exception from this rule is if you ever plan to change the parameter
4671 value, and store a new object in it. In that case, you @emph{must}
4672 @code{GCPRO} the parameter, because otherwise the new object will not be
4675 So, if you create any Lisp objects (remember, this happens in all sorts
4676 of circumstances, e.g. with @code{Fcons()}, etc.), you are responsible
4677 for @code{GCPRO}ing them, unless you are @emph{absolutely sure} that
4678 there's no possibility that a garbage-collection can occur while you
4679 need to use the object. Even then, consider @code{GCPRO}ing.
4682 A garbage collection can occur whenever anything calls @code{Feval}, or
4683 whenever a QUIT can occur where execution can continue past
4684 this. (Remember, this is almost anywhere.)
4687 If you have the @emph{least smidgeon of doubt} about whether
4688 you need to @code{GCPRO}, you should @code{GCPRO}.
4691 Beware of @code{GCPRO}ing something that is uninitialized. If you have
4692 any shade of doubt about this, initialize all your variables to @code{Qnil}.
4695 Be careful of traps, like calling @code{Fcons()} in the argument to
4696 another function. By the ``caller protects'' law, you should be
4697 @code{GCPRO}ing the newly-created cons, but you aren't. A certain
4698 number of functions that are commonly called on freshly created stuff
4699 (e.g. @code{nconc2()}, @code{Fsignal()}), break the ``caller protects''
4700 law and go ahead and @code{GCPRO} their arguments so as to simplify
4701 things, but make sure and check if it's OK whenever doing something like
4705 Once again, remember to @code{GCPRO}! Bugs resulting from insufficient
4706 @code{GCPRO}ing are intermittent and extremely difficult to track down,
4707 often showing up in crashes inside of @code{garbage-collect} or in
4708 weirdly corrupted objects or even in incorrect values in a totally
4709 different section of code.
4712 @cindex garbage collection, conservative
4713 @cindex conservative garbage collection
4714 Given the extremely error-prone nature of the @code{GCPRO} scheme, and
4715 the difficulties in tracking down, it should be considered a deficiency
4716 in the XEmacs code. A solution to this problem would involve
4717 implementing so-called @dfn{conservative} garbage collection for the C
4718 stack. That involves looking through all of stack memory and treating
4719 anything that looks like a reference to an object as a reference. This
4720 will result in a few objects not getting collected when they should, but
4721 it obviates the need for @code{GCPRO}ing, and allows garbage collection
4722 to happen at any point at all, such as during object allocation.
4724 @node Garbage Collection - Step by Step, Integers and Characters, GCPROing, Allocation of Objects in XEmacs Lisp
4725 @section Garbage Collection - Step by Step
4726 @cindex garbage collection step by step
4730 * garbage_collect_1::
4733 * sweep_lcrecords_1::
4734 * compact_string_chars::
4736 * sweep_bit_vectors_1::
4739 @node Invocation, garbage_collect_1, Garbage Collection - Step by Step, Garbage Collection - Step by Step
4740 @subsection Invocation
4741 @cindex garbage collection, invocation
4743 The first thing that anyone should know about garbage collection is:
4744 when and how the garbage collector is invoked. One might think that this
4745 could happen every time new memory is allocated, e.g. new objects are
4746 created, but this is @emph{not} the case. Instead, we have the following
4749 The entry point of any process of garbage collection is an invocation
4750 of the function @code{garbage_collect_1} in file @code{alloc.c}. The
4751 invocation can occur @emph{explicitly} by calling the function
4752 @code{Fgarbage_collect} (in addition this function provides information
4753 about the freed memory), or can occur @emph{implicitly} in four different
4757 In function @code{main_1} in file @code{emacs.c}. This function is called
4758 at each startup of xemacs. The garbage collection is invoked after all
4759 initial creations are completed, but only if a special internal error
4760 checking-constant @code{ERROR_CHECK_GC} is defined.
4762 In function @code{disksave_object_finalization} in file
4763 @code{alloc.c}. The only purpose of this function is to clear the
4764 objects from memory which need not be stored with xemacs when we dump out
4765 an executable. This is only done by @code{Fdump_emacs} or by
4766 @code{Fdump_emacs_data} respectively (both in @code{emacs.c}). The
4767 actual clearing is accomplished by making these objects unreachable and
4768 starting a garbage collection. The function is only used while building
4771 In function @code{Feval / eval} in file @code{eval.c}. Each time the
4772 well known and often used function eval is called to evaluate a form,
4773 one of the first things that could happen, is a potential call of
4774 @code{garbage_collect_1}. There exist three global variables,
4775 @code{consing_since_gc} (counts the created cons-cells since the last
4776 garbage collection), @code{gc_cons_threshold} (a specified threshold
4777 after which a garbage collection occurs) and @code{always_gc}. If
4778 @code{always_gc} is set or if the threshold is exceeded, the garbage
4779 collection will start.
4781 In function @code{Ffuncall / funcall} in file @code{eval.c}. This
4782 function evaluates calls of elisp functions and works according to
4786 The upshot is that garbage collection can basically occur everywhere
4787 @code{Feval}, respectively @code{Ffuncall}, is used - either directly or
4788 through another function. Since calls to these two functions are hidden
4789 in various other functions, many calls to @code{garbage_collect_1} are
4790 not obviously foreseeable, and therefore unexpected. Instances where
4791 they are used that are worth remembering are various elisp commands, as
4792 for example @code{or}, @code{and}, @code{if}, @code{cond}, @code{while},
4793 @code{setq}, etc., miscellaneous @code{gui_item_...} functions,
4794 everything related to @code{eval} (@code{Feval_buffer}, @code{call0},
4795 ...) and inside @code{Fsignal}. The latter is used to handle signals, as
4796 for example the ones raised by every @code{QUITE}-macro triggered after
4799 @node garbage_collect_1, mark_object, Invocation, Garbage Collection - Step by Step
4800 @subsection @code{garbage_collect_1}
4801 @cindex @code{garbage_collect_1}
4803 We can now describe exactly what happens after the invocation takes
4807 There are several cases in which the garbage collector is left immediately:
4808 when we are already garbage collecting (@code{gc_in_progress}), when
4809 the garbage collection is somehow forbidden
4810 (@code{gc_currently_forbidden}), when we are currently displaying something
4811 (@code{in_display}) or when we are preparing for the armageddon of the
4812 whole system (@code{preparing_for_armageddon}).
4814 Next the correct frame in which to put
4815 all the output occurring during garbage collecting is determined. In
4816 order to be able to restore the old display's state after displaying the
4817 message, some data about the current cursor position has to be
4818 saved. The variables @code{pre_gc_cursor} and @code{cursor_changed} take
4821 The state of @code{gc_currently_forbidden} must be restored after
4822 the garbage collection, no matter what happens during the process. We
4823 accomplish this by @code{record_unwind_protect}ing the suitable function
4824 @code{restore_gc_inhibit} together with the current value of
4825 @code{gc_currently_forbidden}.
4827 If we are concurrently running an interactive xemacs session, the next step
4828 is simply to show the garbage collector's cursor/message.
4830 The following steps are the intrinsic steps of the garbage collector,
4831 therefore @code{gc_in_progress} is set.
4833 For debugging purposes, it is possible to copy the current C stack
4834 frame. However, this seems to be a currently unused feature.
4836 Before actually starting to go over all live objects, references to
4837 objects that are no longer used are pruned. We only have to do this for events
4838 (@code{clear_event_resource}) and for specifiers
4839 (@code{cleanup_specifiers}).
4841 Now the mark phase begins and marks all accessible elements. In order to
4843 all slots that serve as roots of accessibility, the function
4844 @code{mark_object} is called for each root individually to go out from
4845 there to mark all reachable objects. All roots that are traversed are
4846 shown in their processed order:
4849 all constant symbols and static variables that are registered via
4850 @code{staticpro}@ in the array @code{staticvec}.
4851 @xref{Adding Global Lisp Variables}.
4853 all Lisp objects that are created in C functions and that must be
4854 protected from freeing them. They are registered in the global
4855 list @code{gcprolist}.
4858 all local variables (i.e. their name fields @code{symbol} and old
4859 values @code{old_values}) that are bound during the evaluation by the Lisp
4860 engine. They are stored in @code{specbinding} structs pushed on a stack
4861 called @code{specpdl}.
4862 @xref{Dynamic Binding; The specbinding Stack; Unwind-Protects}.
4864 all catch blocks that the Lisp engine encounters during the evaluation
4865 cause the creation of structs @code{catchtag} inserted in the list
4866 @code{catchlist}. Their tag (@code{tag}) and value (@code{val} fields
4867 are freshly created objects and therefore have to be marked.
4868 @xref{Catch and Throw}.
4870 every function application pushes new structs @code{backtrace}
4871 on the call stack of the Lisp engine (@code{backtrace_list}). The unique
4872 parts that have to be marked are the fields for each function
4873 (@code{function}) and all their arguments (@code{args}).
4876 all objects that are used by the redisplay engine that must not be freed
4877 are marked by a special function called @code{mark_redisplay} (in
4878 @code{redisplay.c}).
4880 all objects created for profiling purposes are allocated by C functions
4881 instead of using the lisp allocation mechanisms. In order to receive the
4882 right ones during the sweep phase, they also have to be marked
4883 manually. That is done by the function @code{mark_profiling_info}
4886 Hash tables in XEmacs belong to a kind of special objects that
4887 make use of a concept often called 'weak pointers'.
4888 To make a long story short, these kind of pointers are not followed
4889 during the estimation of the live objects during garbage collection.
4890 Any object referenced only by weak pointers is collected
4891 anyway, and the reference to it is cleared. In hash tables there are
4892 different usage patterns of them, manifesting in different types of hash
4893 tables, namely 'non-weak', 'weak', 'key-weak' and 'value-weak'
4894 (internally also 'key-car-weak' and 'value-car-weak') hash tables, each
4895 clearing entries depending on different conditions. More information can
4896 be found in the documentation to the function @code{make-hash-table}.
4898 Because there are complicated dependency rules about when and what to
4899 mark while processing weak hash tables, the standard @code{marker}
4900 method is only active if it is marking non-weak hash tables. As soon as
4901 a weak component is in the table, the hash table entries are ignored
4902 while marking. Instead their marking is done each separately by the
4903 function @code{finish_marking_weak_hash_tables}. This function iterates
4904 over each hash table entry @code{hentries} for each weak hash table in
4905 @code{Vall_weak_hash_tables}. Depending on the type of a table, the
4906 appropriate action is performed.
4907 If a table is acting as @code{HASH_TABLE_KEY_WEAK}, and a key already marked,
4908 everything reachable from the @code{value} component is marked. If it is
4909 acting as a @code{HASH_TABLE_VALUE_WEAK} and the value component is
4910 already marked, the marking starts beginning only from the
4911 @code{key} component.
4912 If it is a @code{HASH_TABLE_KEY_CAR_WEAK} and the car
4913 of the key entry is already marked, we mark both the @code{key} and
4914 @code{value} components.
4915 Finally, if the table is of the type @code{HASH_TABLE_VALUE_CAR_WEAK}
4916 and the car of the value components is already marked, again both the
4917 @code{key} and the @code{value} components get marked.
4919 Again, there are lists with comparable properties called weak
4920 lists. There exist different peculiarities of their types called
4921 @code{simple}, @code{assoc}, @code{key-assoc} and
4922 @code{value-assoc}. You can find further details about them in the
4923 description to the function @code{make-weak-list}. The scheme of their
4924 marking is similar: all weak lists are listed in @code{Qall_weak_lists},
4925 therefore we iterate over them. The marking is advanced until we hit an
4926 already marked pair. Then we know that during a former run all
4927 the rest has been marked completely. Again, depending on the special
4928 type of the weak list, our jobs differ. If it is a @code{WEAK_LIST_SIMPLE}
4929 and the elem is marked, we mark the @code{cons} part. If it is a
4930 @code{WEAK_LIST_ASSOC} and not a pair or a pair with both marked car and
4931 cdr, we mark the @code{cons} and the @code{elem}. If it is a
4932 @code{WEAK_LIST_KEY_ASSOC} and not a pair or a pair with a marked car of
4933 the elem, we mark the @code{cons} and the @code{elem}. Finally, if it is
4934 a @code{WEAK_LIST_VALUE_ASSOC} and not a pair or a pair with a marked
4935 cdr of the elem, we mark both the @code{cons} and the @code{elem}.
4937 Since, by marking objects in reach from weak hash tables and weak lists,
4938 other objects could get marked, this perhaps implies further marking of
4939 other weak objects, both finishing functions are redone as long as
4940 yet unmarked objects get freshly marked.
4943 After completing the special marking for the weak hash tables and for the weak
4944 lists, all entries that point to objects that are going to be swept in
4945 the further process are useless, and therefore have to be removed from
4946 the table or the list.
4948 The function @code{prune_weak_hash_tables} does the job for weak hash
4949 tables. Totally unmarked hash tables are removed from the list
4950 @code{Vall_weak_hash_tables}. The other ones are treated more carefully
4951 by scanning over all entries and removing one as soon as one of
4952 the components @code{key} and @code{value} is unmarked.
4954 The same idea applies to the weak lists. It is accomplished by
4955 @code{prune_weak_lists}: An unmarked list is pruned from
4956 @code{Vall_weak_lists} immediately. A marked list is treated more
4957 carefully by going over it and removing just the unmarked pairs.
4960 The function @code{prune_specifiers} checks all listed specifiers held
4961 in @code{Vall_specifiers} and removes the ones from the lists that are
4965 All syntax tables are stored in a list called
4966 @code{Vall_syntax_tables}. The function @code{prune_syntax_tables} walks
4967 through it and unlinks the tables that are unmarked.
4970 Next, we will attack the complete sweeping - the function
4971 @code{gc_sweep} which holds the predominance.
4973 First, all the variables with respect to garbage collection are
4974 reset. @code{consing_since_gc} - the counter of the created cells since
4975 the last garbage collection - is set back to 0, and
4976 @code{gc_in_progress} is not @code{true} anymore.
4978 In case the session is interactive, the displayed cursor and message are
4981 The state of @code{gc_inhibit} is restored to the former value by
4982 unwinding the stack.
4984 A small memory reserve is always held back that can be reached by
4985 @code{breathing_space}. If nothing more is left, we create a new reserve
4989 @node mark_object, gc_sweep, garbage_collect_1, Garbage Collection - Step by Step
4990 @subsection @code{mark_object}
4991 @cindex @code{mark_object}
4993 The first thing that is checked while marking an object is whether the
4994 object is a real Lisp object @code{Lisp_Type_Record} or just an integer
4995 or a character. Integers and characters are the only two types that are
4996 stored directly - without another level of indirection, and therefore they
4997 don't have to be marked and collected.
4998 @xref{How Lisp Objects Are Represented in C}.
5000 The second case is the one we have to handle. It is the one when we are
5001 dealing with a pointer to a Lisp object. But, there exist also three
5002 possibilities, that prevent us from doing anything while marking: The
5003 object is read only which prevents it from being garbage collected,
5004 i.e. marked (@code{C_READONLY_RECORD_HEADER}). The object in question is
5005 already marked, and need not be marked for the second time (checked by
5006 @code{MARKED_RECORD_HEADER_P}). If it is a special, unmarkable object
5007 (@code{UNMARKABLE_RECORD_HEADER_P}, apparently, these are objects that
5008 sit in some const space, and can therefore not be marked, see
5009 @code{this_one_is_unmarkable} in @code{alloc.c}).
5011 Now, the actual marking is feasible. We do so by once using the macro
5012 @code{MARK_RECORD_HEADER} to mark the object itself (actually the
5013 special flag in the lrecord header), and calling its special marker
5014 "method" @code{marker} if available. The marker method marks every
5015 other object that is in reach from our current object. Note, that these
5016 marker methods should not call @code{mark_object} recursively, but
5017 instead should return the next object from where further marking has to
5020 In case another object was returned, as mentioned before, we reiterate
5021 the whole @code{mark_object} process beginning with this next object.
5023 @node gc_sweep, sweep_lcrecords_1, mark_object, Garbage Collection - Step by Step
5024 @subsection @code{gc_sweep}
5025 @cindex @code{gc_sweep}
5027 The job of this function is to free all unmarked records from memory. As
5028 we know, there are different types of objects implemented and managed, and
5029 consequently different ways to free them from memory.
5030 @xref{Introduction to Allocation}.
5032 We start with all objects stored through @code{lcrecords}. All
5033 bulkier objects are allocated and handled using that scheme of
5034 @code{lcrecords}. Each object is @code{malloc}ed separately
5035 instead of placing it in one of the contiguous frob blocks. All types
5036 that are currently stored
5037 using @code{lcrecords}'s @code{alloc_lcrecord} and
5038 @code{make_lcrecord_list} are the types: vectors, buffers,
5039 char-table, char-table-entry, console, weak-list, database, device,
5040 ldap, hash-table, command-builder, extent-auxiliary, extent-info, face,
5041 coding-system, frame, image-instance, glyph, popup-data, gui-item,
5042 keymap, charset, color_instance, font_instance, opaque, opaque-list,
5043 process, range-table, specifier, symbol-value-buffer-local,
5044 symbol-value-lisp-magic, symbol-value-varalias, toolbar-button,
5045 tooltalk-message, tooltalk-pattern, window, and window-configuration. We
5046 take care of them in the fist place
5047 in order to be able to handle and to finalize items stored in them more
5048 easily. The function @code{sweep_lcrecords_1} as described below is
5049 doing the whole job for us.
5050 For a description about the internals: @xref{lrecords}.
5052 Our next candidates are the other objects that behave quite differently
5053 than everything else: the strings. They consists of two parts, a
5054 fixed-size portion (@code{struct Lisp_String}) holding the string's
5055 length, its property list and a pointer to the second part, and the
5056 actual string data, which is stored in string-chars blocks comparable to
5057 frob blocks. In this block, the data is not only freed, but also a
5058 compression of holes is made, i.e. all strings are relocated together.
5059 @xref{String}. This compacting phase is performed by the function
5060 @code{compact_string_chars}, the actual sweeping by the function
5061 @code{sweep_strings} is described below.
5063 After that, the other types are swept step by step using functions
5064 @code{sweep_conses}, @code{sweep_bit_vectors_1},
5065 @code{sweep_compiled_functions}, @code{sweep_floats},
5066 @code{sweep_symbols}, @code{sweep_extents}, @code{sweep_markers} and
5067 @code{sweep_extents}. They are the fixed-size types cons, floats,
5068 compiled-functions, symbol, marker, extent, and event stored in
5069 so-called "frob blocks", and therefore we can basically do the same on
5070 every type objects, using the same macros, especially defined only to
5071 handle everything with respect to fixed-size blocks. The only fixed-size
5072 type that is not handled here are the fixed-size portion of strings,
5073 because we took special care of them earlier.
5075 The only big exceptions are bit vectors stored differently and
5076 therefore treated differently by the function @code{sweep_bit_vectors_1}
5079 At first, we need some brief information about how
5080 these fixed-size types are managed in general, in order to understand
5081 how the sweeping is done. They have all a fixed size, and are therefore
5082 stored in big blocks of memory - allocated at once - that can hold a
5083 certain amount of objects of one type. The macro
5084 @code{DECLARE_FIXED_TYPE_ALLOC} creates the suitable structures for
5085 every type. More precisely, we have the block struct
5086 (holding a pointer to the previous block @code{prev} and the
5087 objects in @code{block[]}), a pointer to current block
5088 (@code{current_..._block)}) and its last index
5089 (@code{current_..._block_index}), and a pointer to the free list that
5090 will be created. Also a macro @code{FIXED_TYPE_FROM_BLOCK} plus some
5091 related macros exists that are used to obtain a new object, either from
5092 the free list @code{ALLOCATE_FIXED_TYPE_1} if there is an unused object
5093 of that type stored or by allocating a completely new block using
5094 @code{ALLOCATE_FIXED_TYPE_FROM_BLOCK}.
5096 The rest works as follows: all of them define a
5097 macro @code{UNMARK_...} that is used to unmark the object. They define a
5098 macro @code{ADDITIONAL_FREE_...} that defines additional work that has
5099 to be done when converting an object from in use to not in use (so far,
5100 only markers use it in order to unchain them). Then, they all call
5101 the macro @code{SWEEP_FIXED_TYPE_BLOCK} instantiated with their type name
5102 and their struct name.
5104 This call in particular does the following: we go over all blocks
5105 starting with the current moving towards the oldest.
5106 For each block, we look at every object in it. If the object already
5107 freed (checked with @code{FREE_STRUCT_P} using the first pointer of the
5108 object), or if it is
5109 set to read only (@code{C_READONLY_RECORD_HEADER_P}, nothing must be
5110 done. If it is unmarked (checked with @code{MARKED_RECORD_HEADER_P}), it
5111 is put in the free list and set free (using the macro
5112 @code{FREE_FIXED_TYPE}, otherwise it stays in the block, but is unmarked
5113 (by @code{UNMARK_...}). While going through one block, we note if the
5114 whole block is empty. If so, the whole block is freed (using
5115 @code{xfree}) and the free list state is set to the state it had before
5116 handling this block.
5118 @node sweep_lcrecords_1, compact_string_chars, gc_sweep, Garbage Collection - Step by Step
5119 @subsection @code{sweep_lcrecords_1}
5120 @cindex @code{sweep_lcrecords_1}
5122 After nullifying the complete lcrecord statistics, we go over all
5123 lcrecords two separate times. They are all chained together in a list with
5124 a head called @code{all_lcrecords}.
5126 The first loop calls for each object its @code{finalizer} method, but only
5127 in the case that it is not read only
5128 (@code{C_READONLY_RECORD_HEADER_P)}, it is not already marked
5129 (@code{MARKED_RECORD_HEADER_P}), it is not already in a free list (list of
5130 freed objects, field @code{free}) and finally it owns a finalizer
5133 The second loop actually frees the appropriate objects again by iterating
5134 through the whole list. In case an object is read only or marked, it
5135 has to persist, otherwise it is manually freed by calling
5136 @code{xfree}. During this loop, the lcrecord statistics are kept up to
5137 date by calling @code{tick_lcrecord_stats} with the right arguments,
5139 @node compact_string_chars, sweep_strings, sweep_lcrecords_1, Garbage Collection - Step by Step
5140 @subsection @code{compact_string_chars}
5141 @cindex @code{compact_string_chars}
5143 The purpose of this function is to compact all the data parts of the
5144 strings that are held in so-called @code{string_chars_block}, i.e. the
5145 strings that do not exceed a certain maximal length.
5147 The procedure with which this is done is as follows. We are keeping two
5148 positions in the @code{string_chars_block}s using two pointer/integer
5149 pairs, namely @code{from_sb}/@code{from_pos} and
5150 @code{to_sb}/@code{to_pos}. They stand for the actual positions, from
5151 where to where, to copy the actually handled string.
5153 While going over all chained @code{string_char_block}s and their held
5154 strings, staring at @code{first_string_chars_block}, both pointers
5155 are advanced and eventually a string is copied from @code{from_sb} to
5156 @code{to_sb}, depending on the status of the pointed at strings.
5158 More precisely, we can distinguish between the following actions.
5161 The string at @code{from_sb}'s position could be marked as free, which
5162 is indicated by an invalid pointer to the pointer that should point back
5163 to the fixed size string object, and which is checked by
5164 @code{FREE_STRUCT_P}. In this case, the @code{from_sb}/@code{from_pos}
5165 is advanced to the next string, and nothing has to be copied.
5167 Also, if a string object itself is unmarked, nothing has to be
5168 copied. We likewise advance the @code{from_sb}/@code{from_pos}
5169 pair as described above.
5171 In all other cases, we have a marked string at hand. The string data
5172 must be moved from the from-position to the to-position. In case
5173 there is not enough space in the actual @code{to_sb}-block, we advance
5174 this pointer to the beginning of the next block before copying. In case the
5175 from and to positions are different, we perform the
5176 actual copying using the library function @code{memmove}.
5179 After compacting, the pointer to the current
5180 @code{string_chars_block}, sitting in @code{current_string_chars_block},
5181 is reset on the last block to which we moved a string,
5182 i.e. @code{to_block}, and all remaining blocks (we know that they just
5183 carry garbage) are explicitly @code{xfree}d.
5185 @node sweep_strings, sweep_bit_vectors_1, compact_string_chars, Garbage Collection - Step by Step
5186 @subsection @code{sweep_strings}
5187 @cindex @code{sweep_strings}
5189 The sweeping for the fixed sized string objects is essentially exactly
5190 the same as it is for all other fixed size types. As before, the freeing
5191 into the suitable free list is done by using the macro
5192 @code{SWEEP_FIXED_SIZE_BLOCK} after defining the right macros
5193 @code{UNMARK_string} and @code{ADDITIONAL_FREE_string}. These two
5194 definitions are a little bit special compared to the ones used
5195 for the other fixed size types.
5197 @code{UNMARK_string} is defined the same way except some additional code
5198 used for updating the bookkeeping information.
5200 For strings, @code{ADDITIONAL_FREE_string} has to do something in
5201 addition: in case, the string was not allocated in a
5202 @code{string_chars_block} because it exceeded the maximal length, and
5203 therefore it was @code{malloc}ed separately, we know also @code{xfree}
5206 @node sweep_bit_vectors_1, , sweep_strings, Garbage Collection - Step by Step
5207 @subsection @code{sweep_bit_vectors_1}
5208 @cindex @code{sweep_bit_vectors_1}
5210 Bit vectors are also one of the rare types that are @code{malloc}ed
5211 individually. Consequently, while sweeping, all further needless
5212 bit vectors must be freed by hand. This is done, as one might imagine,
5213 the expected way: since they are all registered in a list called
5214 @code{all_bit_vectors}, all elements of that list are traversed,
5215 all unmarked bit vectors are unlinked by calling @code{xfree} and all of
5216 them become unmarked.
5217 In addition, the bookkeeping information used for garbage
5218 collector's output purposes is updated.
5220 @node Integers and Characters, Allocation from Frob Blocks, Garbage Collection - Step by Step, Allocation of Objects in XEmacs Lisp
5221 @section Integers and Characters
5223 Integer and character Lisp objects are created from integers using the
5224 macros @code{XSETINT()} and @code{XSETCHAR()} or the equivalent
5225 functions @code{make_int()} and @code{make_char()}. (These are actually
5226 macros on most systems.) These functions basically just do some moving
5227 of bits around, since the integral value of the object is stored
5228 directly in the @code{Lisp_Object}.
5230 @code{XSETINT()} and the like will truncate values given to them that
5231 are too big; i.e. you won't get the value you expected but the tag bits
5232 will at least be correct.
5234 @node Allocation from Frob Blocks, lrecords, Integers and Characters, Allocation of Objects in XEmacs Lisp
5235 @section Allocation from Frob Blocks
5237 The uninitialized memory required by a @code{Lisp_Object} of a particular type
5239 @code{ALLOCATE_FIXED_TYPE()}. This only occurs inside of the
5240 lowest-level object-creating functions in @file{alloc.c}:
5241 @code{Fcons()}, @code{make_float()}, @code{Fmake_byte_code()},
5242 @code{Fmake_symbol()}, @code{allocate_extent()},
5243 @code{allocate_event()}, @code{Fmake_marker()}, and
5244 @code{make_uninit_string()}. The idea is that, for each type, there are
5245 a number of frob blocks (each 2K in size); each frob block is divided up
5246 into object-sized chunks. Each frob block will have some of these
5247 chunks that are currently assigned to objects, and perhaps some that are
5248 free. (If a frob block has nothing but free chunks, it is freed at the
5249 end of the garbage collection cycle.) The free chunks are stored in a
5250 free list, which is chained by storing a pointer in the first four bytes
5251 of the chunk. (Except for the free chunks at the end of the last frob
5252 block, which are handled using an index which points past the end of the
5253 last-allocated chunk in the last frob block.)
5254 @code{ALLOCATE_FIXED_TYPE()} first tries to retrieve a chunk from the
5255 free list; if that fails, it calls
5256 @code{ALLOCATE_FIXED_TYPE_FROM_BLOCK()}, which looks at the end of the
5257 last frob block for space, and creates a new frob block if there is
5258 none. (There are actually two versions of these macros, one of which is
5259 more defensive but less efficient and is used for error-checking.)
5261 @node lrecords, Low-level allocation, Allocation from Frob Blocks, Allocation of Objects in XEmacs Lisp
5264 [see @file{lrecord.h}]
5266 All lrecords have at the beginning of their structure a @code{struct
5267 lrecord_header}. This just contains a type number and some flags,
5268 including the mark bit. All builtin type numbers are defined as
5269 constants in @code{enum lrecord_type}, to allow the compiler to generate
5270 more efficient code for @code{@var{type}P}. The type number, thru the
5271 @code{lrecord_implementation_table}, gives access to a @code{struct
5272 lrecord_implementation}, which is a structure containing method pointers
5273 and such. There is one of these for each type, and it is a global,
5274 constant, statically-declared structure that is declared in the
5275 @code{DEFINE_LRECORD_IMPLEMENTATION()} macro.
5277 Simple lrecords (of type (b) above) just have a @code{struct
5278 lrecord_header} at their beginning. lcrecords, however, actually have a
5279 @code{struct lcrecord_header}. This, in turn, has a @code{struct
5280 lrecord_header} at its beginning, so sanity is preserved; but it also
5281 has a pointer used to chain all lcrecords together, and a special ID
5282 field used to distinguish one lcrecord from another. (This field is used
5283 only for debugging and could be removed, but the space gain is not
5286 Simple lrecords are created using @code{ALLOCATE_FIXED_TYPE()}, just
5287 like for other frob blocks. The only change is that the implementation
5288 pointer must be initialized correctly. (The implementation structure for
5289 an lrecord, or rather the pointer to it, is named @code{lrecord_float},
5290 @code{lrecord_extent}, @code{lrecord_buffer}, etc.)
5292 lcrecords are created using @code{alloc_lcrecord()}. This takes a
5293 size to allocate and an implementation pointer. (The size needs to be
5294 passed because some lcrecords, such as window configurations, are of
5295 variable size.) This basically just @code{malloc()}s the storage,
5296 initializes the @code{struct lcrecord_header}, and chains the lcrecord
5297 onto the head of the list of all lcrecords, which is stored in the
5298 variable @code{all_lcrecords}. The calls to @code{alloc_lcrecord()}
5299 generally occur in the lowest-level allocation function for each lrecord
5302 Whenever you create an lrecord, you need to call either
5303 @code{DEFINE_LRECORD_IMPLEMENTATION()} or
5304 @code{DEFINE_LRECORD_SEQUENCE_IMPLEMENTATION()}. This needs to be
5305 specified in a @file{.c} file, at the top level. What this actually
5306 does is define and initialize the implementation structure for the
5307 lrecord. (And possibly declares a function @code{error_check_foo()} that
5308 implements the @code{XFOO()} macro when error-checking is enabled.) The
5309 arguments to the macros are the actual type name (this is used to
5310 construct the C variable name of the lrecord implementation structure
5311 and related structures using the @samp{##} macro concatenation
5312 operator), a string that names the type on the Lisp level (this may not
5313 be the same as the C type name; typically, the C type name has
5314 underscores, while the Lisp string has dashes), various method pointers,
5315 and the name of the C structure that contains the object. The methods
5316 are used to encapsulate type-specific information about the object, such
5317 as how to print it or mark it for garbage collection, so that it's easy
5318 to add new object types without having to add a specific case for each
5319 new type in a bunch of different places.
5321 The difference between @code{DEFINE_LRECORD_IMPLEMENTATION()} and
5322 @code{DEFINE_LRECORD_SEQUENCE_IMPLEMENTATION()} is that the former is
5323 used for fixed-size object types and the latter is for variable-size
5324 object types. Most object types are fixed-size; some complex
5325 types, however (e.g. window configurations), are variable-size.
5326 Variable-size object types have an extra method, which is called
5327 to determine the actual size of a particular object of that type.
5328 (Currently this is only used for keeping allocation statistics.)
5330 For the purpose of keeping allocation statistics, the allocation
5331 engine keeps a list of all the different types that exist. Note that,
5332 since @code{DEFINE_LRECORD_IMPLEMENTATION()} is a macro that is
5333 specified at top-level, there is no way for it to initialize the global
5334 data structures containing type information, like
5335 @code{lrecord_implementations_table}. For this reason a call to
5336 @code{INIT_LRECORD_IMPLEMENTATION} must be added to the same source file
5337 containing @code{DEFINE_LRECORD_IMPLEMENTATION}, but instead of to the
5338 top level, to one of the init functions, typically
5339 @code{syms_of_@var{foo}.c}. @code{INIT_LRECORD_IMPLEMENTATION} must be
5340 called before an object of this type is used.
5342 The type number is also used to index into an array holding the number
5343 of objects of each type and the total memory allocated for objects of
5344 that type. The statistics in this array are computed during the sweep
5345 stage. These statistics are returned by the call to
5346 @code{garbage-collect}.
5348 Note that for every type defined with a @code{DEFINE_LRECORD_*()}
5349 macro, there needs to be a @code{DECLARE_LRECORD_IMPLEMENTATION()}
5350 somewhere in a @file{.h} file, and this @file{.h} file needs to be
5351 included by @file{inline.c}.
5353 Furthermore, there should generally be a set of @code{XFOOBAR()},
5354 @code{FOOBARP()}, etc. macros in a @file{.h} (or occasionally @file{.c})
5355 file. To create one of these, copy an existing model and modify as
5358 The various methods in the lrecord implementation structure are:
5363 A @dfn{mark} method. This is called during the marking stage and passed
5364 a function pointer (usually the @code{mark_object()} function), which is
5365 used to mark an object. All Lisp objects that are contained within the
5366 object need to be marked by applying this function to them. The mark
5367 method should also return a Lisp object, which should be either nil or
5368 an object to mark. (This can be used in lieu of calling
5369 @code{mark_object()} on the object, to reduce the recursion depth, and
5370 consequently should be the most heavily nested sub-object, such as a
5373 @strong{Please note:} When the mark method is called, garbage collection
5374 is in progress, and special precautions need to be taken when accessing
5375 objects; see section (B) above.
5377 If your mark method does not need to do anything, it can be
5381 A @dfn{print} method. This is called to create a printed representation
5382 of the object, whenever @code{princ}, @code{prin1}, or the like is
5383 called. It is passed the object, a stream to which the output is to be
5384 directed, and an @code{escapeflag} which indicates whether the object's
5385 printed representation should be @dfn{escaped} so that it is
5386 readable. (This corresponds to the difference between @code{princ} and
5387 @code{prin1}.) Basically, @dfn{escaped} means that strings will have
5388 quotes around them and confusing characters in the strings such as
5389 quotes, backslashes, and newlines will be backslashed; and that special
5390 care will be taken to make symbols print in a readable fashion
5391 (e.g. symbols that look like numbers will be backslashed). Other
5392 readable objects should perhaps pass @code{escapeflag} on when
5393 sub-objects are printed, so that readability is preserved when necessary
5394 (or if not, always pass in a 1 for @code{escapeflag}). Non-readable
5395 objects should in general ignore @code{escapeflag}, except that some use
5396 it as an indication that more verbose output should be given.
5398 Sub-objects are printed using @code{print_internal()}, which takes
5399 exactly the same arguments as are passed to the print method.
5401 Literal C strings should be printed using @code{write_c_string()},
5402 or @code{write_string_1()} for non-null-terminated strings.
5404 Functions that do not have a readable representation should check the
5405 @code{print_readably} flag and signal an error if it is set.
5407 If you specify NULL for the print method, the
5408 @code{default_object_printer()} will be used.
5411 A @dfn{finalize} method. This is called at the beginning of the sweep
5412 stage on lcrecords that are about to be freed, and should be used to
5413 perform any extra object cleanup. This typically involves freeing any
5414 extra @code{malloc()}ed memory associated with the object, releasing any
5415 operating-system and window-system resources associated with the object
5416 (e.g. pixmaps, fonts), etc.
5418 The finalize method can be NULL if nothing needs to be done.
5420 WARNING #1: The finalize method is also called at the end of the dump
5421 phase; this time with the for_disksave parameter set to non-zero. The
5422 object is @emph{not} about to disappear, so you have to make sure to
5423 @emph{not} free any extra @code{malloc()}ed memory if you're going to
5424 need it later. (Also, signal an error if there are any operating-system
5425 and window-system resources here, because they can't be dumped.)
5427 Finalize methods should, as a rule, set to zero any pointers after
5428 they've been freed, and check to make sure pointers are not zero before
5429 freeing. Although I'm pretty sure that finalize methods are not called
5430 twice on the same object (except for the @code{for_disksave} proviso),
5431 we've gotten nastily burned in some cases by not doing this.
5433 WARNING #2: The finalize method is @emph{only} called for
5434 lcrecords, @emph{not} for simply lrecords. If you need a
5435 finalize method for simple lrecords, you have to stick
5436 it in the @code{ADDITIONAL_FREE_foo()} macro in @file{alloc.c}.
5438 WARNING #3: Things are in an @emph{extremely} bizarre state
5439 when @code{ADDITIONAL_FREE_foo()} is called, so you have to
5440 be incredibly careful when writing one of these functions.
5441 See the comment in @code{gc_sweep()}. If you ever have to add
5442 one of these, consider using an lcrecord or dealing with
5443 the problem in a different fashion.
5446 An @dfn{equal} method. This compares the two objects for similarity,
5447 when @code{equal} is called. It should compare the contents of the
5448 objects in some reasonable fashion. It is passed the two objects and a
5449 @dfn{depth} value, which is used to catch circular objects. To compare
5450 sub-Lisp-objects, call @code{internal_equal()} and bump the depth value
5451 by one. If this value gets too high, a @code{circular-object} error
5454 If this is NULL, objects are @code{equal} only when they are @code{eq},
5458 A @dfn{hash} method. This is used to hash objects when they are to be
5459 compared with @code{equal}. The rule here is that if two objects are
5460 @code{equal}, they @emph{must} hash to the same value; i.e. your hash
5461 function should use some subset of the sub-fields of the object that are
5462 compared in the ``equal'' method. If you specify this method as
5463 @code{NULL}, the object's pointer will be used as the hash, which will
5464 @emph{fail} if the object has an @code{equal} method, so don't do this.
5466 To hash a sub-Lisp-object, call @code{internal_hash()}. Bump the
5467 depth by one, just like in the ``equal'' method.
5469 To convert a Lisp object directly into a hash value (using
5470 its pointer), use @code{LISP_HASH()}. This is what happens when
5471 the hash method is NULL.
5473 To hash two or more values together into a single value, use
5474 @code{HASH2()}, @code{HASH3()}, @code{HASH4()}, etc.
5477 @dfn{getprop}, @dfn{putprop}, @dfn{remprop}, and @dfn{plist} methods.
5478 These are used for object types that have properties. I don't feel like
5479 documenting them here. If you create one of these objects, you have to
5480 use different macros to define them,
5481 i.e. @code{DEFINE_LRECORD_IMPLEMENTATION_WITH_PROPS()} or
5482 @code{DEFINE_LRECORD_SEQUENCE_IMPLEMENTATION_WITH_PROPS()}.
5485 A @dfn{size_in_bytes} method, when the object is of variable-size.
5486 (i.e. declared with a @code{_SEQUENCE_IMPLEMENTATION} macro.) This should
5487 simply return the object's size in bytes, exactly as you might expect.
5488 For an example, see the methods for window configurations and opaques.
5491 @node Low-level allocation, Cons, lrecords, Allocation of Objects in XEmacs Lisp
5492 @section Low-level allocation
5494 Memory that you want to allocate directly should be allocated using
5495 @code{xmalloc()} rather than @code{malloc()}. This implements
5496 error-checking on the return value, and once upon a time did some more
5497 vital stuff (i.e. @code{BLOCK_INPUT}, which is no longer necessary).
5498 Free using @code{xfree()}, and realloc using @code{xrealloc()}. Note
5499 that @code{xmalloc()} will do a non-local exit if the memory can't be
5500 allocated. (Many functions, however, do not expect this, and thus XEmacs
5501 will likely crash if this happens. @strong{This is a bug.} If you can,
5502 you should strive to make your function handle this OK. However, it's
5503 difficult in the general circumstance, perhaps requiring extra
5504 unwind-protects and such.)
5506 Note that XEmacs provides two separate replacements for the standard
5507 @code{malloc()} library function. These are called @dfn{old GNU malloc}
5508 (@file{malloc.c}) and @dfn{new GNU malloc} (@file{gmalloc.c}),
5509 respectively. New GNU malloc is better in pretty much every way than
5510 old GNU malloc, and should be used if possible. (It used to be that on
5511 some systems, the old one worked but the new one didn't. I think this
5512 was due specifically to a bug in SunOS, which the new one now works
5513 around; so I don't think the old one ever has to be used any more.) The
5514 primary difference between both of these mallocs and the standard system
5515 malloc is that they are much faster, at the expense of increased space.
5516 The basic idea is that memory is allocated in fixed chunks of powers of
5517 two. This allows for basically constant malloc time, since the various
5518 chunks can just be kept on a number of free lists. (The standard system
5519 malloc typically allocates arbitrary-sized chunks and has to spend some
5520 time, sometimes a significant amount of time, walking the heap looking
5521 for a free block to use and cleaning things up.) The new GNU malloc
5522 improves on things by allocating large objects in chunks of 4096 bytes
5523 rather than in ever larger powers of two, which results in ever larger
5524 wastage. There is a slight speed loss here, but it's of doubtful
5527 NOTE: Apparently there is a third-generation GNU malloc that is
5528 significantly better than the new GNU malloc, and should probably
5529 be included in XEmacs.
5531 There is also the relocating allocator, @file{ralloc.c}. This actually
5532 moves blocks of memory around so that the @code{sbrk()} pointer shrunk
5533 and virtual memory released back to the system. On some systems,
5534 this is a big win. On all systems, it causes a noticeable (and
5535 sometimes huge) speed penalty, so I turn it off by default.
5536 @file{ralloc.c} only works with the new GNU malloc in @file{gmalloc.c}.
5537 There are also two versions of @file{ralloc.c}, one that uses @code{mmap()}
5538 rather than block copies to move data around. This purports to
5539 be faster, although that depends on the amount of data that would
5540 have had to be block copied and the system-call overhead for
5541 @code{mmap()}. I don't know exactly how this works, except that the
5542 relocating-allocation routines are pretty much used only for
5543 the memory allocated for a buffer, which is the biggest consumer
5544 of space, esp. of space that may get freed later.
5546 Note that the GNU mallocs have some ``memory warning'' facilities.
5547 XEmacs taps into them and issues a warning through the standard
5548 warning system, when memory gets to 75%, 85%, and 95% full.
5549 (On some systems, the memory warnings are not functional.)
5551 Allocated memory that is going to be used to make a Lisp object
5552 is created using @code{allocate_lisp_storage()}. This just calls
5553 @code{xmalloc()}. It used to verify that the pointer to the memory can
5554 fit into a Lisp word, before the current Lisp object representation was
5555 introduced. @code{allocate_lisp_storage()} is called by
5556 @code{alloc_lcrecord()}, @code{ALLOCATE_FIXED_TYPE()}, and the vector
5557 and bit-vector creation routines. These routines also call
5558 @code{INCREMENT_CONS_COUNTER()} at the appropriate times; this keeps
5559 statistics on how much memory is allocated, so that garbage-collection
5560 can be invoked when the threshold is reached.
5562 @node Cons, Vector, Low-level allocation, Allocation of Objects in XEmacs Lisp
5565 Conses are allocated in standard frob blocks. The only thing to
5566 note is that conses can be explicitly freed using @code{free_cons()}
5567 and associated functions @code{free_list()} and @code{free_alist()}. This
5568 immediately puts the conses onto the cons free list, and decrements
5569 the statistics on memory allocation appropriately. This is used
5570 to good effect by some extremely commonly-used code, to avoid
5571 generating extra objects and thereby triggering GC sooner.
5572 However, you have to be @emph{extremely} careful when doing this.
5573 If you mess this up, you will get BADLY BURNED, and it has happened
5576 @node Vector, Bit Vector, Cons, Allocation of Objects in XEmacs Lisp
5579 As mentioned above, each vector is @code{malloc()}ed individually, and
5580 all are threaded through the variable @code{all_vectors}. Vectors are
5581 marked strangely during garbage collection, by kludging the size field.
5582 Note that the @code{struct Lisp_Vector} is declared with its
5583 @code{contents} field being a @emph{stretchy} array of one element. It
5584 is actually @code{malloc()}ed with the right size, however, and access
5585 to any element through the @code{contents} array works fine.
5587 @node Bit Vector, Symbol, Vector, Allocation of Objects in XEmacs Lisp
5590 Bit vectors work exactly like vectors, except for more complicated
5591 code to access an individual bit, and except for the fact that bit
5592 vectors are lrecords while vectors are not. (The only difference here is
5593 that there's an lrecord implementation pointer at the beginning and the
5594 tag field in bit vector Lisp words is ``lrecord'' rather than
5597 @node Symbol, Marker, Bit Vector, Allocation of Objects in XEmacs Lisp
5600 Symbols are also allocated in frob blocks. Symbols in the awful
5601 horrible obarray structure are chained through their @code{next} field.
5603 Remember that @code{intern} looks up a symbol in an obarray, creating
5606 @node Marker, String, Symbol, Allocation of Objects in XEmacs Lisp
5609 Markers are allocated in frob blocks, as usual. They are kept
5610 in a buffer unordered, but in a doubly-linked list so that they
5611 can easily be removed. (Formerly this was a singly-linked list,
5612 but in some cases garbage collection took an extraordinarily
5613 long time due to the O(N^2) time required to remove lots of
5614 markers from a buffer.) Markers are removed from a buffer in
5615 the finalize stage, in @code{ADDITIONAL_FREE_marker()}.
5617 @node String, Compiled Function, Marker, Allocation of Objects in XEmacs Lisp
5620 As mentioned above, strings are a special case. A string is logically
5621 two parts, a fixed-size object (containing the length, property list,
5622 and a pointer to the actual data), and the actual data in the string.
5623 The fixed-size object is a @code{struct Lisp_String} and is allocated in
5624 frob blocks, as usual. The actual data is stored in special
5625 @dfn{string-chars blocks}, which are 8K blocks of memory.
5626 Currently-allocated strings are simply laid end to end in these
5627 string-chars blocks, with a pointer back to the @code{struct Lisp_String}
5628 stored before each string in the string-chars block. When a new string
5629 needs to be allocated, the remaining space at the end of the last
5630 string-chars block is used if there's enough, and a new string-chars
5631 block is created otherwise.
5633 There are never any holes in the string-chars blocks due to the string
5634 compaction and relocation that happens at the end of garbage collection.
5635 During the sweep stage of garbage collection, when objects are
5636 reclaimed, the garbage collector goes through all string-chars blocks,
5637 looking for unused strings. Each chunk of string data is preceded by a
5638 pointer to the corresponding @code{struct Lisp_String}, which indicates
5639 both whether the string is used and how big the string is, i.e. how to
5640 get to the next chunk of string data. Holes are compressed by
5641 block-copying the next string into the empty space and relocating the
5642 pointer stored in the corresponding @code{struct Lisp_String}.
5643 @strong{This means you have to be careful with strings in your code.}
5644 See the section above on @code{GCPRO}ing.
5646 Note that there is one situation not handled: a string that is too big
5647 to fit into a string-chars block. Such strings, called @dfn{big
5648 strings}, are all @code{malloc()}ed as their own block. (#### Although it
5649 would make more sense for the threshold for big strings to be somewhat
5650 lower, e.g. 1/2 or 1/4 the size of a string-chars block. It seems that
5651 this was indeed the case formerly---indeed, the threshold was set at
5652 1/8---but Mly forgot about this when rewriting things for 19.8.)
5654 Note also that the string data in string-chars blocks is padded as
5655 necessary so that proper alignment constraints on the @code{struct
5656 Lisp_String} back pointers are maintained.
5658 Finally, strings can be resized. This happens in Mule when a
5659 character is substituted with a different-length character, or during
5660 modeline frobbing. (You could also export this to Lisp, but it's not
5661 done so currently.) Resizing a string is a potentially tricky process.
5662 If the change is small enough that the padding can absorb it, nothing
5663 other than a simple memory move needs to be done. Keep in mind,
5664 however, that the string can't shrink too much because the offset to the
5665 next string in the string-chars block is computed by looking at the
5666 length and rounding to the nearest multiple of four or eight. If the
5667 string would shrink or expand beyond the correct padding, new string
5668 data needs to be allocated at the end of the last string-chars block and
5669 the data moved appropriately. This leaves some dead string data, which
5670 is marked by putting a special marker of 0xFFFFFFFF in the @code{struct
5671 Lisp_String} pointer before the data (there's no real @code{struct
5672 Lisp_String} to point to and relocate), and storing the size of the dead
5673 string data (which would normally be obtained from the now-non-existent
5674 @code{struct Lisp_String}) at the beginning of the dead string data gap.
5675 The string compactor recognizes this special 0xFFFFFFFF marker and
5676 handles it correctly.
5678 @node Compiled Function, , String, Allocation of Objects in XEmacs Lisp
5679 @section Compiled Function
5684 @node Dumping, Events and the Event Loop, Allocation of Objects in XEmacs Lisp, Top
5687 @section What is dumping and its justification
5689 The C code of XEmacs is just a Lisp engine with a lot of built-in
5690 primitives useful for writing an editor. The editor itself is written
5691 mostly in Lisp, and represents around 100K lines of code. Loading and
5692 executing the initialization of all this code takes a bit a time (five
5693 to ten times the usual startup time of current xemacs) and requires
5694 having all the lisp source files around. Having to reload them each
5695 time the editor is started would not be acceptable.
5697 The traditional solution to this problem is called dumping: the build
5698 process first creates the lisp engine under the name @file{temacs}, then
5699 runs it until it has finished loading and initializing all the lisp
5700 code, and eventually creates a new executable called @file{xemacs}
5701 including both the object code in @file{temacs} and all the contents of
5702 the memory after the initialization.
5704 This solution, while working, has a huge problem: the creation of the
5705 new executable from the actual contents of memory is an extremely
5706 system-specific process, quite error-prone, and which interferes with a
5707 lot of system libraries (like malloc). It is even getting worse
5708 nowadays with libraries using constructors which are automatically
5709 called when the program is started (even before main()) which tend to
5710 crash when they are called multiple times, once before dumping and once
5711 after (IRIX 6.x libz.so pulls in some C++ image libraries thru
5712 dependencies which have this problem). Writing the dumper is also one
5713 of the most difficult parts of porting XEmacs to a new operating system.
5714 Basically, `dumping' is an operation that is just not officially
5715 supported on many operating systems.
5717 The aim of the portable dumper is to solve the same problem as the
5718 system-specific dumper, that is to be able to reload quickly, using only
5719 a small number of files, the fully initialized lisp part of the editor,
5720 without any system-specific hacks.
5724 * Data descriptions::
5727 * Remaining issues::
5730 @node Overview, Data descriptions, Dumping, Dumping
5733 The portable dumping system has to:
5737 At dump time, write all initialized, non-quickly-rebuildable data to a
5738 file [Note: currently named @file{xemacs.dmp}, but the name will
5739 change], along with all informations needed for the reloading.
5742 When starting xemacs, reload the dump file, relocate it to its new
5743 starting address if needed, and reinitialize all pointers to this
5744 data. Also, rebuild all the quickly rebuildable data.
5747 @node Data descriptions, Dumping phase, Overview, Dumping
5748 @section Data descriptions
5750 The more complex task of the dumper is to be able to write lisp objects
5751 (lrecords) and C structs to disk and reload them at a different address,
5752 updating all the pointers they include in the process. This is done by
5753 using external data descriptions that give information about the layout
5754 of the structures in memory.
5756 The specification of these descriptions is in lrecord.h. A description
5757 of an lrecord is an array of struct lrecord_description. Each of these
5758 structs include a type, an offset in the structure and some optional
5759 parameters depending on the type. For instance, here is the string
5763 static const struct lrecord_description string_description[] = @{
5764 @{ XD_BYTECOUNT, offsetof (Lisp_String, size) @},
5765 @{ XD_OPAQUE_DATA_PTR, offsetof (Lisp_String, data), XD_INDIRECT(0, 1) @},
5766 @{ XD_LISP_OBJECT, offsetof (Lisp_String, plist) @},
5771 The first line indicates a member of type Bytecount, which is used by
5772 the next, indirect directive. The second means "there is a pointer to
5773 some opaque data in the field @code{data}". The length of said data is
5774 given by the expression @code{XD_INDIRECT(0, 1)}, which means "the value
5775 in the 0th line of the description (welcome to C) plus one". The third
5776 line means "there is a Lisp_Object member @code{plist} in the Lisp_String
5777 structure". @code{XD_END} then ends the description.
5779 This gives us all the information we need to move around what is pointed
5780 to by a structure (C or lrecord) and, by transitivity, everything that
5781 it points to. The only missing information for dumping is the size of
5782 the structure. For lrecords, this is part of the
5783 lrecord_implementation, so we don't need to duplicate it. For C
5784 structures we use a struct struct_description, which includes a size
5785 field and a pointer to an associated array of lrecord_description.
5787 @node Dumping phase, Reloading phase, Data descriptions, Dumping
5788 @section Dumping phase
5790 Dumping is done by calling the function pdump() (in dumper.c) which is
5791 invoked from Fdump_emacs (in emacs.c). This function performs a number
5795 * Object inventory::
5796 * Address allocation::
5799 * Pointers dumping::
5802 @node Object inventory, Address allocation, Dumping phase, Dumping phase
5803 @subsection Object inventory
5805 The first task is to build the list of the objects to dump. This
5813 We end up with one @code{pdump_entry_list_elmt} per object group (arrays
5814 of C structs are kept together) which includes a pointer to the first
5815 object of the group, the per-object size and the count of objects in the
5816 group, along with some other information which is initialized later.
5818 These entries are linked together in @code{pdump_entry_list} structures
5819 and can be enumerated thru either:
5823 the @code{pdump_object_table}, an array of @code{pdump_entry_list}, one
5824 per lrecord type, indexed by type number.
5827 the @code{pdump_opaque_data_list}, used for the opaque data which does
5828 not include pointers, and hence does not need descriptions.
5831 the @code{pdump_struct_table}, which is a vector of
5832 @code{struct_description}/@code{pdump_entry_list} pairs, used for
5833 non-opaque C structures.
5836 This uses a marking strategy similar to the garbage collector. Some
5841 We do not use the mark bit (which does not exist for C structures
5842 anyway), we use a big hash table instead.
5845 We do not use the mark function of lrecords but instead rely on the
5846 external descriptions. This happens essentially because we need to
5847 follow pointers to C structures and opaque data in addition to
5848 Lisp_Object members.
5851 This is done by @code{pdump_register_object}, which handles Lisp_Object
5852 variables, and pdump_register_struct which handles C structures, which
5853 both delegate the description management to pdump_register_sub.
5855 The hash table doubles as a map object to pdump_entry_list_elmt (i.e.
5856 allows us to look up a pdump_entry_list_elmt with the object it points
5857 to). Entries are added with @code{pdump_add_entry()} and looked up with
5858 @code{pdump_get_entry()}. There is no need for entry removal. The hash
5859 value is computed quite basically from the object pointer by
5860 @code{pdump_make_hash()}.
5862 The roots for the marking are:
5866 the @code{staticpro}'ed variables (there is a special @code{staticpro_nodump()}
5867 call for protected variables we do not want to dump).
5870 the @code{pdump_wire}'d variables (@code{staticpro} is equivalent to
5871 @code{staticpro_nodump()} + @code{pdump_wire()}).
5874 the @code{dumpstruct}'ed variables, which points to C structures.
5877 This does not include the GCPRO'ed variables, the specbinds, the
5878 catchtags, the backlist, the redisplay or the profiling info, since we
5879 do not want to rebuild the actual chain of lisp calls which end up to
5880 the dump-emacs call, only the global variables.
5882 Weak lists and weak hash tables are dumped as if they were their
5883 non-weak equivalent (without changing their type, of course). This has
5884 not yet been a problem.
5886 @node Address allocation, The header, Object inventory, Dumping phase
5887 @subsection Address allocation
5890 The next step is to allocate the offsets of each of the objects in the
5891 final dump file. This is done by @code{pdump_allocate_offset()} which
5892 is called indirectly by @code{pdump_scan_by_alignment()}.
5894 The strategy to deal with alignment problems uses these facts:
5898 real world alignment requirements are powers of two.
5901 the C compiler is required to adjust the size of a struct so that you
5902 can have an array of them next to each other. This means you can have a
5903 upper bound of the alignment requirements of a given structure by
5904 looking at which power of two its size is a multiple.
5907 the non-variant part of variable size lrecords has an alignment
5911 Hence, for each lrecord type, C struct type or opaque data block the
5912 alignment requirement is computed as a power of two, with a minimum of
5913 2^2 for lrecords. @code{pdump_scan_by_alignment()} then scans all the
5914 @code{pdump_entry_list_elmt}'s, the ones with the highest requirements
5915 first. This ensures the best packing.
5917 The maximum alignment requirement we take into account is 2^8.
5919 @code{pdump_allocate_offset()} only has to do a linear allocation,
5920 starting at offset 256 (this leaves room for the header and keep the
5923 @node The header, Data dumping, Address allocation, Dumping phase
5924 @subsection The header
5926 The next step creates the file and writes a header with a signature and
5927 some random informations in it (number of staticpro, number of assigned
5928 lrecord types, etc...). The reloc_address field, which indicates at
5929 which address the file should be loaded if we want to avoid post-reload
5930 relocation, is set to 0. It then seeks to offset 256 (base offset for
5933 @node Data dumping, Pointers dumping, The header, Dumping phase
5934 @subsection Data dumping
5936 The data is dumped in the same order as the addresses were allocated by
5937 @code{pdump_dump_data()}, called from @code{pdump_scan_by_alignment()}.
5938 This function copies the data to a temporary buffer, relocates all
5939 pointers in the object to the addresses allocated in step Address
5940 Allocation, and writes it to the file. Using the same order means that,
5941 if we are careful with lrecords whose size is not a multiple of 4, we
5942 are ensured that the object is always written at the offset in the file
5943 allocated in step Address Allocation.
5945 @node Pointers dumping, , Data dumping, Dumping phase
5946 @subsection Pointers dumping
5948 A bunch of tables needed to reassign properly the global pointers are
5949 then written. They are:
5955 the dumpstruct array
5957 the lrecord_implementation_table array
5959 a vector of all the offsets to the objects in the file that include a
5960 description (for faster relocation at reload time)
5962 the pdump_wired and pdump_wired_list arrays
5965 For each of the arrays we write both the pointer to the variables and
5966 the relocated offset of the object they point to. Since these variables
5967 are global, the pointers are still valid when restarting the program and
5968 are used to regenerate the global pointers.
5970 The @code{pdump_wired_list} array is a special case. The variables it
5971 points to are the head of weak linked lists of lisp objects of the same
5972 type. Not all objects of this list are dumped so the relocated pointer
5973 we associate with them points to the first dumped object of the list, or
5974 Qnil if none is available. This is also the reason why they are not
5975 used as roots for the purpose of object enumeration.
5977 This is the end of the dumping part.
5979 @node Reloading phase, Remaining issues, Dumping phase, Dumping
5980 @section Reloading phase
5982 @subsection File loading
5984 The file is mmap'ed in memory (which ensures a PAGESIZE alignment, at
5985 least 4096), or if mmap is unavailable or fails, a 256-bytes aligned
5986 malloc is done and the file is loaded.
5988 Some variables are reinitialized from the values found in the header.
5990 The difference between the actual loading address and the reloc_address
5991 is computed and will be used for all the relocations.
5994 @subsection Putting back the staticvec
5996 The staticvec array is memcpy'd from the file and the variables it
5997 points to are reset to the relocated objects addresses.
6000 @subsection Putting back the dumpstructed variables
6002 The variables pointed to by dumpstruct in the dump phase are reset to
6003 the right relocated object addresses.
6006 @subsection lrecord_implementations_table
6008 The lrecord_implementations_table is reset to its dump time state and
6009 the right lrecord_type_index values are put in.
6012 @subsection Object relocation
6014 All the objects are relocated using their description and their offset
6015 by @code{pdump_reloc_one}. This step is unnecessary if the
6016 reloc_address is equal to the file loading address.
6019 @subsection Putting back the pdump_wire and pdump_wire_list variables
6021 Same as Putting back the dumpstructed variables.
6024 @subsection Reorganize the hash tables
6026 Since some of the hash values in the lisp hash tables are
6027 address-dependent, their layout is now wrong. So we go through each of
6028 them and have them resorted by calling @code{pdump_reorganize_hash_table}.
6030 @node Remaining issues, , Reloading phase, Dumping
6031 @section Remaining issues
6033 The build process will have to start a post-dump xemacs, ask it the
6034 loading address (which will, hopefully, be always the same between
6035 different xemacs invocations) and relocate the file to the new address.
6036 This way the object relocation phase will not have to be done, which
6037 means no writes in the objects and that, because of the use of mmap, the
6038 dumped data will be shared between all the xemacs running on the
6041 Some executable signature will be necessary to ensure that a given dump
6042 file is really associated with a given executable, or random crashes
6043 will occur. Maybe a random number set at compile or configure time thru
6044 a define. This will also allow for having differently-compiled xemacsen
6045 on the same system (mule and no-mule comes to mind).
6047 The DOC file contents should probably end up in the dump file.
6050 @node Events and the Event Loop, Evaluation; Stack Frames; Bindings, Dumping, Top
6051 @chapter Events and the Event Loop
6054 * Introduction to Events::
6056 * Specifics of the Event Gathering Mechanism::
6057 * Specifics About the Emacs Event::
6058 * The Event Stream Callback Routines::
6059 * Other Event Loop Functions::
6060 * Converting Events::
6061 * Dispatching Events; The Command Builder::
6064 @node Introduction to Events, Main Loop, Events and the Event Loop, Events and the Event Loop
6065 @section Introduction to Events
6067 An event is an object that encapsulates information about an
6068 interesting occurrence in the operating system. Events are
6069 generated either by user action, direct (e.g. typing on the
6070 keyboard or moving the mouse) or indirect (moving another
6071 window, thereby generating an expose event on an Emacs frame),
6072 or as a result of some other typically asynchronous action happening,
6073 such as output from a subprocess being ready or a timer expiring.
6074 Events come into the system in an asynchronous fashion (typically
6075 through a callback being called) and are converted into a
6076 synchronous event queue (first-in, first-out) in a process that
6077 we will call @dfn{collection}.
6079 Note that each application has its own event queue. (It is
6080 immaterial whether the collection process directly puts the
6081 events in the proper application's queue, or puts them into
6082 a single system queue, which is later split up.)
6084 The most basic level of event collection is done by the
6085 operating system or window system. Typically, XEmacs does
6086 its own event collection as well. Often there are multiple
6087 layers of collection in XEmacs, with events from various
6088 sources being collected into a queue, which is then combined
6089 with other sources to go into another queue (i.e. a second
6090 level of collection), with perhaps another level on top of
6093 XEmacs has its own types of events (called @dfn{Emacs events}),
6094 which provides an abstract layer on top of the system-dependent
6095 nature of the most basic events that are received. Part of the
6096 complex nature of the XEmacs event collection process involves
6097 converting from the operating-system events into the proper
6098 Emacs events---there may not be a one-to-one correspondence.
6100 Emacs events are documented in @file{events.h}; I'll discuss them
6103 @node Main Loop, Specifics of the Event Gathering Mechanism, Introduction to Events, Events and the Event Loop
6106 The @dfn{command loop} is the top-level loop that the editor is always
6107 running. It loops endlessly, calling @code{next-event} to retrieve an
6108 event and @code{dispatch-event} to execute it. @code{dispatch-event} does
6109 the appropriate thing with non-user events (process, timeout,
6110 magic, eval, mouse motion); this involves calling a Lisp handler
6111 function, redrawing a newly-exposed part of a frame, reading
6112 subprocess output, etc. For user events, @code{dispatch-event}
6113 looks up the event in relevant keymaps or menubars; when a
6114 full key sequence or menubar selection is reached, the appropriate
6115 function is executed. @code{dispatch-event} may have to keep state
6116 across calls; this is done in the ``command-builder'' structure
6117 associated with each console (remember, there's usually only
6118 one console), and the engine that looks up keystrokes and
6119 constructs full key sequences is called the @dfn{command builder}.
6120 This is documented elsewhere.
6122 The guts of the command loop are in @code{command_loop_1()}. This
6123 function doesn't catch errors, though---that's the job of
6124 @code{command_loop_2()}, which is a condition-case (i.e. error-trapping)
6125 wrapper around @code{command_loop_1()}. @code{command_loop_1()} never
6126 returns, but may get thrown out of.
6128 When an error occurs, @code{cmd_error()} is called, which usually
6129 invokes the Lisp error handler in @code{command-error}; however, a
6130 default error handler is provided if @code{command-error} is @code{nil}
6131 (e.g. during startup). The purpose of the error handler is simply to
6132 display the error message and do associated cleanup; it does not need to
6133 throw anywhere. When the error handler finishes, the condition-case in
6134 @code{command_loop_2()} will finish and @code{command_loop_2()} will
6135 reinvoke @code{command_loop_1()}.
6137 @code{command_loop_2()} is invoked from three places: from
6138 @code{initial_command_loop()} (called from @code{main()} at the end of
6139 internal initialization), from the Lisp function @code{recursive-edit},
6140 and from @code{call_command_loop()}.
6142 @code{call_command_loop()} is called when a macro is started and when
6143 the minibuffer is entered; normal termination of the macro or minibuffer
6144 causes a throw out of the recursive command loop. (To
6145 @code{execute-kbd-macro} for macros and @code{exit} for minibuffers.
6146 Note also that the low-level minibuffer-entering function,
6147 @code{read-minibuffer-internal}, provides its own error handling and
6148 does not need @code{command_loop_2()}'s error encapsulation; so it tells
6149 @code{call_command_loop()} to invoke @code{command_loop_1()} directly.)
6151 Note that both read-minibuffer-internal and recursive-edit set up a
6152 catch for @code{exit}; this is why @code{abort-recursive-edit}, which
6153 throws to this catch, exits out of either one.
6155 @code{initial_command_loop()}, called from @code{main()}, sets up a
6156 catch for @code{top-level} when invoking @code{command_loop_2()},
6157 allowing functions to throw all the way to the top level if they really
6158 need to. Before invoking @code{command_loop_2()},
6159 @code{initial_command_loop()} calls @code{top_level_1()}, which handles
6160 all of the startup stuff (creating the initial frame, handling the
6161 command-line options, loading the user's @file{.emacs} file, etc.). The
6162 function that actually does this is in Lisp and is pointed to by the
6163 variable @code{top-level}; normally this function is
6164 @code{normal-top-level}. @code{top_level_1()} is just an error-handling
6165 wrapper similar to @code{command_loop_2()}. Note also that
6166 @code{initial_command_loop()} sets up a catch for @code{top-level} when
6167 invoking @code{top_level_1()}, just like when it invokes
6168 @code{command_loop_2()}.
6170 @node Specifics of the Event Gathering Mechanism, Specifics About the Emacs Event, Main Loop, Events and the Event Loop
6171 @section Specifics of the Event Gathering Mechanism
6173 Here is an approximate diagram of the collection processes
6174 at work in XEmacs, under TTY's (TTY's are simpler than X
6175 so we'll look at this first):
6179 asynch. asynch. asynch. asynch. [Collectors in
6180 kbd events kbd events process process the OS]
6183 | | | | SIGINT, [signal handlers
6184 | | | | SIGQUIT, in XEmacs]
6186 file file file file SIGALRM
6187 desc. desc. desc. desc. |
6188 (TTY) (TTY) (pipe) (pipe) |
6189 | | | | fake timeouts
6197 ------>-----------<----------------<----------------
6200 | [collected using select() in emacs_tty_next_event()
6201 | and converted to the appropriate Emacs event]
6204 V (above this line is TTY-specific)
6205 Emacs -----------------------------------------------
6206 event (below this line is the generic event mechanism)
6209 was there if not, call
6210 a SIGINT? emacs_tty_next_event()
6217 | [collected in event_stream_next_event();
6218 | SIGINT is converted using maybe_read_quit_event()]
6223 \---->------>----- maybe_kbd_translate() ---->---\
6227 command event queue |
6229 (contains events that were event queue, call
6230 read earlier but not processed, event_stream_next_event()
6231 typically when waiting in a |
6232 sit-for, sleep-for, etc. for |
6233 a particular event to be received) |
6237 ---->------------------------------------<----
6240 | next_event_internal()]
6242 unread- unread- event from |
6243 command- command- keyboard else, call
6244 events event macro next_event_internal()
6249 --------->----------------------<------------
6251 | [collected in `next-event', which may loop
6252 | more than once if the event it gets is on
6253 | a dead frame, device, etc.]
6257 feed into top-level event loop,
6258 which repeatedly calls `next-event'
6259 and then dispatches the event
6260 using `dispatch-event'
6263 Notice the separation between TTY-specific and generic event mechanism.
6264 When using the Xt-based event loop, the TTY-specific stuff is replaced
6265 but the rest stays the same.
6267 It's also important to realize that only one different kind of
6268 system-specific event loop can be operating at a time, and must be able
6269 to receive all kinds of events simultaneously. For the two existing
6270 event loops (implemented in @file{event-tty.c} and @file{event-Xt.c},
6271 respectively), the TTY event loop @emph{only} handles TTY consoles,
6272 while the Xt event loop handles @emph{both} TTY and X consoles. This
6273 situation is different from all of the output handlers, where you simply
6274 have one per console type.
6276 Here's the Xt Event Loop Diagram (notice that below a certain point,
6277 it's the same as the above diagram):
6280 asynch. asynch. asynch. asynch. [Collectors in
6281 kbd kbd process process the OS]
6282 events events output output
6284 | | | | asynch. asynch. [Collectors in the
6285 | | | | X X OS and X Window System]
6286 | | | | events events
6289 | | | | | | SIGINT, [signal handlers
6290 | | | | | | SIGQUIT, in XEmacs]
6291 | | | | | | SIGWINCH,
6295 | | | | | | | timeouts
6300 file file file file file file file |
6301 desc. desc. desc. desc. desc. desc. desc. |
6302 (TTY) (TTY) (pipe) (pipe) (socket) (socket) (pipe) |
6307 --->----------------------------------------<---------<------
6309 | | |[collected using select() in
6310 | | | _XtWaitForSomething(), called
6311 | | | from XtAppProcessEvent(), called
6312 | | | in emacs_Xt_next_event();
6313 | | | dispatched to various callbacks]
6316 emacs_Xt_ p_s_callback(), | [popup_selection_callback]
6317 event_handler() x_u_v_s_callback(),| [x_update_vertical_scrollbar_
6318 | x_u_h_s_callback(),| callback]
6319 | search_callback() | [x_update_horizontal_scrollbar_
6323 enqueue_Xt_ signal_special_ |
6324 dispatch_event() Xt_user_event() |
6329 | dispatch_event() |
6336 dispatch Xt_what_callback()
6343 ---->-----------<--------
6346 | [collected and converted as appropriate in
6347 | emacs_Xt_next_event()]
6350 V (above this line is Xt-specific)
6351 Emacs ------------------------------------------------
6352 event (below this line is the generic event mechanism)
6355 was there if not, call
6356 a SIGINT? emacs_Xt_next_event()
6363 | [collected in event_stream_next_event();
6364 | SIGINT is converted using maybe_read_quit_event()]
6369 \---->------>----- maybe_kbd_translate() -->-----\
6373 command event queue |
6375 (contains events that were event queue, call
6376 read earlier but not processed, event_stream_next_event()
6377 typically when waiting in a |
6378 sit-for, sleep-for, etc. for |
6379 a particular event to be received) |
6383 ---->----------------------------------<------
6386 | next_event_internal()]
6388 unread- unread- event from |
6389 command- command- keyboard else, call
6390 events event macro next_event_internal()
6395 --------->----------------------<------------
6397 | [collected in `next-event', which may loop
6398 | more than once if the event it gets is on
6399 | a dead frame, device, etc.]
6403 feed into top-level event loop,
6404 which repeatedly calls `next-event'
6405 and then dispatches the event
6406 using `dispatch-event'
6409 @node Specifics About the Emacs Event, The Event Stream Callback Routines, Specifics of the Event Gathering Mechanism, Events and the Event Loop
6410 @section Specifics About the Emacs Event
6412 @node The Event Stream Callback Routines, Other Event Loop Functions, Specifics About the Emacs Event, Events and the Event Loop
6413 @section The Event Stream Callback Routines
6415 @node Other Event Loop Functions, Converting Events, The Event Stream Callback Routines, Events and the Event Loop
6416 @section Other Event Loop Functions
6418 @code{detect_input_pending()} and @code{input-pending-p} look for
6419 input by calling @code{event_stream->event_pending_p} and looking in
6420 @code{[V]unread-command-event} and the @code{command_event_queue} (they
6421 do not check for an executing keyboard macro, though).
6423 @code{discard-input} cancels any command events pending (and any
6424 keyboard macros currently executing), and puts the others onto the
6425 @code{command_event_queue}. There is a comment about a ``race
6426 condition'', which is not a good sign.
6428 @code{next-command-event} and @code{read-char} are higher-level
6429 interfaces to @code{next-event}. @code{next-command-event} gets the
6430 next @dfn{command} event (i.e. keypress, mouse event, menu selection,
6431 or scrollbar action), calling @code{dispatch-event} on any others.
6432 @code{read-char} calls @code{next-command-event} and uses
6433 @code{event_to_character()} to return the character equivalent. With
6434 the right kind of input method support, it is possible for (read-char)
6435 to return a Kanji character.
6437 @node Converting Events, Dispatching Events; The Command Builder, Other Event Loop Functions, Events and the Event Loop
6438 @section Converting Events
6440 @code{character_to_event()}, @code{event_to_character()},
6441 @code{event-to-character}, and @code{character-to-event} convert between
6442 characters and keypress events corresponding to the characters. If the
6443 event was not a keypress, @code{event_to_character()} returns -1 and
6444 @code{event-to-character} returns @code{nil}. These functions convert
6445 between character representation and the split-up event representation
6446 (keysym plus mod keys).
6448 @node Dispatching Events; The Command Builder, , Converting Events, Events and the Event Loop
6449 @section Dispatching Events; The Command Builder
6453 @node Evaluation; Stack Frames; Bindings, Symbols and Variables, Events and the Event Loop, Top
6454 @chapter Evaluation; Stack Frames; Bindings
6458 * Dynamic Binding; The specbinding Stack; Unwind-Protects::
6459 * Simple Special Forms::
6463 @node Evaluation, Dynamic Binding; The specbinding Stack; Unwind-Protects, Evaluation; Stack Frames; Bindings, Evaluation; Stack Frames; Bindings
6466 @code{Feval()} evaluates the form (a Lisp object) that is passed to
6467 it. Note that evaluation is only non-trivial for two types of objects:
6468 symbols and conses. A symbol is evaluated simply by calling
6469 @code{symbol-value} on it and returning the value.
6471 Evaluating a cons means calling a function. First, @code{eval} checks
6472 to see if garbage-collection is necessary, and calls
6473 @code{garbage_collect_1()} if so. It then increases the evaluation
6474 depth by 1 (@code{lisp_eval_depth}, which is always less than
6475 @code{max_lisp_eval_depth}) and adds an element to the linked list of
6476 @code{struct backtrace}'s (@code{backtrace_list}). Each such structure
6477 contains a pointer to the function being called plus a list of the
6478 function's arguments. Originally these values are stored unevalled, and
6479 as they are evaluated, the backtrace structure is updated. Garbage
6480 collection pays attention to the objects pointed to in the backtrace
6481 structures (garbage collection might happen while a function is being
6482 called or while an argument is being evaluated, and there could easily
6483 be no other references to the arguments in the argument list; once an
6484 argument is evaluated, however, the unevalled version is not needed by
6485 eval, and so the backtrace structure is changed).
6487 At this point, the function to be called is determined by looking at
6488 the car of the cons (if this is a symbol, its function definition is
6489 retrieved and the process repeated). The function should then consist
6490 of either a @code{Lisp_Subr} (built-in function written in C), a
6491 @code{Lisp_Compiled_Function} object, or a cons whose car is one of the
6492 symbols @code{autoload}, @code{macro} or @code{lambda}.
6494 If the function is a @code{Lisp_Subr}, the lisp object points to a
6495 @code{struct Lisp_Subr} (created by @code{DEFUN()}), which contains a
6496 pointer to the C function, a minimum and maximum number of arguments
6497 (or possibly the special constants @code{MANY} or @code{UNEVALLED}), a
6498 pointer to the symbol referring to that subr, and a couple of other
6499 things. If the subr wants its arguments @code{UNEVALLED}, they are
6500 passed raw as a list. Otherwise, an array of evaluated arguments is
6501 created and put into the backtrace structure, and either passed whole
6502 (@code{MANY}) or each argument is passed as a C argument.
6504 If the function is a @code{Lisp_Compiled_Function},
6505 @code{funcall_compiled_function()} is called. If the function is a
6506 lambda list, @code{funcall_lambda()} is called. If the function is a
6507 macro, [..... fill in] is done. If the function is an autoload,
6508 @code{do_autoload()} is called to load the definition and then eval
6509 starts over [explain this more].
6511 When @code{Feval()} exits, the evaluation depth is reduced by one, the
6512 debugger is called if appropriate, and the current backtrace structure
6513 is removed from the list.
6515 Both @code{funcall_compiled_function()} and @code{funcall_lambda()} need
6516 to go through the list of formal parameters to the function and bind
6517 them to the actual arguments, checking for @code{&rest} and
6518 @code{&optional} symbols in the formal parameters and making sure the
6519 number of actual arguments is correct.
6520 @code{funcall_compiled_function()} can do this a little more
6521 efficiently, since the formal parameter list can be checked for sanity
6522 when the compiled function object is created.
6524 @code{funcall_lambda()} simply calls @code{Fprogn} to execute the code
6527 @code{funcall_compiled_function()} calls the real byte-code interpreter
6528 @code{execute_optimized_program()} on the byte-code instructions, which
6529 are converted into an internal form for faster execution.
6531 When a compiled function is executed for the first time by
6532 @code{funcall_compiled_function()}, or during the dump phase of building
6533 XEmacs, the byte-code instructions are converted from a
6534 @code{Lisp_String} (which is inefficient to access, especially in the
6535 presence of MULE) into a @code{Lisp_Opaque} object containing an array
6536 of unsigned char, which can be directly executed by the byte-code
6537 interpreter. At this time the byte code is also analyzed for validity
6538 and transformed into a more optimized form, so that
6539 @code{execute_optimized_program()} can really fly.
6541 Here are some of the optimizations performed by the internal byte-code
6545 References to the @code{constants} array are checked for out-of-range
6546 indices, so that the byte interpreter doesn't have to.
6548 References to the @code{constants} array that will be used as a Lisp
6549 variable are checked for being correct non-constant (i.e. not @code{t},
6550 @code{nil}, or @code{keywordp}) symbols, so that the byte interpreter
6553 The maximum number of variable bindings in the byte-code is
6554 pre-computed, so that space on the @code{specpdl} stack can be
6555 pre-reserved once for the whole function execution.
6557 All byte-code jumps are relative to the current program counter instead
6558 of the start of the program, thereby saving a register.
6560 One-byte relative jumps are converted from the byte-code form of unsigned
6561 chars offset by 127 to machine-friendly signed chars.
6564 Of course, this transformation of the @code{instructions} should not be
6565 visible to the user, so @code{Fcompiled_function_instructions()} needs
6566 to know how to convert the optimized opaque object back into a Lisp
6567 string that is identical to the original string from the @file{.elc}
6568 file. (Actually, the resulting string may (rarely) contain slightly
6569 different, yet equivalent, byte code.)
6571 @code{Ffuncall()} implements Lisp @code{funcall}. @code{(funcall fun
6572 x1 x2 x3 ...)} is equivalent to @code{(eval (list fun (quote x1) (quote
6573 x2) (quote x3) ...))}. @code{Ffuncall()} contains its own code to do
6574 the evaluation, however, and is very similar to @code{Feval()}.
6576 From the performance point of view, it is worth knowing that most of the
6577 time in Lisp evaluation is spent executing @code{Lisp_Subr} and
6578 @code{Lisp_Compiled_Function} objects via @code{Ffuncall()} (not
6581 @code{Fapply()} implements Lisp @code{apply}, which is very similar to
6582 @code{funcall} except that if the last argument is a list, the result is the
6583 same as if each of the arguments in the list had been passed separately.
6584 @code{Fapply()} does some business to expand the last argument if it's a
6585 list, then calls @code{Ffuncall()} to do the work.
6587 @code{apply1()}, @code{call0()}, @code{call1()}, @code{call2()}, and
6588 @code{call3()} call a function, passing it the argument(s) given (the
6589 arguments are given as separate C arguments rather than being passed as
6590 an array). @code{apply1()} uses @code{Fapply()} while the others use
6591 @code{Ffuncall()} to do the real work.
6593 @node Dynamic Binding; The specbinding Stack; Unwind-Protects, Simple Special Forms, Evaluation, Evaluation; Stack Frames; Bindings
6594 @section Dynamic Binding; The specbinding Stack; Unwind-Protects
6600 Lisp_Object old_value;
6601 Lisp_Object (*func) (Lisp_Object); /* for unwind-protect */
6605 @code{struct specbinding} is used for local-variable bindings and
6606 unwind-protects. @code{specpdl} holds an array of @code{struct specbinding}'s,
6607 @code{specpdl_ptr} points to the beginning of the free bindings in the
6608 array, @code{specpdl_size} specifies the total number of binding slots
6609 in the array, and @code{max_specpdl_size} specifies the maximum number
6610 of bindings the array can be expanded to hold. @code{grow_specpdl()}
6611 increases the size of the @code{specpdl} array, multiplying its size by
6612 2 but never exceeding @code{max_specpdl_size} (except that if this
6613 number is less than 400, it is first set to 400).
6615 @code{specbind()} binds a symbol to a value and is used for local
6616 variables and @code{let} forms. The symbol and its old value (which
6617 might be @code{Qunbound}, indicating no prior value) are recorded in the
6618 specpdl array, and @code{specpdl_size} is increased by 1.
6620 @code{record_unwind_protect()} implements an @dfn{unwind-protect},
6621 which, when placed around a section of code, ensures that some specified
6622 cleanup routine will be executed even if the code exits abnormally
6623 (e.g. through a @code{throw} or quit). @code{record_unwind_protect()}
6624 simply adds a new specbinding to the @code{specpdl} array and stores the
6625 appropriate information in it. The cleanup routine can either be a C
6626 function, which is stored in the @code{func} field, or a @code{progn}
6627 form, which is stored in the @code{old_value} field.
6629 @code{unbind_to()} removes specbindings from the @code{specpdl} array
6630 until the specified position is reached. Each specbinding can be one of
6635 an unwind-protect with a C cleanup function (@code{func} is not 0, and
6636 @code{old_value} holds an argument to be passed to the function);
6638 an unwind-protect with a Lisp form (@code{func} is 0, @code{symbol}
6639 is @code{nil}, and @code{old_value} holds the form to be executed with
6640 @code{Fprogn()}); or
6642 a local-variable binding (@code{func} is 0, @code{symbol} is not
6643 @code{nil}, and @code{old_value} holds the old value, which is stored as
6644 the symbol's value).
6647 @node Simple Special Forms, Catch and Throw, Dynamic Binding; The specbinding Stack; Unwind-Protects, Evaluation; Stack Frames; Bindings
6648 @section Simple Special Forms
6650 @code{or}, @code{and}, @code{if}, @code{cond}, @code{progn},
6651 @code{prog1}, @code{prog2}, @code{setq}, @code{quote}, @code{function},
6652 @code{let*}, @code{let}, @code{while}
6654 All of these are very simple and work as expected, calling
6655 @code{Feval()} or @code{Fprogn()} as necessary and (in the case of
6656 @code{let} and @code{let*}) using @code{specbind()} to create bindings
6657 and @code{unbind_to()} to undo the bindings when finished.
6659 Note that, with the exception of @code{Fprogn}, these functions are
6660 typically called in real life only in interpreted code, since the byte
6661 compiler knows how to convert calls to these functions directly into
6664 @node Catch and Throw, , Simple Special Forms, Evaluation; Stack Frames; Bindings
6665 @section Catch and Throw
6672 struct catchtag *next;
6673 struct gcpro *gcpro;
6675 struct backtrace *backlist;
6676 int lisp_eval_depth;
6681 @code{catch} is a Lisp function that places a catch around a body of
6682 code. A catch is a means of non-local exit from the code. When a catch
6683 is created, a tag is specified, and executing a @code{throw} to this tag
6684 will exit from the body of code caught with this tag, and its value will
6685 be the value given in the call to @code{throw}. If there is no such
6686 call, the code will be executed normally.
6688 Information pertaining to a catch is held in a @code{struct catchtag},
6689 which is placed at the head of a linked list pointed to by
6690 @code{catchlist}. @code{internal_catch()} is passed a C function to
6691 call (@code{Fprogn()} when Lisp @code{catch} is called) and arguments to
6692 give it, and places a catch around the function. Each @code{struct
6693 catchtag} is held in the stack frame of the @code{internal_catch()}
6694 instance that created the catch.
6696 @code{internal_catch()} is fairly straightforward. It stores into the
6697 @code{struct catchtag} the tag name and the current values of
6698 @code{backtrace_list}, @code{lisp_eval_depth}, @code{gcprolist}, and the
6699 offset into the @code{specpdl} array, sets a jump point with @code{_setjmp()}
6700 (storing the jump point into the @code{struct catchtag}), and calls the
6701 function. Control will return to @code{internal_catch()} either when
6702 the function exits normally or through a @code{_longjmp()} to this jump
6703 point. In the latter case, @code{throw} will store the value to be
6704 returned into the @code{struct catchtag} before jumping. When it's
6705 done, @code{internal_catch()} removes the @code{struct catchtag} from
6706 the catchlist and returns the proper value.
6708 @code{Fthrow()} goes up through the catchlist until it finds one with
6709 a matching tag. It then calls @code{unbind_catch()} to restore
6710 everything to what it was when the appropriate catch was set, stores the
6711 return value in the @code{struct catchtag}, and jumps (with
6712 @code{_longjmp()}) to its jump point.
6714 @code{unbind_catch()} removes all catches from the catchlist until it
6715 finds the correct one. Some of the catches might have been placed for
6716 error-trapping, and if so, the appropriate entries on the handlerlist
6717 must be removed (see ``errors''). @code{unbind_catch()} also restores
6718 the values of @code{gcprolist}, @code{backtrace_list}, and
6719 @code{lisp_eval}, and calls @code{unbind_to()} to undo any specbindings
6720 created since the catch.
6723 @node Symbols and Variables, Buffers and Textual Representation, Evaluation; Stack Frames; Bindings, Top
6724 @chapter Symbols and Variables
6727 * Introduction to Symbols::
6732 @node Introduction to Symbols, Obarrays, Symbols and Variables, Symbols and Variables
6733 @section Introduction to Symbols
6735 A symbol is basically just an object with four fields: a name (a
6736 string), a value (some Lisp object), a function (some Lisp object), and
6737 a property list (usually a list of alternating keyword/value pairs).
6738 What makes symbols special is that there is usually only one symbol with
6739 a given name, and the symbol is referred to by name. This makes a
6740 symbol a convenient way of calling up data by name, i.e. of implementing
6741 variables. (The variable's value is stored in the @dfn{value slot}.)
6742 Similarly, functions are referenced by name, and the definition of the
6743 function is stored in a symbol's @dfn{function slot}. This means that
6744 there can be a distinct function and variable with the same name. The
6745 property list is used as a more general mechanism of associating
6746 additional values with particular names, and once again the namespace is
6747 independent of the function and variable namespaces.
6749 @node Obarrays, Symbol Values, Introduction to Symbols, Symbols and Variables
6752 The identity of symbols with their names is accomplished through a
6753 structure called an obarray, which is just a poorly-implemented hash
6754 table mapping from strings to symbols whose name is that string. (I say
6755 ``poorly implemented'' because an obarray appears in Lisp as a vector
6756 with some hidden fields rather than as its own opaque type. This is an
6757 Emacs Lisp artifact that should be fixed.)
6759 Obarrays are implemented as a vector of some fixed size (which should
6760 be a prime for best results), where each ``bucket'' of the vector
6761 contains one or more symbols, threaded through a hidden @code{next}
6762 field in the symbol. Lookup of a symbol in an obarray, and adding a
6763 symbol to an obarray, is accomplished through standard hash-table
6766 The standard Lisp function for working with symbols and obarrays is
6767 @code{intern}. This looks up a symbol in an obarray given its name; if
6768 it's not found, a new symbol is automatically created with the specified
6769 name, added to the obarray, and returned. This is what happens when the
6770 Lisp reader encounters a symbol (or more precisely, encounters the name
6771 of a symbol) in some text that it is reading. There is a standard
6772 obarray called @code{obarray} that is used for this purpose, although
6773 the Lisp programmer is free to create his own obarrays and @code{intern}
6776 Note that, once a symbol is in an obarray, it stays there until
6777 something is done about it, and the standard obarray @code{obarray}
6778 always stays around, so once you use any particular variable name, a
6779 corresponding symbol will stay around in @code{obarray} until you exit
6782 Note that @code{obarray} itself is a variable, and as such there is a
6783 symbol in @code{obarray} whose name is @code{"obarray"} and which
6784 contains @code{obarray} as its value.
6786 Note also that this call to @code{intern} occurs only when in the Lisp
6787 reader, not when the code is executed (at which point the symbol is
6788 already around, stored as such in the definition of the function).
6790 You can create your own obarray using @code{make-vector} (this is
6791 horrible but is an artifact) and intern symbols into that obarray.
6792 Doing that will result in two or more symbols with the same name.
6793 However, at most one of these symbols is in the standard @code{obarray}:
6794 You cannot have two symbols of the same name in any particular obarray.
6795 Note that you cannot add a symbol to an obarray in any fashion other
6796 than using @code{intern}: i.e. you can't take an existing symbol and put
6797 it in an existing obarray. Nor can you change the name of an existing
6798 symbol. (Since obarrays are vectors, you can violate the consistency of
6799 things by storing directly into the vector, but let's ignore that
6802 Usually symbols are created by @code{intern}, but if you really want,
6803 you can explicitly create a symbol using @code{make-symbol}, giving it
6804 some name. The resulting symbol is not in any obarray (i.e. it is
6805 @dfn{uninterned}), and you can't add it to any obarray. Therefore its
6806 primary purpose is as a symbol to use in macros to avoid namespace
6807 pollution. It can also be used as a carrier of information, but cons
6808 cells could probably be used just as well.
6810 You can also use @code{intern-soft} to look up a symbol but not create
6811 a new one, and @code{unintern} to remove a symbol from an obarray. This
6812 returns the removed symbol. (Remember: You can't put the symbol back
6813 into any obarray.) Finally, @code{mapatoms} maps over all of the symbols
6816 @node Symbol Values, , Obarrays, Symbols and Variables
6817 @section Symbol Values
6819 The value field of a symbol normally contains a Lisp object. However,
6820 a symbol can be @dfn{unbound}, meaning that it logically has no value.
6821 This is internally indicated by storing a special Lisp object, called
6822 @dfn{the unbound marker} and stored in the global variable
6823 @code{Qunbound}. The unbound marker is of a special Lisp object type
6824 called @dfn{symbol-value-magic}. It is impossible for the Lisp
6825 programmer to directly create or access any object of this type.
6827 @strong{You must not let any ``symbol-value-magic'' object escape to
6828 the Lisp level.} Printing any of these objects will cause the message
6829 @samp{INTERNAL EMACS BUG} to appear as part of the print representation.
6830 (You may see this normally when you call @code{debug_print()} from the
6831 debugger on a Lisp object.) If you let one of these objects escape to
6832 the Lisp level, you will violate a number of assumptions contained in
6833 the C code and make the unbound marker not function right.
6835 When a symbol is created, its value field (and function field) are set
6836 to @code{Qunbound}. The Lisp programmer can restore these conditions
6837 later using @code{makunbound} or @code{fmakunbound}, and can query to
6838 see whether the value of function fields are @dfn{bound} (i.e. have a
6839 value other than @code{Qunbound}) using @code{boundp} and
6840 @code{fboundp}. The fields are set to a normal Lisp object using
6841 @code{set} (or @code{setq}) and @code{fset}.
6843 Other symbol-value-magic objects are used as special markers to
6844 indicate variables that have non-normal properties. This includes any
6845 variables that are tied into C variables (setting the variable magically
6846 sets some global variable in the C code, and likewise for retrieving the
6847 variable's value), variables that magically tie into slots in the
6848 current buffer, variables that are buffer-local, etc. The
6849 symbol-value-magic object is stored in the value cell in place of
6850 a normal object, and the code to retrieve a symbol's value
6851 (i.e. @code{symbol-value}) knows how to do special things with them.
6852 This means that you should not just fetch the value cell directly if you
6853 want a symbol's value.
6855 The exact workings of this are rather complex and involved and are
6856 well-documented in comments in @file{buffer.c}, @file{symbols.c}, and
6859 @node Buffers and Textual Representation, MULE Character Sets and Encodings, Symbols and Variables, Top
6860 @chapter Buffers and Textual Representation
6863 * Introduction to Buffers:: A buffer holds a block of text such as a file.
6864 * The Text in a Buffer:: Representation of the text in a buffer.
6865 * Buffer Lists:: Keeping track of all buffers.
6866 * Markers and Extents:: Tagging locations within a buffer.
6867 * Bufbytes and Emchars:: Representation of individual characters.
6868 * The Buffer Object:: The Lisp object corresponding to a buffer.
6871 @node Introduction to Buffers, The Text in a Buffer, Buffers and Textual Representation, Buffers and Textual Representation
6872 @section Introduction to Buffers
6874 A buffer is logically just a Lisp object that holds some text.
6875 In this, it is like a string, but a buffer is optimized for
6876 frequent insertion and deletion, while a string is not. Furthermore:
6880 Buffers are @dfn{permanent} objects, i.e. once you create them, they
6881 remain around, and need to be explicitly deleted before they go away.
6883 Each buffer has a unique name, which is a string. Buffers are
6884 normally referred to by name. In this respect, they are like
6887 Buffers have a default insertion position, called @dfn{point}.
6888 Inserting text (unless you explicitly give a position) goes at point,
6889 and moves point forward past the text. This is what is going on when
6890 you type text into Emacs.
6892 Buffers have lots of extra properties associated with them.
6894 Buffers can be @dfn{displayed}. What this means is that there
6895 exist a number of @dfn{windows}, which are objects that correspond
6896 to some visible section of your display, and each window has
6897 an associated buffer, and the current contents of the buffer
6898 are shown in that section of the display. The redisplay mechanism
6899 (which takes care of doing this) knows how to look at the
6900 text of a buffer and come up with some reasonable way of displaying
6901 this. Many of the properties of a buffer control how the
6902 buffer's text is displayed.
6904 One buffer is distinguished and called the @dfn{current buffer}. It is
6905 stored in the variable @code{current_buffer}. Buffer operations operate
6906 on this buffer by default. When you are typing text into a buffer, the
6907 buffer you are typing into is always @code{current_buffer}. Switching
6908 to a different window changes the current buffer. Note that Lisp code
6909 can temporarily change the current buffer using @code{set-buffer} (often
6910 enclosed in a @code{save-excursion} so that the former current buffer
6911 gets restored when the code is finished). However, calling
6912 @code{set-buffer} will NOT cause a permanent change in the current
6913 buffer. The reason for this is that the top-level event loop sets
6914 @code{current_buffer} to the buffer of the selected window, each time
6915 it finishes executing a user command.
6918 Make sure you understand the distinction between @dfn{current buffer}
6919 and @dfn{buffer of the selected window}, and the distinction between
6920 @dfn{point} of the current buffer and @dfn{window-point} of the selected
6921 window. (This latter distinction is explained in detail in the section
6924 @node The Text in a Buffer, Buffer Lists, Introduction to Buffers, Buffers and Textual Representation
6925 @section The Text in a Buffer
6927 The text in a buffer consists of a sequence of zero or more
6928 characters. A @dfn{character} is an integer that logically represents
6929 a letter, number, space, or other unit of text. Most of the characters
6930 that you will typically encounter belong to the ASCII set of characters,
6931 but there are also characters for various sorts of accented letters,
6932 special symbols, Chinese and Japanese ideograms (i.e. Kanji, Katakana,
6933 etc.), Cyrillic and Greek letters, etc. The actual number of possible
6934 characters is quite large.
6936 For now, we can view a character as some non-negative integer that
6937 has some shape that defines how it typically appears (e.g. as an
6938 uppercase A). (The exact way in which a character appears depends on the
6939 font used to display the character.) The internal type of characters in
6940 the C code is an @code{Emchar}; this is just an @code{int}, but using a
6941 symbolic type makes the code clearer.
6943 Between every character in a buffer is a @dfn{buffer position} or
6944 @dfn{character position}. We can speak of the character before or after
6945 a particular buffer position, and when you insert a character at a
6946 particular position, all characters after that position end up at new
6947 positions. When we speak of the character @dfn{at} a position, we
6948 really mean the character after the position. (This schizophrenia
6949 between a buffer position being ``between'' a character and ``on'' a
6950 character is rampant in Emacs.)
6952 Buffer positions are numbered starting at 1. This means that
6953 position 1 is before the first character, and position 0 is not
6954 valid. If there are N characters in a buffer, then buffer
6955 position N+1 is after the last one, and position N+2 is not valid.
6957 The internal makeup of the Emchar integer varies depending on whether
6958 we have compiled with MULE support. If not, the Emchar integer is an
6959 8-bit integer with possible values from 0 - 255. 0 - 127 are the
6960 standard ASCII characters, while 128 - 255 are the characters from the
6961 ISO-8859-1 character set. If we have compiled with MULE support, an
6962 Emchar is a 19-bit integer, with the various bits having meanings
6963 according to a complex scheme that will be detailed later. The
6964 characters numbered 0 - 255 still have the same meanings as for the
6965 non-MULE case, though.
6967 Internally, the text in a buffer is represented in a fairly simple
6968 fashion: as a contiguous array of bytes, with a @dfn{gap} of some size
6969 in the middle. Although the gap is of some substantial size in bytes,
6970 there is no text contained within it: From the perspective of the text
6971 in the buffer, it does not exist. The gap logically sits at some buffer
6972 position, between two characters (or possibly at the beginning or end of
6973 the buffer). Insertion of text in a buffer at a particular position is
6974 always accomplished by first moving the gap to that position
6975 (i.e. through some block moving of text), then writing the text into the
6976 beginning of the gap, thereby shrinking the gap. If the gap shrinks
6977 down to nothing, a new gap is created. (What actually happens is that a
6978 new gap is ``created'' at the end of the buffer's text, which requires
6979 nothing more than changing a couple of indices; then the gap is
6980 ``moved'' to the position where the insertion needs to take place by
6981 moving up in memory all the text after that position.) Similarly,
6982 deletion occurs by moving the gap to the place where the text is to be
6983 deleted, and then simply expanding the gap to include the deleted text.
6984 (@dfn{Expanding} and @dfn{shrinking} the gap as just described means
6985 just that the internal indices that keep track of where the gap is
6986 located are changed.)
6988 Note that the total amount of memory allocated for a buffer text never
6989 decreases while the buffer is live. Therefore, if you load up a
6990 20-megabyte file and then delete all but one character, there will be a
6991 20-megabyte gap, which won't get any smaller (except by inserting
6992 characters back again). Once the buffer is killed, the memory allocated
6993 for the buffer text will be freed, but it will still be sitting on the
6994 heap, taking up virtual memory, and will not be released back to the
6995 operating system. (However, if you have compiled XEmacs with rel-alloc,
6996 the situation is different. In this case, the space @emph{will} be
6997 released back to the operating system. However, this tends to result in a
6998 noticeable speed penalty.)
7000 Astute readers may notice that the text in a buffer is represented as
7001 an array of @emph{bytes}, while (at least in the MULE case) an Emchar is
7002 a 19-bit integer, which clearly cannot fit in a byte. This means (of
7003 course) that the text in a buffer uses a different representation from
7004 an Emchar: specifically, the 19-bit Emchar becomes a series of one to
7005 four bytes. The conversion between these two representations is complex
7006 and will be described later.
7008 In the non-MULE case, everything is very simple: An Emchar
7009 is an 8-bit value, which fits neatly into one byte.
7011 If we are given a buffer position and want to retrieve the
7012 character at that position, we need to follow these steps:
7016 Pretend there's no gap, and convert the buffer position into a @dfn{byte
7017 index} that indexes to the appropriate byte in the buffer's stream of
7018 textual bytes. By convention, byte indices begin at 1, just like buffer
7019 positions. In the non-MULE case, byte indices and buffer positions are
7020 identical, since one character equals one byte.
7022 Convert the byte index into a @dfn{memory index}, which takes the gap
7023 into account. The memory index is a direct index into the block of
7024 memory that stores the text of a buffer. This basically just involves
7025 checking to see if the byte index is past the gap, and if so, adding the
7026 size of the gap to it. By convention, memory indices begin at 1, just
7027 like buffer positions and byte indices, and when referring to the
7028 position that is @dfn{at} the gap, we always use the memory position at
7029 the @emph{beginning}, not at the end, of the gap.
7031 Fetch the appropriate bytes at the determined memory position.
7033 Convert these bytes into an Emchar.
7036 In the non-Mule case, (3) and (4) boil down to a simple one-byte
7039 Note that we have defined three types of positions in a buffer:
7043 @dfn{buffer positions} or @dfn{character positions}, typedef @code{Bufpos}
7045 @dfn{byte indices}, typedef @code{Bytind}
7047 @dfn{memory indices}, typedef @code{Memind}
7050 All three typedefs are just @code{int}s, but defining them this way makes
7051 things a lot clearer.
7053 Most code works with buffer positions. In particular, all Lisp code
7054 that refers to text in a buffer uses buffer positions. Lisp code does
7055 not know that byte indices or memory indices exist.
7057 Finally, we have a typedef for the bytes in a buffer. This is a
7058 @code{Bufbyte}, which is an unsigned char. Referring to them as
7059 Bufbytes underscores the fact that we are working with a string of bytes
7060 in the internal Emacs buffer representation rather than in one of a
7061 number of possible alternative representations (e.g. EUC-encoded text,
7064 @node Buffer Lists, Markers and Extents, The Text in a Buffer, Buffers and Textual Representation
7065 @section Buffer Lists
7067 Recall earlier that buffers are @dfn{permanent} objects, i.e. that
7068 they remain around until explicitly deleted. This entails that there is
7069 a list of all the buffers in existence. This list is actually an
7070 assoc-list (mapping from the buffer's name to the buffer) and is stored
7071 in the global variable @code{Vbuffer_alist}.
7073 The order of the buffers in the list is important: the buffers are
7074 ordered approximately from most-recently-used to least-recently-used.
7075 Switching to a buffer using @code{switch-to-buffer},
7076 @code{pop-to-buffer}, etc. and switching windows using
7077 @code{other-window}, etc. usually brings the new current buffer to the
7078 front of the list. @code{switch-to-buffer}, @code{other-buffer},
7079 etc. look at the beginning of the list to find an alternative buffer to
7080 suggest. You can also explicitly move a buffer to the end of the list
7081 using @code{bury-buffer}.
7083 In addition to the global ordering in @code{Vbuffer_alist}, each frame
7084 has its own ordering of the list. These lists always contain the same
7085 elements as in @code{Vbuffer_alist} although possibly in a different
7086 order. @code{buffer-list} normally returns the list for the selected
7087 frame. This allows you to work in separate frames without things
7088 interfering with each other.
7090 The standard way to look up a buffer given a name is
7091 @code{get-buffer}, and the standard way to create a new buffer is
7092 @code{get-buffer-create}, which looks up a buffer with a given name,
7093 creating a new one if necessary. These operations correspond exactly
7094 with the symbol operations @code{intern-soft} and @code{intern},
7095 respectively. You can also force a new buffer to be created using
7096 @code{generate-new-buffer}, which takes a name and (if necessary) makes
7097 a unique name from this by appending a number, and then creates the
7098 buffer. This is basically like the symbol operation @code{gensym}.
7100 @node Markers and Extents, Bufbytes and Emchars, Buffer Lists, Buffers and Textual Representation
7101 @section Markers and Extents
7103 Among the things associated with a buffer are things that are
7104 logically attached to certain buffer positions. This can be used to
7105 keep track of a buffer position when text is inserted and deleted, so
7106 that it remains at the same spot relative to the text around it; to
7107 assign properties to particular sections of text; etc. There are two
7108 such objects that are useful in this regard: they are @dfn{markers} and
7111 A @dfn{marker} is simply a flag placed at a particular buffer
7112 position, which is moved around as text is inserted and deleted.
7113 Markers are used for all sorts of purposes, such as the @code{mark} that
7114 is the other end of textual regions to be cut, copied, etc.
7116 An @dfn{extent} is similar to two markers plus some associated
7117 properties, and is used to keep track of regions in a buffer as text is
7118 inserted and deleted, and to add properties (e.g. fonts) to particular
7119 regions of text. The external interface of extents is explained
7122 The important thing here is that markers and extents simply contain
7123 buffer positions in them as integers, and every time text is inserted or
7124 deleted, these positions must be updated. In order to minimize the
7125 amount of shuffling that needs to be done, the positions in markers and
7126 extents (there's one per marker, two per extent) and stored in Meminds.
7127 This means that they only need to be moved when the text is physically
7128 moved in memory; since the gap structure tries to minimize this, it also
7129 minimizes the number of marker and extent indices that need to be
7130 adjusted. Look in @file{insdel.c} for the details of how this works.
7132 One other important distinction is that markers are @dfn{temporary}
7133 while extents are @dfn{permanent}. This means that markers disappear as
7134 soon as there are no more pointers to them, and correspondingly, there
7135 is no way to determine what markers are in a buffer if you are just
7136 given the buffer. Extents remain in a buffer until they are detached
7137 (which could happen as a result of text being deleted) or the buffer is
7138 deleted, and primitives do exist to enumerate the extents in a buffer.
7140 @node Bufbytes and Emchars, The Buffer Object, Markers and Extents, Buffers and Textual Representation
7141 @section Bufbytes and Emchars
7145 @node The Buffer Object, , Bufbytes and Emchars, Buffers and Textual Representation
7146 @section The Buffer Object
7148 Buffers contain fields not directly accessible by the Lisp programmer.
7149 We describe them here, naming them by the names used in the C code.
7150 Many are accessible indirectly in Lisp programs via Lisp primitives.
7154 The buffer name is a string that names the buffer. It is guaranteed to
7155 be unique. @xref{Buffer Names,,, lispref, XEmacs Lisp Programmer's
7159 This field contains the time when the buffer was last saved, as an
7160 integer. @xref{Buffer Modification,,, lispref, XEmacs Lisp Programmer's
7164 This field contains the modification time of the visited file. It is
7165 set when the file is written or read. Every time the buffer is written
7166 to the file, this field is compared to the modification time of the
7167 file. @xref{Buffer Modification,,, lispref, XEmacs Lisp Programmer's
7170 @item auto_save_modified
7171 This field contains the time when the buffer was last auto-saved.
7173 @item last_window_start
7174 This field contains the @code{window-start} position in the buffer as of
7175 the last time the buffer was displayed in a window.
7178 This field points to the buffer's undo list. @xref{Undo,,, lispref,
7179 XEmacs Lisp Programmer's Manual}.
7181 @item syntax_table_v
7182 This field contains the syntax table for the buffer. @xref{Syntax
7183 Tables,,, lispref, XEmacs Lisp Programmer's Manual}.
7185 @item downcase_table
7186 This field contains the conversion table for converting text to lower
7187 case. @xref{Case Tables,,, lispref, XEmacs Lisp Programmer's Manual}.
7190 This field contains the conversion table for converting text to upper
7191 case. @xref{Case Tables,,, lispref, XEmacs Lisp Programmer's Manual}.
7193 @item case_canon_table
7194 This field contains the conversion table for canonicalizing text for
7195 case-folding search. @xref{Case Tables,,, lispref, XEmacs Lisp
7196 Programmer's Manual}.
7198 @item case_eqv_table
7199 This field contains the equivalence table for case-folding search.
7200 @xref{Case Tables,,, lispref, XEmacs Lisp Programmer's Manual}.
7203 This field contains the buffer's display table, or @code{nil} if it
7204 doesn't have one. @xref{Display Tables,,, lispref, XEmacs Lisp
7205 Programmer's Manual}.
7208 This field contains the chain of all markers that currently point into
7209 the buffer. Deletion of text in the buffer, and motion of the buffer's
7210 gap, must check each of these markers and perhaps update it.
7211 @xref{Markers,,, lispref, XEmacs Lisp Programmer's Manual}.
7214 This field is a flag that tells whether a backup file has been made for
7215 the visited file of this buffer.
7218 This field contains the mark for the buffer. The mark is a marker,
7219 hence it is also included on the list @code{markers}. @xref{The Mark,,,
7220 lispref, XEmacs Lisp Programmer's Manual}.
7223 This field is non-@code{nil} if the buffer's mark is active.
7225 @item local_var_alist
7226 This field contains the association list describing the variables local
7227 in this buffer, and their values, with the exception of local variables
7228 that have special slots in the buffer object. (Those slots are omitted
7229 from this table.) @xref{Buffer-Local Variables,,, lispref, XEmacs Lisp
7230 Programmer's Manual}.
7232 @item modeline_format
7233 This field contains a Lisp object which controls how to display the mode
7234 line for this buffer. @xref{Modeline Format,,, lispref, XEmacs Lisp
7235 Programmer's Manual}.
7238 This field holds the buffer's base buffer (if it is an indirect buffer),
7242 @node MULE Character Sets and Encodings, The Lisp Reader and Compiler, Buffers and Textual Representation, Top
7243 @chapter MULE Character Sets and Encodings
7245 Recall that there are two primary ways that text is represented in
7246 XEmacs. The @dfn{buffer} representation sees the text as a series of
7247 bytes (Bufbytes), with a variable number of bytes used per character.
7248 The @dfn{character} representation sees the text as a series of integers
7249 (Emchars), one per character. The character representation is a cleaner
7250 representation from a theoretical standpoint, and is thus used in many
7251 cases when lots of manipulations on a string need to be done. However,
7252 the buffer representation is the standard representation used in both
7253 Lisp strings and buffers, and because of this, it is the ``default''
7254 representation that text comes in. The reason for using this
7255 representation is that it's compact and is compatible with ASCII.
7260 * Internal Mule Encodings::
7264 @node Character Sets, Encodings, MULE Character Sets and Encodings, MULE Character Sets and Encodings
7265 @section Character Sets
7267 A character set (or @dfn{charset}) is an ordered set of characters. A
7268 particular character in a charset is indexed using one or more
7269 @dfn{position codes}, which are non-negative integers. The number of
7270 position codes needed to identify a particular character in a charset is
7271 called the @dfn{dimension} of the charset. In XEmacs/Mule, all charsets
7272 have dimension 1 or 2, and the size of all charsets (except for a few
7273 special cases) is either 94, 96, 94 by 94, or 96 by 96. The range of
7274 position codes used to index characters from any of these types of
7275 character sets is as follows:
7278 Charset type Position code 1 Position code 2
7279 ------------------------------------------------------------
7282 94x94 33 - 126 33 - 126
7283 96x96 32 - 127 32 - 127
7286 Note that in the above cases position codes do not start at an
7287 expected value such as 0 or 1. The reason for this will become clear
7290 For example, Latin-1 is a 96-character charset, and JISX0208 (the
7291 Japanese national character set) is a 94x94-character charset.
7293 [Note that, although the ranges above define the @emph{valid} position
7294 codes for a charset, some of the slots in a particular charset may in
7295 fact be empty. This is the case for JISX0208, for example, where (e.g.)
7296 all the slots whose first position code is in the range 118 - 127 are
7299 There are three charsets that do not follow the above rules. All of
7300 them have one dimension, and have ranges of position codes as follows:
7303 Charset name Position code 1
7304 ------------------------------------
7307 Composite 0 - some large number
7310 (The upper bound of the position code for composite characters has not
7311 yet been determined, but it will probably be at least 16,383).
7313 ASCII is the union of two subsidiary character sets: Printing-ASCII
7314 (the printing ASCII character set, consisting of position codes 33 -
7315 126, like for a standard 94-character charset) and Control-ASCII (the
7316 non-printing characters that would appear in a binary file with codes 0
7319 Control-1 contains the non-printing characters that would appear in a
7320 binary file with codes 128 - 159.
7322 Composite contains characters that are generated by overstriking one
7323 or more characters from other charsets.
7325 Note that some characters in ASCII, and all characters in Control-1,
7326 are @dfn{control} (non-printing) characters. These have no printed
7327 representation but instead control some other function of the printing
7328 (e.g. TAB or 8 moves the current character position to the next tab
7329 stop). All other characters in all charsets are @dfn{graphic}
7330 (printing) characters.
7332 When a binary file is read in, the bytes in the file are assigned to
7333 character sets as follows:
7336 Bytes Character set Range
7337 --------------------------------------------------
7338 0 - 127 ASCII 0 - 127
7339 128 - 159 Control-1 0 - 31
7340 160 - 255 Latin-1 32 - 127
7343 This is a bit ad-hoc but gets the job done.
7345 @node Encodings, Internal Mule Encodings, Character Sets, MULE Character Sets and Encodings
7348 An @dfn{encoding} is a way of numerically representing characters from
7349 one or more character sets. If an encoding only encompasses one
7350 character set, then the position codes for the characters in that
7351 character set could be used directly. This is not possible, however, if
7352 more than one character set is to be used in the encoding.
7354 For example, the conversion detailed above between bytes in a binary
7355 file and characters is effectively an encoding that encompasses the
7356 three character sets ASCII, Control-1, and Latin-1 in a stream of 8-bit
7359 Thus, an encoding can be viewed as a way of encoding characters from a
7360 specified group of character sets using a stream of bytes, each of which
7361 contains a fixed number of bits (but not necessarily 8, as in the common
7364 Here are descriptions of a couple of common
7368 * Japanese EUC (Extended Unix Code)::
7372 @node Japanese EUC (Extended Unix Code), JIS7, Encodings, Encodings
7373 @subsection Japanese EUC (Extended Unix Code)
7375 This encompasses the character sets Printing-ASCII, Japanese-JISX0201,
7376 and Japanese-JISX0208-Kana (half-width katakana, the right half of
7377 JISX0201). It uses 8-bit bytes.
7379 Note that Printing-ASCII and Japanese-JISX0201-Kana are 94-character
7380 charsets, while Japanese-JISX0208 is a 94x94-character charset.
7382 The encoding is as follows:
7385 Character set Representation (PC=position-code)
7386 ------------- --------------
7388 Japanese-JISX0201-Kana 0x8E | PC1 + 0x80
7389 Japanese-JISX0208 PC1 + 0x80 | PC2 + 0x80
7390 Japanese-JISX0212 PC1 + 0x80 | PC2 + 0x80
7394 @node JIS7, , Japanese EUC (Extended Unix Code), Encodings
7397 This encompasses the character sets Printing-ASCII,
7398 Japanese-JISX0201-Roman (the left half of JISX0201; this character set
7399 is very similar to Printing-ASCII and is a 94-character charset),
7400 Japanese-JISX0208, and Japanese-JISX0201-Kana. It uses 7-bit bytes.
7402 Unlike Japanese EUC, this is a @dfn{modal} encoding, which
7403 means that there are multiple states that the encoding can
7404 be in, which affect how the bytes are to be interpreted.
7405 Special sequences of bytes (called @dfn{escape sequences})
7406 are used to change states.
7408 The encoding is as follows:
7411 Character set Representation (PC=position-code)
7412 ------------- --------------
7414 Japanese-JISX0201-Roman PC1
7415 Japanese-JISX0201-Kana PC1
7416 Japanese-JISX0208 PC1 PC2
7419 Escape sequence ASCII equivalent Meaning
7420 --------------- ---------------- -------
7421 0x1B 0x28 0x4A ESC ( J invoke Japanese-JISX0201-Roman
7422 0x1B 0x28 0x49 ESC ( I invoke Japanese-JISX0201-Kana
7423 0x1B 0x24 0x42 ESC $ B invoke Japanese-JISX0208
7424 0x1B 0x28 0x42 ESC ( B invoke Printing-ASCII
7427 Initially, Printing-ASCII is invoked.
7429 @node Internal Mule Encodings, CCL, Encodings, MULE Character Sets and Encodings
7430 @section Internal Mule Encodings
7432 In XEmacs/Mule, each character set is assigned a unique number, called a
7433 @dfn{leading byte}. This is used in the encodings of a character.
7434 Leading bytes are in the range 0x80 - 0xFF (except for ASCII, which has
7435 a leading byte of 0), although some leading bytes are reserved.
7437 Charsets whose leading byte is in the range 0x80 - 0x9F are called
7438 @dfn{official} and are used for built-in charsets. Other charsets are
7439 called @dfn{private} and have leading bytes in the range 0xA0 - 0xFF;
7440 these are user-defined charsets.
7445 Character set Leading byte
7446 ------------- ------------
7449 Dimension-1 Official 0x81 - 0x8D
7452 Dimension-2 Official 0x90 - 0x99
7453 (0x9A - 0x9D are free;
7454 0x9E and 0x9F are reserved)
7455 Dimension-1 Private 0xA0 - 0xEF
7456 Dimension-2 Private 0xF0 - 0xFF
7459 There are two internal encodings for characters in XEmacs/Mule. One is
7460 called @dfn{string encoding} and is an 8-bit encoding that is used for
7461 representing characters in a buffer or string. It uses 1 to 4 bytes per
7462 character. The other is called @dfn{character encoding} and is a 19-bit
7463 encoding that is used for representing characters individually in a
7466 (In the following descriptions, we'll ignore composite characters for
7467 the moment. We also give a general (structural) overview first,
7468 followed later by the exact details.)
7471 * Internal String Encoding::
7472 * Internal Character Encoding::
7475 @node Internal String Encoding, Internal Character Encoding, Internal Mule Encodings, Internal Mule Encodings
7476 @subsection Internal String Encoding
7478 ASCII characters are encoded using their position code directly. Other
7479 characters are encoded using their leading byte followed by their
7480 position code(s) with the high bit set. Characters in private character
7481 sets have their leading byte prefixed with a @dfn{leading byte prefix},
7482 which is either 0x9E or 0x9F. (No character sets are ever assigned these
7483 leading bytes.) Specifically:
7486 Character set Encoding (PC=position-code, LB=leading-byte)
7487 ------------- --------
7489 Control-1 LB | PC1 + 0xA0 |
7490 Dimension-1 official LB | PC1 + 0x80 |
7491 Dimension-1 private 0x9E | LB | PC1 + 0x80 |
7492 Dimension-2 official LB | PC1 + 0x80 | PC2 + 0x80 |
7493 Dimension-2 private 0x9F | LB | PC1 + 0x80 | PC2 + 0x80
7496 The basic characteristic of this encoding is that the first byte
7497 of all characters is in the range 0x00 - 0x9F, and the second and
7498 following bytes of all characters is in the range 0xA0 - 0xFF.
7499 This means that it is impossible to get out of sync, or more
7504 Given any byte position, the beginning of the character it is
7505 within can be determined in constant time.
7507 Given any byte position at the beginning of a character, the
7508 beginning of the next character can be determined in constant
7511 Given any byte position at the beginning of a character, the
7512 beginning of the previous character can be determined in constant
7515 Textual searches can simply treat encoded strings as if they
7516 were encoded in a one-byte-per-character fashion rather than
7517 the actual multi-byte encoding.
7520 None of the standard non-modal encodings meet all of these
7521 conditions. For example, EUC satisfies only (2) and (3), while
7522 Shift-JIS and Big5 (not yet described) satisfy only (2). (All
7523 non-modal encodings must satisfy (2), in order to be unambiguous.)
7525 @node Internal Character Encoding, , Internal String Encoding, Internal Mule Encodings
7526 @subsection Internal Character Encoding
7528 One 19-bit word represents a single character. The word is
7529 separated into three fields:
7532 Bit number: 18 17 16 15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
7533 <------------> <------------------> <------------------>
7537 Note that fields 2 and 3 hold 7 bits each, while field 1 holds 5 bits.
7540 Character set Field 1 Field 2 Field 3
7541 ------------- ------- ------- -------
7546 Dimension-1 official 0 LB - 0x80 PC1
7547 range: (01 - 0D) (20 - 7F)
7548 Dimension-1 private 0 LB - 0x80 PC1
7549 range: (20 - 6F) (20 - 7F)
7550 Dimension-2 official LB - 0x8F PC1 PC2
7551 range: (01 - 0A) (20 - 7F) (20 - 7F)
7552 Dimension-2 private LB - 0xE1 PC1 PC2
7553 range: (0F - 1E) (20 - 7F) (20 - 7F)
7557 Note that character codes 0 - 255 are the same as the ``binary encoding''
7560 @node CCL, , Internal Mule Encodings, MULE Character Sets and Encodings
7565 CCL_PROGRAM := (CCL_MAIN_BLOCK
7568 CCL_MAIN_BLOCK := CCL_BLOCK
7569 CCL_EOF_BLOCK := CCL_BLOCK
7571 CCL_BLOCK := STATEMENT | (STATEMENT [STATEMENT ...])
7573 SET | IF | BRANCH | LOOP | REPEAT | BREAK
7576 SET := (REG = EXPRESSION) | (REG SELF_OP EXPRESSION)
7579 EXPRESSION := ARG | (EXPRESSION OP ARG)
7581 IF := (if EXPRESSION CCL_BLOCK CCL_BLOCK)
7582 BRANCH := (branch EXPRESSION CCL_BLOCK [CCL_BLOCK ...])
7583 LOOP := (loop STATEMENT [STATEMENT ...])
7586 | (write-repeat [REG | INT-OR-CHAR | string])
7587 | (write-read-repeat REG [INT-OR-CHAR | string | ARRAY]?)
7588 READ := (read REG) | (read REG REG)
7589 | (read-if REG ARITH_OP ARG CCL_BLOCK CCL_BLOCK)
7590 | (read-branch REG CCL_BLOCK [CCL_BLOCK ...])
7591 WRITE := (write REG) | (write REG REG)
7592 | (write INT-OR-CHAR) | (write STRING) | STRING
7596 REG := r0 | r1 | r2 | r3 | r4 | r5 | r6 | r7
7597 ARG := REG | INT-OR-CHAR
7598 OP := + | - | * | / | % | & | '|' | ^ | << | >> | <8 | >8 | //
7599 | < | > | == | <= | >= | !=
7601 += | -= | *= | /= | %= | &= | '|=' | ^= | <<= | >>=
7602 ARRAY := '[' INT-OR-CHAR ... ']'
7603 INT-OR-CHAR := INT | CHAR
7607 The machine code consists of a vector of 32-bit words.
7608 The first such word specifies the start of the EOF section of the code;
7609 this is the code executed to handle any stuff that needs to be done
7610 (e.g. designating back to ASCII and left-to-right mode) after all
7611 other encoded/decoded data has been written out. This is not used for
7612 charset CCL programs.
7614 REGISTER: 0..7 -- refered by RRR or rrr
7616 OPERATOR BIT FIELD (27-bit): XXXXXXXXXXXXXXX RRR TTTTT
7617 TTTTT (5-bit): operator type
7618 RRR (3-bit): register number
7619 XXXXXXXXXXXXXXXX (15-bit):
7620 CCCCCCCCCCCCCCC: constant or address
7621 000000000000rrr: register number
7648 OPERATORS: TTTTT RRR XX..
7650 SetCS: 00000 RRR C...C RRR = C...C
7651 SetCL: 00001 RRR ..... RRR = c...c
7653 SetR: 00010 RRR ..rrr RRR = rrr
7654 SetA: 00011 RRR ..rrr RRR = array[rrr]
7655 C.............C size of array = C...C
7656 c.............c contents = c...c
7658 Jump: 00100 000 c...c jump to c...c
7659 JumpCond: 00101 RRR c...c if (!RRR) jump to c...c
7660 WriteJump: 00110 RRR c...c Write1 RRR, jump to c...c
7661 WriteReadJump: 00111 RRR c...c Write1, Read1 RRR, jump to c...c
7662 WriteCJump: 01000 000 c...c Write1 C...C, jump to c...c
7664 WriteCReadJump: 01001 RRR c...c Write1 C...C, Read1 RRR,
7665 C.............C and jump to c...c
7666 WriteSJump: 01010 000 c...c WriteS, jump to c...c
7670 WriteSReadJump: 01011 RRR c...c WriteS, Read1 RRR, jump to c...c
7674 WriteAReadJump: 01100 RRR c...c WriteA, Read1 RRR, jump to c...c
7675 C.............C size of array = C...C
7676 c.............c contents = c...c
7678 Branch: 01101 RRR C...C if (RRR >= 0 && RRR < C..)
7679 c.............c branch to (RRR+1)th address
7680 Read1: 01110 RRR ... read 1-byte to RRR
7681 Read2: 01111 RRR ..rrr read 2-byte to RRR and rrr
7682 ReadBranch: 10000 RRR C...C Read1 and Branch
7685 Write1: 10001 RRR ..... write 1-byte RRR
7686 Write2: 10010 RRR ..rrr write 2-byte RRR and rrr
7687 WriteC: 10011 000 ..... write 1-char C...CC
7689 WriteS: 10100 000 ..... write C..-byte of string
7693 WriteA: 10101 RRR ..... write array[RRR]
7694 C.............C size of array = C...C
7695 c.............c contents = c...c
7697 End: 10110 000 ..... terminate the execution
7699 SetSelfCS: 10111 RRR C...C RRR AAAAA= C...C
7701 SetSelfCL: 11000 RRR ..... RRR AAAAA= c...c
7704 SetSelfR: 11001 RRR ..Rrr RRR AAAAA= rrr
7706 SetExprCL: 11010 RRR ..Rrr RRR = rrr AAAAA c...c
7709 SetExprR: 11011 RRR ..rrr RRR = rrr AAAAA Rrr
7712 JumpCondC: 11100 RRR c...c if !(RRR AAAAA C..) jump to c...c
7715 JumpCondR: 11101 RRR c...c if !(RRR AAAAA rrr) jump to c...c
7718 ReadJumpCondC: 11110 RRR c...c Read1 and JumpCondC
7721 ReadJumpCondR: 11111 RRR c...c Read1 and JumpCondR
7726 @node The Lisp Reader and Compiler, Lstreams, MULE Character Sets and Encodings, Top
7727 @chapter The Lisp Reader and Compiler
7731 @node Lstreams, Consoles; Devices; Frames; Windows, The Lisp Reader and Compiler, Top
7734 An @dfn{lstream} is an internal Lisp object that provides a generic
7735 buffering stream implementation. Conceptually, you send data to the
7736 stream or read data from the stream, not caring what's on the other end
7737 of the stream. The other end could be another stream, a file
7738 descriptor, a stdio stream, a fixed block of memory, a reallocating
7739 block of memory, etc. The main purpose of the stream is to provide a
7740 standard interface and to do buffering. Macros are defined to read or
7741 write characters, so the calling functions do not have to worry about
7742 blocking data together in order to achieve efficiency.
7745 * Creating an Lstream:: Creating an lstream object.
7746 * Lstream Types:: Different sorts of things that are streamed.
7747 * Lstream Functions:: Functions for working with lstreams.
7748 * Lstream Methods:: Creating new lstream types.
7751 @node Creating an Lstream, Lstream Types, Lstreams, Lstreams
7752 @section Creating an Lstream
7754 Lstreams come in different types, depending on what is being interfaced
7755 to. Although the primitive for creating new lstreams is
7756 @code{Lstream_new()}, generally you do not call this directly. Instead,
7757 you call some type-specific creation function, which creates the lstream
7758 and initializes it as appropriate for the particular type.
7760 All lstream creation functions take a @var{mode} argument, specifying
7761 what mode the lstream should be opened as. This controls whether the
7762 lstream is for input and output, and optionally whether data should be
7763 blocked up in units of MULE characters. Note that some types of
7764 lstreams can only be opened for input; others only for output; and
7765 others can be opened either way. #### Richard Mlynarik thinks that
7766 there should be a strict separation between input and output streams,
7767 and he's probably right.
7769 @var{mode} is a string, one of
7777 Open for reading, but ``read'' never returns partial MULE characters.
7779 Open for writing, but never writes partial MULE characters.
7782 @node Lstream Types, Lstream Functions, Creating an Lstream, Lstreams
7783 @section Lstream Types
7794 @item resizing-buffer
7807 @node Lstream Functions, Lstream Methods, Lstream Types, Lstreams
7808 @section Lstream Functions
7810 @deftypefun {Lstream *} Lstream_new (Lstream_implementation *@var{imp}, const char *@var{mode})
7811 Allocate and return a new Lstream. This function is not really meant to
7812 be called directly; rather, each stream type should provide its own
7813 stream creation function, which creates the stream and does any other
7814 necessary creation stuff (e.g. opening a file).
7817 @deftypefun void Lstream_set_buffering (Lstream *@var{lstr}, Lstream_buffering @var{buffering}, int @var{buffering_size})
7818 Change the buffering of a stream. See @file{lstream.h}. By default the
7819 buffering is @code{STREAM_BLOCK_BUFFERED}.
7822 @deftypefun int Lstream_flush (Lstream *@var{lstr})
7823 Flush out any pending unwritten data in the stream. Clear any buffered
7824 input data. Returns 0 on success, -1 on error.
7827 @deftypefn Macro int Lstream_putc (Lstream *@var{stream}, int @var{c})
7828 Write out one byte to the stream. This is a macro and so it is very
7829 efficient. The @var{c} argument is only evaluated once but the @var{stream}
7830 argument is evaluated more than once. Returns 0 on success, -1 on
7834 @deftypefn Macro int Lstream_getc (Lstream *@var{stream})
7835 Read one byte from the stream. This is a macro and so it is very
7836 efficient. The @var{stream} argument is evaluated more than once. Return
7837 value is -1 for EOF or error.
7840 @deftypefn Macro void Lstream_ungetc (Lstream *@var{stream}, int @var{c})
7841 Push one byte back onto the input queue. This will be the next byte
7842 read from the stream. Any number of bytes can be pushed back and will
7843 be read in the reverse order they were pushed back---most recent
7844 first. (This is necessary for consistency---if there are a number of
7845 bytes that have been unread and I read and unread a byte, it needs to be
7846 the first to be read again.) This is a macro and so it is very
7847 efficient. The @var{c} argument is only evaluated once but the @var{stream}
7848 argument is evaluated more than once.
7851 @deftypefun int Lstream_fputc (Lstream *@var{stream}, int @var{c})
7852 @deftypefunx int Lstream_fgetc (Lstream *@var{stream})
7853 @deftypefunx void Lstream_fungetc (Lstream *@var{stream}, int @var{c})
7854 Function equivalents of the above macros.
7857 @deftypefun ssize_t Lstream_read (Lstream *@var{stream}, void *@var{data}, size_t @var{size})
7858 Read @var{size} bytes of @var{data} from the stream. Return the number
7859 of bytes read. 0 means EOF. -1 means an error occurred and no bytes
7863 @deftypefun ssize_t Lstream_write (Lstream *@var{stream}, void *@var{data}, size_t @var{size})
7864 Write @var{size} bytes of @var{data} to the stream. Return the number
7865 of bytes written. -1 means an error occurred and no bytes were written.
7868 @deftypefun void Lstream_unread (Lstream *@var{stream}, void *@var{data}, size_t @var{size})
7869 Push back @var{size} bytes of @var{data} onto the input queue. The next
7870 call to @code{Lstream_read()} with the same size will read the same
7871 bytes back. Note that this will be the case even if there is other
7872 pending unread data.
7875 @deftypefun int Lstream_close (Lstream *@var{stream})
7876 Close the stream. All data will be flushed out.
7879 @deftypefun void Lstream_reopen (Lstream *@var{stream})
7880 Reopen a closed stream. This enables I/O on it again. This is not
7881 meant to be called except from a wrapper routine that reinitializes
7882 variables and such---the close routine may well have freed some
7883 necessary storage structures, for example.
7886 @deftypefun void Lstream_rewind (Lstream *@var{stream})
7887 Rewind the stream to the beginning.
7890 @node Lstream Methods, , Lstream Functions, Lstreams
7891 @section Lstream Methods
7893 @deftypefn {Lstream Method} ssize_t reader (Lstream *@var{stream}, unsigned char *@var{data}, size_t @var{size})
7894 Read some data from the stream's end and store it into @var{data}, which
7895 can hold @var{size} bytes. Return the number of bytes read. A return
7896 value of 0 means no bytes can be read at this time. This may be because
7897 of an EOF, or because there is a granularity greater than one byte that
7898 the stream imposes on the returned data, and @var{size} is less than
7899 this granularity. (This will happen frequently for streams that need to
7900 return whole characters, because @code{Lstream_read()} calls the reader
7901 function repeatedly until it has the number of bytes it wants or until 0
7902 is returned.) The lstream functions do not treat a 0 return as EOF or
7903 do anything special; however, the calling function will interpret any 0
7904 it gets back as EOF. This will normally not happen unless the caller
7905 calls @code{Lstream_read()} with a very small size.
7907 This function can be @code{NULL} if the stream is output-only.
7910 @deftypefn {Lstream Method} ssize_t writer (Lstream *@var{stream}, const unsigned char *@var{data}, size_t @var{size})
7911 Send some data to the stream's end. Data to be sent is in @var{data}
7912 and is @var{size} bytes. Return the number of bytes sent. This
7913 function can send and return fewer bytes than is passed in; in that
7914 case, the function will just be called again until there is no data left
7915 or 0 is returned. A return value of 0 means that no more data can be
7916 currently stored, but there is no error; the data will be squirreled
7917 away until the writer can accept data. (This is useful, e.g., if you're
7918 dealing with a non-blocking file descriptor and are getting
7919 @code{EWOULDBLOCK} errors.) This function can be @code{NULL} if the
7920 stream is input-only.
7923 @deftypefn {Lstream Method} int rewinder (Lstream *@var{stream})
7924 Rewind the stream. If this is @code{NULL}, the stream is not seekable.
7927 @deftypefn {Lstream Method} int seekable_p (Lstream *@var{stream})
7928 Indicate whether this stream is seekable---i.e. it can be rewound.
7929 This method is ignored if the stream does not have a rewind method. If
7930 this method is not present, the result is determined by whether a rewind
7934 @deftypefn {Lstream Method} int flusher (Lstream *@var{stream})
7935 Perform any additional operations necessary to flush the data in this
7939 @deftypefn {Lstream Method} int pseudo_closer (Lstream *@var{stream})
7942 @deftypefn {Lstream Method} int closer (Lstream *@var{stream})
7943 Perform any additional operations necessary to close this stream down.
7944 May be @code{NULL}. This function is called when @code{Lstream_close()}
7945 is called or when the stream is garbage-collected. When this function
7946 is called, all pending data in the stream will already have been written
7950 @deftypefn {Lstream Method} Lisp_Object marker (Lisp_Object @var{lstream}, void (*@var{markfun}) (Lisp_Object))
7951 Mark this object for garbage collection. Same semantics as a standard
7952 @code{Lisp_Object} marker. This function can be @code{NULL}.
7955 @node Consoles; Devices; Frames; Windows, The Redisplay Mechanism, Lstreams, Top
7956 @chapter Consoles; Devices; Frames; Windows
7959 * Introduction to Consoles; Devices; Frames; Windows::
7961 * Window Hierarchy::
7962 * The Window Object::
7965 @node Introduction to Consoles; Devices; Frames; Windows, Point, Consoles; Devices; Frames; Windows, Consoles; Devices; Frames; Windows
7966 @section Introduction to Consoles; Devices; Frames; Windows
7968 A window-system window that you see on the screen is called a
7969 @dfn{frame} in Emacs terminology. Each frame is subdivided into one or
7970 more non-overlapping panes, called (confusingly) @dfn{windows}. Each
7971 window displays the text of a buffer in it. (See above on Buffers.) Note
7972 that buffers and windows are independent entities: Two or more windows
7973 can be displaying the same buffer (potentially in different locations),
7974 and a buffer can be displayed in no windows.
7976 A single display screen that contains one or more frames is called
7977 a @dfn{display}. Under most circumstances, there is only one display.
7978 However, more than one display can exist, for example if you have
7979 a @dfn{multi-headed} console, i.e. one with a single keyboard but
7980 multiple displays. (Typically in such a situation, the various
7981 displays act like one large display, in that the mouse is only
7982 in one of them at a time, and moving the mouse off of one moves
7983 it into another.) In some cases, the different displays will
7984 have different characteristics, e.g. one color and one mono.
7986 XEmacs can display frames on multiple displays. It can even deal
7987 simultaneously with frames on multiple keyboards (called @dfn{consoles} in
7988 XEmacs terminology). Here is one case where this might be useful: You
7989 are using XEmacs on your workstation at work, and leave it running.
7990 Then you go home and dial in on a TTY line, and you can use the
7991 already-running XEmacs process to display another frame on your local
7994 Thus, there is a hierarchy console -> display -> frame -> window.
7995 There is a separate Lisp object type for each of these four concepts.
7996 Furthermore, there is logically a @dfn{selected console},
7997 @dfn{selected display}, @dfn{selected frame}, and @dfn{selected window}.
7998 Each of these objects is distinguished in various ways, such as being the
7999 default object for various functions that act on objects of that type.
8000 Note that every containing object remembers the ``selected'' object
8001 among the objects that it contains: e.g. not only is there a selected
8002 window, but every frame remembers the last window in it that was
8003 selected, and changing the selected frame causes the remembered window
8004 within it to become the selected window. Similar relationships apply
8005 for consoles to devices and devices to frames.
8007 @node Point, Window Hierarchy, Introduction to Consoles; Devices; Frames; Windows, Consoles; Devices; Frames; Windows
8010 Recall that every buffer has a current insertion position, called
8011 @dfn{point}. Now, two or more windows may be displaying the same buffer,
8012 and the text cursor in the two windows (i.e. @code{point}) can be in
8013 two different places. You may ask, how can that be, since each
8014 buffer has only one value of @code{point}? The answer is that each window
8015 also has a value of @code{point} that is squirreled away in it. There
8016 is only one selected window, and the value of ``point'' in that buffer
8017 corresponds to that window. When the selected window is changed
8018 from one window to another displaying the same buffer, the old
8019 value of @code{point} is stored into the old window's ``point'' and the
8020 value of @code{point} from the new window is retrieved and made the
8021 value of @code{point} in the buffer. This means that @code{window-point}
8022 for the selected window is potentially inaccurate, and if you
8023 want to retrieve the correct value of @code{point} for a window,
8024 you must special-case on the selected window and retrieve the
8025 buffer's point instead. This is related to why @code{save-window-excursion}
8026 does not save the selected window's value of @code{point}.
8028 @node Window Hierarchy, The Window Object, Point, Consoles; Devices; Frames; Windows
8029 @section Window Hierarchy
8030 @cindex window hierarchy
8031 @cindex hierarchy of windows
8033 If a frame contains multiple windows (panes), they are always created
8034 by splitting an existing window along the horizontal or vertical axis.
8035 Terminology is a bit confusing here: to @dfn{split a window
8036 horizontally} means to create two side-by-side windows, i.e. to make a
8037 @emph{vertical} cut in a window. Likewise, to @dfn{split a window
8038 vertically} means to create two windows, one above the other, by making
8039 a @emph{horizontal} cut.
8041 If you split a window and then split again along the same axis, you
8042 will end up with a number of panes all arranged along the same axis.
8043 The precise way in which the splits were made should not be important,
8044 and this is reflected internally. Internally, all windows are arranged
8045 in a tree, consisting of two types of windows, @dfn{combination} windows
8046 (which have children, and are covered completely by those children) and
8047 @dfn{leaf} windows, which have no children and are visible. Every
8048 combination window has two or more children, all arranged along the same
8049 axis. There are (logically) two subtypes of windows, depending on
8050 whether their children are horizontally or vertically arrayed. There is
8051 always one root window, which is either a leaf window (if the frame
8052 contains only one window) or a combination window (if the frame contains
8053 more than one window). In the latter case, the root window will have
8054 two or more children, either horizontally or vertically arrayed, and
8055 each of those children will be either a leaf window or another
8058 Here are some rules:
8062 Horizontal combination windows can never have children that are
8063 horizontal combination windows; same for vertical.
8066 Only leaf windows can be split (obviously) and this splitting does one
8067 of two things: (a) turns the leaf window into a combination window and
8068 creates two new leaf children, or (b) turns the leaf window into one of
8069 the two new leaves and creates the other leaf. Rule (1) dictates which
8070 of these two outcomes happens.
8073 Every combination window must have at least two children.
8076 Leaf windows can never become combination windows. They can be deleted,
8077 however. If this results in a violation of (3), the parent combination
8078 window also gets deleted.
8081 All functions that accept windows must be prepared to accept combination
8082 windows, and do something sane (e.g. signal an error if so).
8083 Combination windows @emph{do} escape to the Lisp level.
8086 All windows have three fields governing their contents:
8087 these are @dfn{hchild} (a list of horizontally-arrayed children),
8088 @dfn{vchild} (a list of vertically-arrayed children), and @dfn{buffer}
8089 (the buffer contained in a leaf window). Exactly one of
8090 these will be non-nil. Remember that @dfn{horizontally-arrayed}
8091 means ``side-by-side'' and @dfn{vertically-arrayed} means
8092 @dfn{one above the other}.
8095 Leaf windows also have markers in their @code{start} (the
8096 first buffer position displayed in the window) and @code{pointm}
8097 (the window's stashed value of @code{point}---see above) fields,
8098 while combination windows have nil in these fields.
8101 The list of children for a window is threaded through the
8102 @code{next} and @code{prev} fields of each child window.
8105 @strong{Deleted windows can be undeleted}. This happens as a result of
8106 restoring a window configuration, and is unlike frames, displays, and
8107 consoles, which, once deleted, can never be restored. Deleting a window
8108 does nothing except set a special @code{dead} bit to 1 and clear out the
8109 @code{next}, @code{prev}, @code{hchild}, and @code{vchild} fields, for
8113 Most frames actually have two top-level windows---one for the
8114 minibuffer and one (the @dfn{root}) for everything else. The modeline
8115 (if present) separates these two. The @code{next} field of the root
8116 points to the minibuffer, and the @code{prev} field of the minibuffer
8117 points to the root. The other @code{next} and @code{prev} fields are
8118 @code{nil}, and the frame points to both of these windows.
8119 Minibuffer-less frames have no minibuffer window, and the @code{next}
8120 and @code{prev} of the root window are @code{nil}. Minibuffer-only
8121 frames have no root window, and the @code{next} of the minibuffer window
8122 is @code{nil} but the @code{prev} points to itself. (#### This is an
8123 artifact that should be fixed.)
8126 @node The Window Object, , Window Hierarchy, Consoles; Devices; Frames; Windows
8127 @section The Window Object
8129 Windows have the following accessible fields:
8133 The frame that this window is on.
8136 Non-@code{nil} if this window is a minibuffer window.
8139 The buffer that the window is displaying. This may change often during
8140 the life of the window.
8143 Non-@code{nil} if this window is dedicated to its buffer.
8146 @cindex window point internals
8147 This is the value of point in the current buffer when this window is
8148 selected; when it is not selected, it retains its previous value.
8151 The position in the buffer that is the first character to be displayed
8155 If this flag is non-@code{nil}, it says that the window has been
8156 scrolled explicitly by the Lisp program. This affects what the next
8157 redisplay does if point is off the screen: instead of scrolling the
8158 window to show the text around point, it moves point to a location that
8162 The @code{modified} field of the window's buffer, as of the last time
8163 a redisplay completed in this window.
8166 The buffer's value of point, as of the last time
8167 a redisplay completed in this window.
8170 This is the left-hand edge of the window, measured in columns. (The
8171 leftmost column on the screen is @w{column 0}.)
8174 This is the top edge of the window, measured in lines. (The top line on
8175 the screen is @w{line 0}.)
8178 The height of the window, measured in lines.
8181 The width of the window, measured in columns.
8184 This is the window that is the next in the chain of siblings. It is
8185 @code{nil} in a window that is the rightmost or bottommost of a group of
8189 This is the window that is the previous in the chain of siblings. It is
8190 @code{nil} in a window that is the leftmost or topmost of a group of
8194 Internally, XEmacs arranges windows in a tree; each group of siblings has
8195 a parent window whose area includes all the siblings. This field points
8196 to a window's parent.
8198 Parent windows do not display buffers, and play little role in display
8199 except to shape their child windows. Emacs Lisp programs usually have
8200 no access to the parent windows; they operate on the windows at the
8201 leaves of the tree, which actually display buffers.
8204 This is the number of columns that the display in the window is scrolled
8205 horizontally to the left. Normally, this is 0.
8208 This is the last time that the window was selected. The function
8209 @code{get-lru-window} uses this field.
8212 The window's display table, or @code{nil} if none is specified for it.
8214 @item update_mode_line
8215 Non-@code{nil} means this window's mode line needs to be updated.
8217 @item base_line_number
8218 The line number of a certain position in the buffer, or @code{nil}.
8219 This is used for displaying the line number of point in the mode line.
8222 The position in the buffer for which the line number is known, or
8223 @code{nil} meaning none is known.
8225 @item region_showing
8226 If the region (or part of it) is highlighted in this window, this field
8227 holds the mark position that made one end of that region. Otherwise,
8228 this field is @code{nil}.
8231 @node The Redisplay Mechanism, Extents, Consoles; Devices; Frames; Windows, Top
8232 @chapter The Redisplay Mechanism
8234 The redisplay mechanism is one of the most complicated sections of
8235 XEmacs, especially from a conceptual standpoint. This is doubly so
8236 because, unlike for the basic aspects of the Lisp interpreter, the
8237 computer science theories of how to efficiently handle redisplay are not
8240 When working with the redisplay mechanism, remember the Golden Rules
8245 It Is Better To Be Correct Than Fast.
8247 Thou Shalt Not Run Elisp From Within Redisplay.
8249 It Is Better To Be Fast Than Not To Be.
8253 * Critical Redisplay Sections::
8254 * Line Start Cache::
8255 * Redisplay Piece by Piece::
8258 @node Critical Redisplay Sections, Line Start Cache, The Redisplay Mechanism, The Redisplay Mechanism
8259 @section Critical Redisplay Sections
8260 @cindex critical redisplay sections
8262 Within this section, we are defenseless and assume that the
8263 following cannot happen:
8269 Lisp code evaluation
8274 We ensure (3) by calling @code{hold_frame_size_changes()}, which
8275 will cause any pending frame size changes to get put on hold
8276 till after the end of the critical section. (1) follows
8277 automatically if (2) is met. #### Unfortunately, there are
8278 some places where Lisp code can be called within this section.
8279 We need to remove them.
8281 If @code{Fsignal()} is called during this critical section, we
8282 will @code{abort()}.
8284 If garbage collection is called during this critical section,
8285 we simply return. #### We should abort instead.
8287 #### If a frame-size change does occur we should probably
8288 actually be preempting redisplay.
8290 @node Line Start Cache, Redisplay Piece by Piece, Critical Redisplay Sections, The Redisplay Mechanism
8291 @section Line Start Cache
8292 @cindex line start cache
8294 The traditional scrolling code in Emacs breaks in a variable height
8295 world. It depends on the key assumption that the number of lines that
8296 can be displayed at any given time is fixed. This led to a complete
8297 separation of the scrolling code from the redisplay code. In order to
8298 fully support variable height lines, the scrolling code must actually be
8299 tightly integrated with redisplay. Only redisplay can determine how
8300 many lines will be displayed on a screen for any given starting point.
8302 What is ideally wanted is a complete list of the starting buffer
8303 position for every possible display line of a buffer along with the
8304 height of that display line. Maintaining such a full list would be very
8305 expensive. We settle for having it include information for all areas
8306 which we happen to generate anyhow (i.e. the region currently being
8307 displayed) and for those areas we need to work with.
8309 In order to ensure that the cache accurately represents what redisplay
8310 would actually show, it is necessary to invalidate it in many
8311 situations. If the buffer changes, the starting positions may no longer
8312 be correct. If a face or an extent has changed then the line heights
8313 may have altered. These events happen frequently enough that the cache
8314 can end up being constantly disabled. With this potentially constant
8315 invalidation when is the cache ever useful?
8317 Even if the cache is invalidated before every single usage, it is
8318 necessary. Scrolling often requires knowledge about display lines which
8319 are actually above or below the visible region. The cache provides a
8320 convenient light-weight method of storing this information for multiple
8321 display regions. This knowledge is necessary for the scrolling code to
8322 always obey the First Golden Rule of Redisplay.
8324 If the cache already contains all of the information that the scrolling
8325 routines happen to need so that it doesn't have to go generate it, then
8326 we are able to obey the Third Golden Rule of Redisplay. The first thing
8327 we do to help out the cache is to always add the displayed region. This
8328 region had to be generated anyway, so the cache ends up getting the
8329 information basically for free. In those cases where a user is simply
8330 scrolling around viewing a buffer there is a high probability that this
8331 is sufficient to always provide the needed information. The second
8332 thing we can do is be smart about invalidating the cache.
8334 TODO---Be smart about invalidating the cache. Potential places:
8338 Insertions at end-of-line which don't cause line-wraps do not alter the
8339 starting positions of any display lines. These types of buffer
8340 modifications should not invalidate the cache. This is actually a large
8341 optimization for redisplay speed as well.
8343 Buffer modifications frequently only affect the display of lines at and
8344 below where they occur. In these situations we should only invalidate
8345 the part of the cache starting at where the modification occurs.
8348 In case you're wondering, the Second Golden Rule of Redisplay is not
8351 @node Redisplay Piece by Piece, , Line Start Cache, The Redisplay Mechanism
8352 @section Redisplay Piece by Piece
8353 @cindex Redisplay Piece by Piece
8355 As you can begin to see redisplay is complex and also not well
8356 documented. Chuck no longer works on XEmacs so this section is my take
8357 on the workings of redisplay.
8359 Redisplay happens in three phases:
8363 Determine desired display in area that needs redisplay.
8364 Implemented by @code{redisplay.c}
8366 Compare desired display with current display
8367 Implemented by @code{redisplay-output.c}
8369 Output changes Implemented by @code{redisplay-output.c},
8370 @code{redisplay-x.c}, @code{redisplay-msw.c} and @code{redisplay-tty.c}
8373 Steps 1 and 2 are device-independent and relatively complex. Step 3 is
8374 mostly device-dependent.
8376 Determining the desired display
8378 Display attributes are stored in @code{display_line} structures. Each
8379 @code{display_line} consists of a set of @code{display_block}'s and each
8380 @code{display_block} contains a number of @code{rune}'s. Generally
8381 dynarr's of @code{display_line}'s are held by each window representing
8382 the current display and the desired display.
8384 The @code{display_line} structures are tightly tied to buffers which
8385 presents a problem for redisplay as this connection is bogus for the
8386 modeline. Hence the @code{display_line} generation routines are
8387 duplicated for generating the modeline. This means that the modeline
8388 display code has many bugs that the standard redisplay code does not.
8390 The guts of @code{display_line} generation are in
8391 @code{create_text_block}, which creates a single display line for the
8392 desired locale. This incrementally parses the characters on the current
8393 line and generates redisplay structures for each.
8395 Gutter redisplay is different. Because the data to display is stored in
8396 a string we cannot use @code{create_text_block}. Instead we use
8397 @code{create_text_string_block} which performs the same function as
8398 @code{create_text_block} but for strings. Many of the complexities of
8399 @code{create_text_block} to do with cursor handling and selective
8400 display have been removed.
8402 @node Extents, Faces, The Redisplay Mechanism, Top
8406 * Introduction to Extents:: Extents are ranges over text, with properties.
8407 * Extent Ordering:: How extents are ordered internally.
8408 * Format of the Extent Info:: The extent information in a buffer or string.
8409 * Zero-Length Extents:: A weird special case.
8410 * Mathematics of Extent Ordering:: A rigorous foundation.
8411 * Extent Fragments:: Cached information useful for redisplay.
8414 @node Introduction to Extents, Extent Ordering, Extents, Extents
8415 @section Introduction to Extents
8417 Extents are regions over a buffer, with a start and an end position
8418 denoting the region of the buffer included in the extent. In
8419 addition, either end can be closed or open, meaning that the endpoint
8420 is or is not logically included in the extent. Insertion of a character
8421 at a closed endpoint causes the character to go inside the extent;
8422 insertion at an open endpoint causes the character to go outside.
8424 Extent endpoints are stored using memory indices (see @file{insdel.c}),
8425 to minimize the amount of adjusting that needs to be done when
8426 characters are inserted or deleted.
8428 (Formerly, extent endpoints at the gap could be either before or
8429 after the gap, depending on the open/closedness of the endpoint.
8430 The intent of this was to make it so that insertions would
8431 automatically go inside or out of extents as necessary with no
8432 further work needing to be done. It didn't work out that way,
8433 however, and just ended up complexifying and buggifying all the
8436 @node Extent Ordering, Format of the Extent Info, Introduction to Extents, Extents
8437 @section Extent Ordering
8439 Extents are compared using memory indices. There are two orderings
8440 for extents and both orders are kept current at all times. The normal
8441 or @dfn{display} order is as follows:
8444 Extent A is ``less than'' extent B,
8445 that is, earlier in the display order,
8446 if: A-start < B-start,
8447 or if: A-start = B-start, and A-end > B-end
8450 So if two extents begin at the same position, the larger of them is the
8451 earlier one in the display order (@code{EXTENT_LESS} is true).
8453 For the e-order, the same thing holds:
8456 Extent A is ``less than'' extent B in e-order,
8457 that is, later in the buffer,
8459 or if: A-end = B-end, and A-start > B-start
8462 So if two extents end at the same position, the smaller of them is the
8463 earlier one in the e-order (@code{EXTENT_E_LESS} is true).
8465 The display order and the e-order are complementary orders: any
8466 theorem about the display order also applies to the e-order if you swap
8467 all occurrences of ``display order'' and ``e-order'', ``less than'' and
8468 ``greater than'', and ``extent start'' and ``extent end''.
8470 @node Format of the Extent Info, Zero-Length Extents, Extent Ordering, Extents
8471 @section Format of the Extent Info
8473 An extent-info structure consists of a list of the buffer or string's
8474 extents and a @dfn{stack of extents} that lists all of the extents over
8475 a particular position. The stack-of-extents info is used for
8476 optimization purposes---it basically caches some info that might
8477 be expensive to compute. Certain otherwise hard computations are easy
8478 given the stack of extents over a particular position, and if the
8479 stack of extents over a nearby position is known (because it was
8480 calculated at some prior point in time), it's easy to move the stack
8481 of extents to the proper position.
8483 Given that the stack of extents is an optimization, and given that
8484 it requires memory, a string's stack of extents is wiped out each
8485 time a garbage collection occurs. Therefore, any time you retrieve
8486 the stack of extents, it might not be there. If you need it to
8487 be there, use the @code{_force} version.
8489 Similarly, a string may or may not have an extent_info structure.
8490 (Generally it won't if there haven't been any extents added to the
8491 string.) So use the @code{_force} version if you need the extent_info
8492 structure to be there.
8494 A list of extents is maintained as a double gap array: one gap array
8495 is ordered by start index (the @dfn{display order}) and the other is
8496 ordered by end index (the @dfn{e-order}). Note that positions in an
8497 extent list should logically be conceived of as referring @emph{to} a
8498 particular extent (as is the norm in programs) rather than sitting
8499 between two extents. Note also that callers of these functions should
8500 not be aware of the fact that the extent list is implemented as an
8501 array, except for the fact that positions are integers (this should be
8502 generalized to handle integers and linked list equally well).
8504 @node Zero-Length Extents, Mathematics of Extent Ordering, Format of the Extent Info, Extents
8505 @section Zero-Length Extents
8507 Extents can be zero-length, and will end up that way if their endpoints
8508 are explicitly set that way or if their detachable property is nil
8509 and all the text in the extent is deleted. (The exception is open-open
8510 zero-length extents, which are barred from existing because there is
8511 no sensible way to define their properties. Deletion of the text in
8512 an open-open extent causes it to be converted into a closed-open
8513 extent.) Zero-length extents are primarily used to represent
8514 annotations, and behave as follows:
8518 Insertion at the position of a zero-length extent expands the extent
8519 if both endpoints are closed; goes after the extent if it is closed-open;
8520 and goes before the extent if it is open-closed.
8523 Deletion of a character on a side of a zero-length extent whose
8524 corresponding endpoint is closed causes the extent to be detached if
8525 it is detachable; if the extent is not detachable or the corresponding
8526 endpoint is open, the extent remains in the buffer, moving as necessary.
8529 Note that closed-open, non-detachable zero-length extents behave
8530 exactly like markers and that open-closed, non-detachable zero-length
8531 extents behave like the ``point-type'' marker in Mule.
8533 @node Mathematics of Extent Ordering, Extent Fragments, Zero-Length Extents, Extents
8534 @section Mathematics of Extent Ordering
8535 @cindex extent mathematics
8536 @cindex mathematics of extents
8537 @cindex extent ordering
8539 @cindex display order of extents
8540 @cindex extents, display order
8541 The extents in a buffer are ordered by ``display order'' because that
8542 is that order that the redisplay mechanism needs to process them in.
8543 The e-order is an auxiliary ordering used to facilitate operations
8544 over extents. The operations that can be performed on the ordered
8545 list of extents in a buffer are
8549 Locate where an extent would go if inserted into the list.
8551 Insert an extent into the list.
8553 Remove an extent from the list.
8555 Map over all the extents that overlap a range.
8558 (4) requires being able to determine the first and last extents
8559 that overlap a range.
8561 NOTE: @dfn{overlap} is used as follows:
8565 two ranges overlap if they have at least one point in common.
8566 Whether the endpoints are open or closed makes a difference here.
8568 a point overlaps a range if the point is contained within the
8569 range; this is equivalent to treating a point @math{P} as the range
8572 In the case of an @emph{extent} overlapping a point or range, the extent
8573 is normally treated as having closed endpoints. This applies
8574 consistently in the discussion of stacks of extents and such below.
8575 Note that this definition of overlap is not necessarily consistent with
8576 the extents that @code{map-extents} maps over, since @code{map-extents}
8577 sometimes pays attention to whether the endpoints of an extents are open
8578 or closed. But for our purposes, it greatly simplifies things to treat
8579 all extents as having closed endpoints.
8582 First, define @math{>}, @math{<}, @math{<=}, etc. as applied to extents
8583 to mean comparison according to the display order. Comparison between
8584 an extent @math{E} and an index @math{I} means comparison between
8585 @math{E} and the range @math{[I, I]}.
8587 Also define @math{e>}, @math{e<}, @math{e<=}, etc. to mean comparison
8588 according to the e-order.
8590 For any range @math{R}, define @math{R(0)} to be the starting index of
8591 the range and @math{R(1)} to be the ending index of the range.
8593 For any extent @math{E}, define @math{E(next)} to be the extent directly
8594 following @math{E}, and @math{E(prev)} to be the extent directly
8595 preceding @math{E}. Assume @math{E(next)} and @math{E(prev)} can be
8596 determined from @math{E} in constant time. (This is because we store
8597 the extent list as a doubly linked list.)
8599 Similarly, define @math{E(e-next)} and @math{E(e-prev)} to be the
8600 extents directly following and preceding @math{E} in the e-order.
8604 Let @math{R} be a range.
8605 Let @math{F} be the first extent overlapping @math{R}.
8606 Let @math{L} be the last extent overlapping @math{R}.
8608 Theorem 1: @math{R(1)} lies between @math{L} and @math{L(next)},
8609 i.e. @math{L <= R(1) < L(next)}.
8611 This follows easily from the definition of display order. The
8612 basic reason that this theorem applies is that the display order
8613 sorts by increasing starting index.
8615 Therefore, we can determine @math{L} just by looking at where we would
8616 insert @math{R(1)} into the list, and if we know @math{F} and are moving
8617 forward over extents, we can easily determine when we've hit @math{L} by
8618 comparing the extent we're at to @math{R(1)}.
8621 Theorem 2: @math{F(e-prev) e< [1, R(0)] e<= F}.
8624 This is the analog of Theorem 1, and applies because the e-order
8625 sorts by increasing ending index.
8627 Therefore, @math{F} can be found in the same amount of time as
8628 operation (1), i.e. the time that it takes to locate where an extent
8629 would go if inserted into the e-order list.
8631 If the lists were stored as balanced binary trees, then operation (1)
8632 would take logarithmic time, which is usually quite fast. However,
8633 currently they're stored as simple doubly-linked lists, and instead we
8634 do some caching to try to speed things up.
8636 Define a @dfn{stack of extents} (or @dfn{SOE}) as the set of extents
8637 (ordered in the display order) that overlap an index @math{I}, together
8638 with the SOE's @dfn{previous} extent, which is an extent that precedes
8639 @math{I} in the e-order. (Hopefully there will not be very many extents
8640 between @math{I} and the previous extent.)
8644 Let @math{I} be an index, let @math{S} be the stack of extents on
8645 @math{I}, let @math{F} be the first extent in @math{S}, and let @math{P}
8646 be @math{S}'s previous extent.
8648 Theorem 3: The first extent in @math{S} is the first extent that overlaps
8649 any range @math{[I, J]}.
8651 Proof: Any extent that overlaps @math{[I, J]} but does not include
8652 @math{I} must have a start index @math{> I}, and thus be greater than
8653 any extent in @math{S}.
8655 Therefore, finding the first extent that overlaps a range @math{R} is
8656 the same as finding the first extent that overlaps @math{R(0)}.
8658 Theorem 4: Let @math{I2} be an index such that @math{I2 > I}, and let
8659 @math{F2} be the first extent that overlaps @math{I2}. Then, either
8660 @math{F2} is in @math{S} or @math{F2} is greater than any extent in
8663 Proof: If @math{F2} does not include @math{I} then its start index is
8664 greater than @math{I} and thus it is greater than any extent in
8665 @math{S}, including @math{F}. Otherwise, @math{F2} includes @math{I}
8666 and thus is in @math{S}, and thus @math{F2 >= F}.
8668 @node Extent Fragments, , Mathematics of Extent Ordering, Extents
8669 @section Extent Fragments
8670 @cindex extent fragment
8672 Imagine that the buffer is divided up into contiguous, non-overlapping
8673 @dfn{runs} of text such that no extent starts or ends within a run
8674 (extents that abut the run don't count).
8676 An extent fragment is a structure that holds data about the run that
8677 contains a particular buffer position (if the buffer position is at the
8678 junction of two runs, the run after the position is used)---the
8679 beginning and end of the run, a list of all of the extents in that run,
8680 the @dfn{merged face} that results from merging all of the faces
8681 corresponding to those extents, the begin and end glyphs at the
8682 beginning of the run, etc. This is the information that redisplay needs
8683 in order to display this run.
8685 Extent fragments have to be very quick to update to a new buffer
8686 position when moving linearly through the buffer. They rely on the
8687 stack-of-extents code, which does the heavy-duty algorithmic work of
8688 determining which extents overly a particular position.
8690 @node Faces, Glyphs, Extents, Top
8695 @node Glyphs, Specifiers, Faces, Top
8698 Glyphs are graphical elements that can be displayed in XEmacs buffers or
8699 gutters. We use the term graphical element here in the broadest possible
8700 sense since glyphs can be as mundane as text to as arcane as a native
8703 In XEmacs, glyphs represent the uninstantiated state of graphical
8704 elements, i.e. they hold all the information necessary to produce an
8705 image on-screen but the image does not exist at this stage.
8707 Glyphs are lazily instantiated by calling one of the glyph
8708 functions. This usually occurs within redisplay when
8709 @code{Fglyph_height} is called. Instantiation causes an image-instance
8710 to be created and cached. This cache is on a device basis for all glyphs
8711 except glyph-widgets, and on a window basis for glyph widgets. The
8712 caching is done by @code{image_instantiate} and is necessary because it
8713 is generally possible to display an image-instance in multiple
8714 domains. For instance if we create a Pixmap, we can actually display
8715 this on multiple windows - even though we only need a single Pixmap
8716 instance to do this. If caching wasn't done then it would be necessary
8717 to create image-instances for every displayable occurrence of a glyph -
8718 and every usage - and this would be extremely memory and cpu intensive.
8720 Widget-glyphs (a.k.a native widgets) are not cached in this way. This is
8721 because widget-glyph image-instances on screen are toolkit windows, and
8722 thus cannot be reused in multiple XEmacs domains. Thus widget-glyphs are
8723 cached on a window basis.
8725 Any action on a glyph first consults the cache before actually
8726 instantiating a widget.
8728 @section Widget-Glyphs in the MS-Windows Environment
8732 @section Widget-Glyphs in the X Environment
8734 Widget-glyphs under X make heavy use of lwlib for manipulating the
8735 native toolkit objects. This is primarily so that different toolkits can
8736 be supported for widget-glyphs, just as they are supported for features
8737 such as menubars etc.
8739 Lwlib is extremely poorly documented and quite hairy so here is my
8740 understanding of what goes on.
8742 Lwlib maintains a set of widget_instances which mirror the hierarchical
8743 state of Xt widgets. I think this is so that widgets can be updated and
8744 manipulated generically by the lwlib library. For instance
8745 update_one_widget_instance can cope with multiple types of widget and
8746 multiple types of toolkit. Each element in the widget hierarchy is updated
8747 from its corresponding widget_instance by walking the widget_instance
8750 This has desirable properties such as lw_modify_all_widgets which is
8751 called from glyphs-x.c and updates all the properties of a widget
8752 without having to know what the widget is or what toolkit it is from.
8753 Unfortunately this also has hairy properties such as making the lwlib
8754 code quite complex. And of course lwlib has to know at some level what
8755 the widget is and how to set its properties.
8757 @node Specifiers, Menus, Glyphs, Top
8762 @node Menus, Subprocesses, Specifiers, Top
8765 A menu is set by setting the value of the variable
8766 @code{current-menubar} (which may be buffer-local) and then calling
8767 @code{set-menubar-dirty-flag} to signal a change. This will cause the
8768 menu to be redrawn at the next redisplay. The format of the data in
8769 @code{current-menubar} is described in @file{menubar.c}.
8771 Internally the data in current-menubar is parsed into a tree of
8772 @code{widget_value's} (defined in @file{lwlib.h}); this is accomplished
8773 by the recursive function @code{menu_item_descriptor_to_widget_value()},
8774 called by @code{compute_menubar_data()}. Such a tree is deallocated
8775 using @code{free_widget_value()}.
8777 @code{update_screen_menubars()} is one of the external entry points.
8778 This checks to see, for each screen, if that screen's menubar needs to
8779 be updated. This is the case if
8783 @code{set-menubar-dirty-flag} was called since the last redisplay. (This
8784 function sets the C variable menubar_has_changed.)
8786 The buffer displayed in the screen has changed.
8788 The screen has no menubar currently displayed.
8791 @code{set_screen_menubar()} is called for each such screen. This
8792 function calls @code{compute_menubar_data()} to create the tree of
8793 widget_value's, then calls @code{lw_create_widget()},
8794 @code{lw_modify_all_widgets()}, and/or @code{lw_destroy_all_widgets()}
8795 to create the X-Toolkit widget associated with the menu.
8797 @code{update_psheets()}, the other external entry point, actually
8798 changes the menus being displayed. It uses the widgets fixed by
8799 @code{update_screen_menubars()} and calls various X functions to ensure
8800 that the menus are displayed properly.
8802 The menubar widget is set up so that @code{pre_activate_callback()} is
8803 called when the menu is first selected (i.e. mouse button goes down),
8804 and @code{menubar_selection_callback()} is called when an item is
8805 selected. @code{pre_activate_callback()} calls the function in
8806 activate-menubar-hook, which can change the menubar (this is described
8807 in @file{menubar.c}). If the menubar is changed,
8808 @code{set_screen_menubars()} is called.
8809 @code{menubar_selection_callback()} enqueues a menu event, putting in it
8810 a function to call (either @code{eval} or @code{call-interactively}) and
8811 its argument, which is the callback function or form given in the menu's
8814 @node Subprocesses, Interface to X Windows, Menus, Top
8815 @chapter Subprocesses
8817 The fields of a process are:
8821 A string, the name of the process.
8824 A list containing the command arguments that were used to start this
8828 A function used to accept output from the process instead of a buffer,
8832 A function called whenever the process receives a signal, or @code{nil}.
8835 The associated buffer of the process.
8838 An integer, the Unix process @sc{id}.
8841 A flag, non-@code{nil} if this is really a child process.
8842 It is @code{nil} for a network connection.
8845 A marker indicating the position of the end of the last output from this
8846 process inserted into the buffer. This is often but not always the end
8849 @item kill_without_query
8850 If this is non-@code{nil}, killing XEmacs while this process is still
8851 running does not ask for confirmation about killing the process.
8853 @item raw_status_low
8854 @itemx raw_status_high
8855 These two fields record 16 bits each of the process status returned by
8856 the @code{wait} system call.
8859 The process status, as @code{process-status} should return it.
8863 If these two fields are not equal, a change in the status of the process
8864 needs to be reported, either by running the sentinel or by inserting a
8865 message in the process buffer.
8868 Non-@code{nil} if communication with the subprocess uses a @sc{pty};
8869 @code{nil} if it uses a pipe.
8872 The file descriptor for input from the process.
8875 The file descriptor for output to the process.
8878 The file descriptor for the terminal that the subprocess is using. (On
8879 some systems, there is no need to record this, so the value is
8883 The name of the terminal that the subprocess is using,
8884 or @code{nil} if it is using pipes.
8887 @node Interface to X Windows, Index , Subprocesses, Top
8888 @chapter Interface to X Windows
8894 @c Print the tables of contents