1 This is ../info/internals.info, produced by makeinfo version 4.0 from
2 internals/internals.texi.
4 INFO-DIR-SECTION XEmacs Editor
6 * Internals: (internals). XEmacs Internals Manual.
9 Copyright (C) 1992 - 1996 Ben Wing. Copyright (C) 1996, 1997 Sun
10 Microsystems. Copyright (C) 1994 - 1998 Free Software Foundation.
11 Copyright (C) 1994, 1995 Board of Trustees, University of Illinois.
13 Permission is granted to make and distribute verbatim copies of this
14 manual provided the copyright notice and this permission notice are
15 preserved on all copies.
17 Permission is granted to copy and distribute modified versions of
18 this manual under the conditions for verbatim copying, provided that the
19 entire resulting derived work is distributed under the terms of a
20 permission notice identical to this one.
22 Permission is granted to copy and distribute translations of this
23 manual into another language, under the above conditions for modified
24 versions, except that this permission notice may be stated in a
25 translation approved by the Foundation.
27 Permission is granted to copy and distribute modified versions of
28 this manual under the conditions for verbatim copying, provided also
29 that the section entitled "GNU General Public License" is included
30 exactly as in the original, and provided that the entire resulting
31 derived work is distributed under the terms of a permission notice
32 identical to this one.
34 Permission is granted to copy and distribute translations of this
35 manual into another language, under the above conditions for modified
36 versions, except that the section entitled "GNU General Public License"
37 may be included in a translation approved by the Free Software
38 Foundation instead of in the original English.
41 File: internals.info, Node: Modules for other Display-Related Lisp Objects, Next: Modules for the Redisplay Mechanism, Prev: Modules for the Basic Displayable Lisp Objects, Up: A Summary of the Various XEmacs Modules
43 Modules for other Display-Related Lisp Objects
44 ==============================================
88 This file provides C support for syntax highlighting--i.e.
89 highlighting different syntactic constructs of a source file in
90 different colors, for easy reading. The C support is provided so that
98 These modules decode GIF-format image files, for use with glyphs.
99 These files were removed due to Unisys patent infringement concerns.
102 File: internals.info, Node: Modules for the Redisplay Mechanism, Next: Modules for Interfacing with the File System, Prev: Modules for other Display-Related Lisp Objects, Up: A Summary of the Various XEmacs Modules
104 Modules for the Redisplay Mechanism
105 ===================================
114 These files provide the redisplay mechanism. As with many other
115 subsystems in XEmacs, there is a clean separation between the general
116 and device-specific support.
118 `redisplay.c' contains the bulk of the redisplay engine. These
119 functions update the redisplay structures (which describe how the screen
120 is to appear) to reflect any changes made to the state of any
121 displayable objects (buffer, frame, window, etc.) since the last time
122 that redisplay was called. These functions are highly optimized to
123 avoid doing more work than necessary (since redisplay is called
124 extremely often and is potentially a huge time sink), and depend heavily
125 on notifications from the objects themselves that changes have occurred,
126 so that redisplay doesn't explicitly have to check each possible object.
127 The redisplay mechanism also contains a great deal of caching to further
128 speed things up; some of this caching is contained within the various
131 `redisplay-output.c' goes through the redisplay structures and
132 converts them into calls to device-specific methods to actually output
135 `redisplay-x.c' and `redisplay-tty.c' are two implementations of
136 these redisplay output methods, for X frames and TTY frames,
141 This module contains various functions and Lisp primitives for
142 converting between buffer positions and screen positions. These
143 functions call the redisplay mechanism to do most of the work, and then
144 examine the redisplay structures to get the necessary information. This
151 These files contain functions for working with the termcap
152 (BSD-style) and terminfo (System V style) databases of terminal
153 capabilities and escape sequences, used when XEmacs is displaying in a
159 These files provide some miscellaneous TTY-output functions and
160 should probably be merged into `redisplay-tty.c'.
163 File: internals.info, Node: Modules for Interfacing with the File System, Next: Modules for Other Aspects of the Lisp Interpreter and Object System, Prev: Modules for the Redisplay Mechanism, Up: A Summary of the Various XEmacs Modules
165 Modules for Interfacing with the File System
166 ============================================
171 These modules implement the "stream" Lisp object type. This is an
172 internal-only Lisp object that implements a generic buffering stream.
173 The idea is to provide a uniform interface onto all sources and sinks of
174 data, including file descriptors, stdio streams, chunks of memory, Lisp
175 buffers, Lisp strings, etc. That way, I/O functions can be written to
176 the stream interface and can transparently handle all possible sources
177 and sinks. (For example, the `read' function can read data from a
178 file, a string, a buffer, or even a function that is called repeatedly
179 to return data, without worrying about where the data is coming from or
180 what-size chunks it is returned in.)
182 Note that in the C code, streams are called "lstreams" (for "Lisp
183 streams") to distinguish them from other kinds of streams, e.g. stdio
184 streams and C++ I/O streams.
186 Similar to other subsystems in XEmacs, lstreams are separated into
187 generic functions and a set of methods for the different types of
188 lstreams. `lstream.c' provides implementations of many different types
189 of streams; others are provided, e.g., in `file-coding.c'.
193 This implements the basic primitives for interfacing with the file
194 system. This includes primitives for reading files into buffers,
195 writing buffers into files, checking for the presence or accessibility
196 of files, canonicalizing file names, etc. Note that these primitives
197 are usually not invoked directly by the user: There is a great deal of
198 higher-level Lisp code that implements the user commands such as
199 `find-file' and `save-buffer'. This is similar to the distinction
200 between the lower-level primitives in `editfns.c' and the higher-level
201 user commands in `commands.c' and `simple.el'.
205 This file provides functions for detecting clashes between different
206 processes (e.g. XEmacs and some external process, or two different
207 XEmacs processes) modifying the same file. (XEmacs can optionally use
208 the `lock/' subdirectory to provide a form of "locking" between
209 different XEmacs processes.) This module is also used by the low-level
210 functions in `insdel.c' to ensure that, if the first modification is
211 being made to a buffer whose corresponding file has been externally
212 modified, the user is made aware of this so that the buffer can be
213 synched up with the external changes if necessary.
217 This file provides some miscellaneous functions that construct a
218 `rwxr-xr-x'-type permissions string (as might appear in an `ls'-style
219 directory listing) given the information returned by the `stat()'
225 These files implement the XEmacs interface to directory searching.
226 This includes a number of primitives for determining the files in a
227 directory and for doing filename completion. (Remember that generic
228 completion is handled by a different mechanism, in `minibuf.c'.)
230 `ndir.h' is a header file used for the directory-searching emulation
231 functions provided in `sysdep.c' (see section J below), for systems
232 that don't provide any directory-searching functions. (On those
233 systems, directories can be read directly as files, and parsed.)
237 This file provides an implementation of the `realpath()' function
238 for expanding symbolic links, on systems that don't implement it or have
239 a broken implementation.
242 File: internals.info, Node: Modules for Other Aspects of the Lisp Interpreter and Object System, Next: Modules for Interfacing with the Operating System, Prev: Modules for Interfacing with the File System, Up: A Summary of the Various XEmacs Modules
244 Modules for Other Aspects of the Lisp Interpreter and Object System
245 ===================================================================
252 These files provide two implementations of hash tables. Files
253 `hash.c' and `hash.h' provide a generic C implementation of hash tables
254 which can stand independently of XEmacs. Files `elhash.c' and
255 `elhash.h' provide a separate implementation of hash tables that can
256 store only Lisp objects, and knows about Lispy things like garbage
257 collection, and implement the "hash-table" Lisp object type.
262 This module implements the "specifier" Lisp object type. This is
263 primarily used for displayable properties, and allows for values that
264 are specific to a particular buffer, window, frame, device, or device
265 class, as well as a default value existing. This is used, for example,
266 to control the height of the horizontal scrollbar or the appearance of
267 the `default', `bold', or other faces. The specifier object consists
268 of a number of specifications, each of which maps from a buffer,
269 window, etc. to a value. The function `specifier-instance' looks up a
270 value given a window (from which a buffer, frame, and device can be
277 `chartab.c' and `chartab.h' implement the "char table" Lisp object
278 type, which maps from characters or certain sorts of character ranges
279 to Lisp objects. The implementation of this object type is optimized
280 for the internal representation of characters. Char tables come in
281 different types, which affect the allowed object types to which a
282 character can be mapped and also dictate certain other properties of
285 `casetab.c' implements one sort of char table, the "case table",
286 which maps characters to other characters of possibly different case.
287 These are used by XEmacs to implement case-changing primitives and to
288 do case-insensitive searching.
293 This module implements "syntax tables", another sort of char table
294 that maps characters into syntax classes that define the syntax of these
295 characters (e.g. a parenthesis belongs to a class of `open' characters
296 that have corresponding `close' characters and can be nested). This
297 module also implements the Lisp "scanner", a set of primitives for
298 scanning over text based on syntax tables. This is used, for example,
299 to find the matching parenthesis in a command such as `forward-sexp',
300 and by `font-lock.c' to locate quoted strings, comments, etc.
304 This module implements various Lisp primitives for upcasing,
305 downcasing and capitalizing strings or regions of buffers.
309 This module implements the "range table" Lisp object type, which
310 provides for a mapping from ranges of integers to arbitrary Lisp
316 This module implements the "opaque" Lisp object type, an
317 internal-only Lisp object that encapsulates an arbitrary block of memory
318 so that it can be managed by the Lisp allocation system. To create an
319 opaque object, you call `make_opaque()', passing a pointer to a block
320 of memory. An object is created that is big enough to hold the memory,
321 which is copied into the object's storage. The object will then stick
322 around as long as you keep pointers to it, after which it will be
323 automatically reclaimed.
325 Opaque objects can also have an arbitrary "mark method" associated
326 with them, in case the block of memory contains other Lisp objects that
327 need to be marked for garbage-collection purposes. (If you need other
328 object methods, such as a finalize method, you should just go ahead and
329 create a new Lisp object type--it's not hard.)
333 This function provides a few primitives for doing dynamic
334 abbreviation expansion. In XEmacs, most of the code for this has been
335 moved into Lisp. Some C code remains for speed and because the
336 primitive `self-insert-command' (which is executed for all
337 self-inserting characters) hooks into the abbrev mechanism.
338 (`self-insert-command' is itself in C only for speed.)
342 This function provides primitives for retrieving the documentation
343 strings of functions and variables. These documentation strings contain
344 certain special markers that get dynamically expanded (e.g. a
345 reverse-lookup is performed on some named functions to retrieve their
346 current key bindings). Some documentation strings (in particular, for
347 the built-in primitives and pre-loaded Lisp functions) are stored
348 externally in a file `DOC' in the `lib-src/' directory and need to be
349 fetched from that file. (Part of the build stage involves building this
350 file, and another part involves constructing an index for this file and
351 embedding it into the executable, so that the functions in `doc.c' do
352 not have to search the entire `DOC' file to find the appropriate
353 documentation string.)
357 This function provides a Lisp primitive that implements the MD5
358 secure hashing scheme, used to create a large hash value of a string of
359 data such that the data cannot be derived from the hash value. This is
360 used for various security applications on the Internet.
363 File: internals.info, Node: Modules for Interfacing with the Operating System, Next: Modules for Interfacing with X Windows, Prev: Modules for Other Aspects of the Lisp Interpreter and Object System, Up: A Summary of the Various XEmacs Modules
365 Modules for Interfacing with the Operating System
366 =================================================
372 These modules allow XEmacs to spawn and communicate with subprocesses
373 and network connections.
375 `callproc.c' implements (through the `call-process' primitive) what
376 are called "synchronous subprocesses". This means that XEmacs runs a
377 program, waits till it's done, and retrieves its output. A typical
378 example might be calling the `ls' program to get a directory listing.
380 `process.c' and `process.h' implement "asynchronous subprocesses".
381 This means that XEmacs starts a program and then continues normally,
382 not waiting for the process to finish. Data can be sent to the process
383 or retrieved from it as it's running. This is used for the `shell'
384 command (which provides a front end onto a shell program such as
385 `csh'), the mail and news readers implemented in XEmacs, etc. The
386 result of calling `start-process' to start a subprocess is a process
387 object, a particular kind of object used to communicate with the
388 subprocess. You can send data to the process by passing the process
389 object and the data to `send-process', and you can specify what happens
390 to data retrieved from the process by setting properties of the process
391 object. (When the process sends data, XEmacs receives a process event,
392 which says that there is data ready. When `dispatch-event' is called
393 on this event, it reads the data from the process and does something
394 with it, as specified by the process object's properties. Typically,
395 this means inserting the data into a buffer or calling a function.)
396 Another property of the process object is called the "sentinel", which
397 is a function that is called when the process terminates.
399 Process objects are also used for network connections (connections
400 to a process running on another machine). Network connections are
401 started with `open-network-stream' but otherwise work just like
407 These modules implement most of the low-level, messy operating-system
408 interface code. This includes various device control (ioctl) operations
409 for file descriptors, TTY's, pseudo-terminals, etc. (usually this stuff
410 is fairly system-dependent; thus the name of this module), and emulation
411 of standard library functions and system calls on systems that don't
412 provide them or have broken versions.
424 These header files provide consistent interfaces onto
425 system-dependent header files and system calls. The idea is that,
426 instead of including a standard header file like `<sys/param.h>' (which
427 may or may not exist on various systems) or having to worry about
428 whether all system provide a particular preprocessor constant, or
429 having to deal with the four different paradigms for manipulating
430 signals, you just include the appropriate `sys*.h' header file, which
431 includes all the right system header files, defines and missing
432 preprocessor constants, provides a uniform interface onto system calls,
435 `sysdir.h' provides a uniform interface onto directory-querying
436 functions. (In some cases, this is in conjunction with emulation
437 functions in `sysdep.c'.)
439 `sysfile.h' includes all the necessary header files for standard
440 system calls (e.g. `read()'), ensures that all necessary `open()' and
441 `stat()' preprocessor constants are defined, and possibly (usually)
442 substitutes sugared versions of `read()', `write()', etc. that
443 automatically restart interrupted I/O operations.
445 `sysfloat.h' includes the necessary header files for floating-point
448 `sysproc.h' includes the necessary header files for calling
449 `select()', `fork()', `execve()', socket operations, and the like, and
450 ensures that the `FD_*()' macros for descriptor-set manipulations are
453 `syspwd.h' includes the necessary header files for obtaining
454 information from `/etc/passwd' (the functions are emulated under VMS).
456 `syssignal.h' includes the necessary header files for
457 signal-handling and provides a uniform interface onto the different
458 signal-handling and signal-blocking paradigms.
460 `systime.h' includes the necessary header files and provides uniform
461 interfaces for retrieving the time of day, setting file
462 access/modification times, getting the amount of time used by the XEmacs
465 `systty.h' buffers against the infinitude of different ways of
468 `syswait.h' provides a uniform way of retrieving the exit status
469 from a `wait()'ed-on process (some systems use a union, others use an
482 These files implement the ability to play various sounds on some
483 types of computers. You have to configure your XEmacs with sound
484 support in order to get this capability.
486 `sound.c' provides the generic interface. It implements various
487 Lisp primitives and variables that let you specify which sounds should
488 be played in certain conditions. (The conditions are identified by
489 symbols, which are passed to `ding' to make a sound. Various standard
490 functions call this function at certain times; if sound support does
491 not exist, a simple beep results.
493 `sgiplay.c', `sunplay.c', `hpplay.c', and `linuxplay.c' interface to
494 the machine's speaker for various different kind of machines. This is
495 called "native" sound.
497 `nas.c' interfaces to a computer somewhere else on the network using
498 the NAS (Network Audio Server) protocol, playing sounds on that
499 machine. This allows you to run XEmacs on a remote machine, with its
500 display set to your local machine, and have the sounds be made on your
501 local machine, provided that you have a NAS server running on your local
504 `libsst.c', `libsst.h', and `libst.h' provide some additional
505 functions for playing sound on a Sun SPARC but are not currently in use.
510 These two modules implement an interface to the ToolTalk protocol,
511 which is an interprocess communication protocol implemented on some
512 versions of Unix. ToolTalk is a high-level protocol that allows
513 processes to register themselves as providers of particular services;
514 other processes can then request a service without knowing or caring
515 exactly who is providing the service. It is similar in spirit to the
516 DDE protocol provided under Microsoft Windows. ToolTalk is a part of
517 the new CDE (Common Desktop Environment) specification and is used to
518 connect the parts of the SPARCWorks development environment.
522 This module provides the ability to retrieve the system's current
523 load average. (The way to do this is highly system-specific,
524 unfortunately, and requires a lot of special-case code.)
528 This module provides a small amount of code used internally at Sun to
529 keep statistics on the usage of XEmacs.
536 These files provide replacement functions and prototypes to fix
537 numerous bugs in early releases of SunOS 4.1.
541 This module provides some terminal-control code necessary on
542 versions of AIX prior to 4.1.
545 File: internals.info, Node: Modules for Interfacing with X Windows, Next: Modules for Internationalization, Prev: Modules for Interfacing with the Operating System, Up: A Summary of the Various XEmacs Modules
547 Modules for Interfacing with X Windows
548 ======================================
552 A file generated from `Emacs.ad', which contains XEmacs-supplied
553 fallback resources (so that XEmacs has pretty defaults).
559 These modules implement an Xt widget class that encapsulates a frame.
560 This is for ease in integrating with Xt. The EmacsFrame widget covers
561 the entire X window except for the menubar; the scrollbars are
562 positioned on top of the EmacsFrame widget.
564 *Warning:* Abandon hope, all ye who enter here. This code took an
565 ungodly amount of time to get right, and is likely to fall apart
566 mercilessly at the slightest change. Such is life under Xt.
572 These modules implement a simple Xt manager (i.e. composite) widget
573 class that simply lets its children set whatever geometry they want.
574 It's amazing that Xt doesn't provide this standardly, but on second
575 thought, it makes sense, considering how amazingly broken Xt is.
582 These modules implement two Xt widget classes that are subclasses of
583 the TopLevelShell and TransientShell classes. This is necessary to deal
584 with more brokenness that Xt has sadistically thrust onto the backs of
590 These modules provide functions for maintenance and caching of GC's
591 (graphics contexts) under the X Window System. This code is junky and
592 needs to be rewritten.
599 This module provides an interface to the X Window System's concept of
600 "selections", the standard way for X applications to communicate with
608 These header files are similar in spirit to the `sys*.h' files and
609 buffer against different implementations of Xt and Motif.
611 * `xintrinsic.h' should be included in place of `<Intrinsic.h>'.
613 * `xintrinsicp.h' should be included in place of `<IntrinsicP.h>'.
615 * `xmmanagerp.h' should be included in place of `<XmManagerP.h>'.
617 * `xmprimitivep.h' should be included in place of `<XmPrimitiveP.h>'.
622 These files provide an emulation of the Xmu library for those systems
623 (i.e. HPUX) that don't provide it as a standard part of X.
625 ExternalClient-Xlib.c
637 These files provide the "external widget" interface, which allows an
638 XEmacs frame to appear as a widget in another application. To do this,
639 you have to configure with `--external-widget'.
641 `ExternalShell*' provides the server (XEmacs) side of the connection.
643 `ExternalClient*' provides the client (other application) side of
644 the connection. These files are not compiled into XEmacs but are
645 compiled into libraries that are then linked into your application.
647 `extw-*' is common code that is used for both the client and server.
649 Don't touch this code; something is liable to break if you do.
652 File: internals.info, Node: Modules for Internationalization, Prev: Modules for Interfacing with X Windows, Up: A Summary of the Various XEmacs Modules
654 Modules for Internationalization
655 ================================
668 These files implement the MULE (Asian-language) support. Note that
669 MULE actually provides a general interface for all sorts of languages,
670 not just Asian languages (although they are generally the most
671 complicated to support). This code is still in beta.
673 `mule-charset.*' and `file-coding.*' provide the heart of the XEmacs
674 MULE support. `mule-charset.*' implements the "charset" Lisp object
675 type, which encapsulates a character set (an ordered one- or
676 two-dimensional set of characters, such as US ASCII or JISX0208 Japanese
679 `file-coding.*' implements the "coding-system" Lisp object type,
680 which encapsulates a method of converting between different encodings.
681 An encoding is a representation of a stream of characters, possibly
682 from multiple character sets, using a stream of bytes or words, and
683 defines (e.g.) which escape sequences are used to specify particular
684 character sets, how the indices for a character are converted into bytes
685 (sometimes this involves setting the high bit; sometimes complicated
686 rearranging of the values takes place, as in the Shift-JIS encoding),
689 `mule-ccl.c' provides the CCL (Code Conversion Language)
690 interpreter. CCL is similar in spirit to Lisp byte code and is used to
691 implement converters for custom encodings.
693 `mule-canna.c' and `mule-wnnfns.c' implement interfaces to external
694 programs used to implement the Canna and WNN input methods,
695 respectively. This is currently in beta.
697 `mule-mcpath.c' provides some functions to allow for pathnames
698 containing extended characters. This code is fragmentary, obsolete, and
699 completely non-working. Instead, PATHNAME-CODING-SYSTEM is used to
700 specify conversions of names of files and directories. The standard C
701 I/O functions like `open()' are wrapped so that conversion occurs
704 `mule.c' provides a few miscellaneous things that should probably be
709 This provides some miscellaneous internationalization code for
710 implementing message translation and interfacing to the Ximp input
711 method. None of this code is currently working.
715 This contains leftover code from an earlier implementation of
716 Asian-language support, and is not currently used.
719 File: internals.info, Node: Allocation of Objects in XEmacs Lisp, Next: Dumping, Prev: A Summary of the Various XEmacs Modules, Up: Top
721 Allocation of Objects in XEmacs Lisp
722 ************************************
726 * Introduction to Allocation::
727 * Garbage Collection::
729 * Garbage Collection - Step by Step::
730 * Integers and Characters::
731 * Allocation from Frob Blocks::
733 * Low-level allocation::
740 * Compiled Function::
743 File: internals.info, Node: Introduction to Allocation, Next: Garbage Collection, Up: Allocation of Objects in XEmacs Lisp
745 Introduction to Allocation
746 ==========================
748 Emacs Lisp, like all Lisps, has garbage collection. This means that
749 the programmer never has to explicitly free (destroy) an object; it
750 happens automatically when the object becomes inaccessible. Most
751 experts agree that garbage collection is a necessity in a modern,
752 high-level language. Its omission from C stems from the fact that C was
753 originally designed to be a nice abstract layer on top of assembly
754 language, for writing kernels and basic system utilities rather than
757 Lisp objects can be created by any of a number of Lisp primitives.
758 Most object types have one or a small number of basic primitives for
759 creating objects. For conses, the basic primitive is `cons'; for
760 vectors, the primitives are `make-vector' and `vector'; for symbols,
761 the primitives are `make-symbol' and `intern'; etc. Some Lisp objects,
762 especially those that are primarily used internally, have no
763 corresponding Lisp primitives. Every Lisp object, though, has at least
764 one C primitive for creating it.
766 Recall from section (VII) that a Lisp object, as stored in a 32-bit
767 or 64-bit word, has a few tag bits, and a "value" that occupies the
768 remainder of the bits. We can separate the different Lisp object types
769 into three broad categories:
771 * (a) Those for whom the value directly represents the contents of
772 the Lisp object. Only two types are in this category: integers and
773 characters. No special allocation or garbage collection is
774 necessary for such objects. Lisp objects of these types do not
775 need to be `GCPRO'ed.
777 In the remaining two categories, the type is stored in the object
778 itself. The tag for all such objects is the generic "lrecord"
779 (Lisp_Type_Record) tag. The first bytes of the object's structure are
780 an integer (actually a char) characterising the object's type and some
781 flags, in particular the mark bit used for garbage collection. A
782 structure describing the type is accessible thru the
783 lrecord_implementation_table indexed with said integer. This structure
784 includes the method pointers and a pointer to a string naming the type.
786 * (b) Those lrecords that are allocated in frob blocks (see above).
787 This includes the objects that are most common and relatively
788 small, and includes conses, strings, subrs, floats, compiled
789 functions, symbols, extents, events, and markers. With the
790 cleanup of frob blocks done in 19.12, it's not terribly hard to
791 add more objects to this category, but it's a bit trickier than
792 adding an object type to type (c) (esp. if the object needs a
793 finalization method), and is not likely to save much space unless
794 the object is small and there are many of them. (In fact, if there
795 are very few of them, it might actually waste space.)
797 * (c) Those lrecords that are individually `malloc()'ed. These are
798 called "lcrecords". All other types are in this category. Adding
799 a new type to this category is comparatively easy, and all types
800 added since 19.8 (when the current allocation scheme was devised,
801 by Richard Mlynarik), with the exception of the character type,
802 have been in this category.
804 Note that bit vectors are a bit of a special case. They are simple
805 lrecords as in category (b), but are individually `malloc()'ed like
806 vectors. You can basically view them as exactly like vectors except
807 that their type is stored in lrecord fashion rather than in
808 directly-tagged fashion.
811 File: internals.info, Node: Garbage Collection, Next: GCPROing, Prev: Introduction to Allocation, Up: Allocation of Objects in XEmacs Lisp
816 Garbage collection is simple in theory but tricky to implement.
817 Emacs Lisp uses the oldest garbage collection method, called "mark and
818 sweep". Garbage collection begins by starting with all accessible
819 locations (i.e. all variables and other slots where Lisp objects might
820 occur) and recursively traversing all objects accessible from those
821 slots, marking each one that is found. We then go through all of
822 memory and free each object that is not marked, and unmarking each
823 object that is marked. Note that "all of memory" means all currently
824 allocated objects. Traversing all these objects means traversing all
825 frob blocks, all vectors (which are chained in one big list), and all
826 lcrecords (which are likewise chained).
828 Garbage collection can be invoked explicitly by calling
829 `garbage-collect' but is also called automatically by `eval', once a
830 certain amount of memory has been allocated since the last garbage
831 collection (according to `gc-cons-threshold').
834 File: internals.info, Node: GCPROing, Next: Garbage Collection - Step by Step, Prev: Garbage Collection, Up: Allocation of Objects in XEmacs Lisp
839 `GCPRO'ing is one of the ugliest and trickiest parts of Emacs
840 internals. The basic idea is that whenever garbage collection occurs,
841 all in-use objects must be reachable somehow or other from one of the
842 roots of accessibility. The roots of accessibility are:
844 1. All objects that have been `staticpro()'d or
845 `staticpro_nodump()'ed. This is used for any global C variables
846 that hold Lisp objects. A call to `staticpro()' happens implicitly
847 as a result of any symbols declared with `defsymbol()' and any
848 variables declared with `DEFVAR_FOO()'. You need to explicitly
849 call `staticpro()' (in the `vars_of_foo()' method of a module) for
850 other global C variables holding Lisp objects. (This typically
851 includes internal lists and such things.). Use
852 `staticpro_nodump()' only in the rare cases when you do not want
853 the pointed variable to be saved at dump time but rather recompute
856 Note that `obarray' is one of the `staticpro()'d things.
857 Therefore, all functions and variables get marked through this.
859 2. Any shadowed bindings that are sitting on the `specpdl' stack.
861 3. Any objects sitting in currently active (Lisp) stack frames,
862 catches, and condition cases.
864 4. A couple of special-case places where active objects are located.
866 5. Anything currently marked with `GCPRO'.
868 Marking with `GCPRO' is necessary because some C functions (quite a
869 lot, in fact), allocate objects during their operation. Quite
870 frequently, there will be no other pointer to the object while the
871 function is running, and if a garbage collection occurs and the object
872 needs to be referenced again, bad things will happen. The solution is
873 to mark those objects with `GCPRO'. Unfortunately this is easy to
874 forget, and there is basically no way around this problem. Here are
877 1. For every `GCPRON', there have to be declarations of `struct gcpro
878 gcpro1, gcpro2', etc.
880 2. You _must_ `UNGCPRO' anything that's `GCPRO'ed, and you _must not_
881 `UNGCPRO' if you haven't `GCPRO'ed. Getting either of these wrong
882 will lead to crashes, often in completely random places unrelated
883 to where the problem lies.
885 3. The way this actually works is that all currently active `GCPRO's
886 are chained through the `struct gcpro' local variables, with the
887 variable `gcprolist' pointing to the head of the list and the nth
888 local `gcpro' variable pointing to the first `gcpro' variable in
889 the next enclosing stack frame. Each `GCPRO'ed thing is an
890 lvalue, and the `struct gcpro' local variable contains a pointer to
891 this lvalue. This is why things will mess up badly if you don't
892 pair up the `GCPRO's and `UNGCPRO's--you will end up with
893 `gcprolist's containing pointers to `struct gcpro's or local
894 `Lisp_Object' variables in no-longer-active stack frames.
896 4. It is actually possible for a single `struct gcpro' to protect a
897 contiguous array of any number of values, rather than just a
898 single lvalue. To effect this, call `GCPRON' as usual on the
899 first object in the array and then set `gcproN.nvars'.
901 5. *Strings are relocated.* What this means in practice is that the
902 pointer obtained using `XSTRING_DATA()' is liable to change at any
903 time, and you should never keep it around past any function call,
904 or pass it as an argument to any function that might cause a
905 garbage collection. This is why a number of functions accept
906 either a "non-relocatable" `char *' pointer or a relocatable Lisp
907 string, and only access the Lisp string's data at the very last
908 minute. In some cases, you may end up having to `alloca()' some
909 space and copy the string's data into it.
911 6. By convention, if you have to nest `GCPRO''s, use `NGCPRON' (along
912 with `struct gcpro ngcpro1, ngcpro2', etc.), `NNGCPRON', etc.
913 This avoids compiler warnings about shadowed locals.
915 7. It is _always_ better to err on the side of extra `GCPRO's rather
916 than too few. The extra cycles spent on this are almost never
917 going to make a whit of difference in the speed of anything.
919 8. The general rule to follow is that caller, not callee, `GCPRO's.
920 That is, you should not have to explicitly `GCPRO' any Lisp objects
921 that are passed in as parameters.
923 One exception from this rule is if you ever plan to change the
924 parameter value, and store a new object in it. In that case, you
925 _must_ `GCPRO' the parameter, because otherwise the new object
926 will not be protected.
928 So, if you create any Lisp objects (remember, this happens in all
929 sorts of circumstances, e.g. with `Fcons()', etc.), you are
930 responsible for `GCPRO'ing them, unless you are _absolutely sure_
931 that there's no possibility that a garbage-collection can occur
932 while you need to use the object. Even then, consider `GCPRO'ing.
934 9. A garbage collection can occur whenever anything calls `Feval', or
935 whenever a QUIT can occur where execution can continue past this.
936 (Remember, this is almost anywhere.)
938 10. If you have the _least smidgeon of doubt_ about whether you need
939 to `GCPRO', you should `GCPRO'.
941 11. Beware of `GCPRO'ing something that is uninitialized. If you have
942 any shade of doubt about this, initialize all your variables to
945 12. Be careful of traps, like calling `Fcons()' in the argument to
946 another function. By the "caller protects" law, you should be
947 `GCPRO'ing the newly-created cons, but you aren't. A certain
948 number of functions that are commonly called on freshly created
949 stuff (e.g. `nconc2()', `Fsignal()'), break the "caller protects"
950 law and go ahead and `GCPRO' their arguments so as to simplify
951 things, but make sure and check if it's OK whenever doing
954 13. Once again, remember to `GCPRO'! Bugs resulting from insufficient
955 `GCPRO'ing are intermittent and extremely difficult to track down,
956 often showing up in crashes inside of `garbage-collect' or in
957 weirdly corrupted objects or even in incorrect values in a totally
958 different section of code.
960 Given the extremely error-prone nature of the `GCPRO' scheme, and
961 the difficulties in tracking down, it should be considered a deficiency
962 in the XEmacs code. A solution to this problem would involve
963 implementing so-called "conservative" garbage collection for the C
964 stack. That involves looking through all of stack memory and treating
965 anything that looks like a reference to an object as a reference. This
966 will result in a few objects not getting collected when they should, but
967 it obviates the need for `GCPRO'ing, and allows garbage collection to
968 happen at any point at all, such as during object allocation.
971 File: internals.info, Node: Garbage Collection - Step by Step, Next: Integers and Characters, Prev: GCPROing, Up: Allocation of Objects in XEmacs Lisp
973 Garbage Collection - Step by Step
974 =================================
979 * garbage_collect_1::
982 * sweep_lcrecords_1::
983 * compact_string_chars::
985 * sweep_bit_vectors_1::
988 File: internals.info, Node: Invocation, Next: garbage_collect_1, Up: Garbage Collection - Step by Step
993 The first thing that anyone should know about garbage collection is:
994 when and how the garbage collector is invoked. One might think that this
995 could happen every time new memory is allocated, e.g. new objects are
996 created, but this is _not_ the case. Instead, we have the following
999 The entry point of any process of garbage collection is an invocation
1000 of the function `garbage_collect_1' in file `alloc.c'. The invocation
1001 can occur _explicitly_ by calling the function `Fgarbage_collect' (in
1002 addition this function provides information about the freed memory), or
1003 can occur _implicitly_ in four different situations:
1004 1. In function `main_1' in file `emacs.c'. This function is called at
1005 each startup of xemacs. The garbage collection is invoked after all
1006 initial creations are completed, but only if a special internal
1007 error checking-constant `ERROR_CHECK_GC' is defined.
1009 2. In function `disksave_object_finalization' in file `alloc.c'. The
1010 only purpose of this function is to clear the objects from memory
1011 which need not be stored with xemacs when we dump out an
1012 executable. This is only done by `Fdump_emacs' or by
1013 `Fdump_emacs_data' respectively (both in `emacs.c'). The actual
1014 clearing is accomplished by making these objects unreachable and
1015 starting a garbage collection. The function is only used while
1018 3. In function `Feval / eval' in file `eval.c'. Each time the well
1019 known and often used function eval is called to evaluate a form,
1020 one of the first things that could happen, is a potential call of
1021 `garbage_collect_1'. There exist three global variables,
1022 `consing_since_gc' (counts the created cons-cells since the last
1023 garbage collection), `gc_cons_threshold' (a specified threshold
1024 after which a garbage collection occurs) and `always_gc'. If
1025 `always_gc' is set or if the threshold is exceeded, the garbage
1026 collection will start.
1028 4. In function `Ffuncall / funcall' in file `eval.c'. This function
1029 evaluates calls of elisp functions and works according to `Feval'.
1031 The upshot is that garbage collection can basically occur everywhere
1032 `Feval', respectively `Ffuncall', is used - either directly or through
1033 another function. Since calls to these two functions are hidden in
1034 various other functions, many calls to `garbage_collect_1' are not
1035 obviously foreseeable, and therefore unexpected. Instances where they
1036 are used that are worth remembering are various elisp commands, as for
1037 example `or', `and', `if', `cond', `while', `setq', etc., miscellaneous
1038 `gui_item_...' functions, everything related to `eval' (`Feval_buffer',
1039 `call0', ...) and inside `Fsignal'. The latter is used to handle
1040 signals, as for example the ones raised by every `QUIT'-macro triggered
1041 after pressing Ctrl-g.