1 This is ../info/internals.info, produced by makeinfo version 4.0 from
2 internals/internals.texi.
4 INFO-DIR-SECTION XEmacs Editor
6 * Internals: (internals). XEmacs Internals Manual.
9 Copyright (C) 1992 - 1996 Ben Wing. Copyright (C) 1996, 1997 Sun
10 Microsystems. Copyright (C) 1994 - 1998 Free Software Foundation.
11 Copyright (C) 1994, 1995 Board of Trustees, University of Illinois.
13 Permission is granted to make and distribute verbatim copies of this
14 manual provided the copyright notice and this permission notice are
15 preserved on all copies.
17 Permission is granted to copy and distribute modified versions of
18 this manual under the conditions for verbatim copying, provided that the
19 entire resulting derived work is distributed under the terms of a
20 permission notice identical to this one.
22 Permission is granted to copy and distribute translations of this
23 manual into another language, under the above conditions for modified
24 versions, except that this permission notice may be stated in a
25 translation approved by the Foundation.
27 Permission is granted to copy and distribute modified versions of
28 this manual under the conditions for verbatim copying, provided also
29 that the section entitled "GNU General Public License" is included
30 exactly as in the original, and provided that the entire resulting
31 derived work is distributed under the terms of a permission notice
32 identical to this one.
34 Permission is granted to copy and distribute translations of this
35 manual into another language, under the above conditions for modified
36 versions, except that the section entitled "GNU General Public License"
37 may be included in a translation approved by the Free Software
38 Foundation instead of in the original English.
41 File: internals.info, Node: Conversion to and from External Data, Next: General Guidelines for Writing Mule-Aware Code, Prev: Working With Character and Byte Positions, Up: Coding for Mule
43 Conversion to and from External Data
44 ------------------------------------
46 When an external function, such as a C library function, returns a
47 `char' pointer, you should almost never treat it as `Bufbyte'. This is
48 because these returned strings may contain 8bit characters which can be
49 misinterpreted by XEmacs, and cause a crash. Likewise, when exporting
50 a piece of internal text to the outside world, you should always
51 convert it to an appropriate external encoding, lest the internal stuff
52 (such as the infamous \201 characters) leak out.
54 The interface to conversion between the internal and external
55 representations of text are the numerous conversion macros defined in
56 `buffer.h'. There used to be a fixed set of external formats supported
57 by these macros, but now any coding system can be used with these
58 macros. The coding system alias mechanism is used to create the
59 following logical coding systems, which replace the fixed external
60 formats. The (dontusethis-set-symbol-value-handler) mechanism was
61 enhanced to make this possible (more work on that is needed - like
62 remove the `dontusethis-' prefix).
65 This is the simplest format and is what we use in the absence of a
66 more appropriate format. This converts according to the `binary'
69 a. On input, bytes 0-255 are converted into (implicitly Latin-1)
70 characters 0-255. A non-Mule xemacs doesn't really know about
71 different character sets and the fonts to display them, so
72 the bytes can be treated as text in different 1-byte
73 encodings by simply setting the appropriate fonts. So in a
74 sense, non-Mule xemacs is a multi-lingual editor if, for
75 example, different fonts are used to display text in
76 different buffers, faces, or windows. The specifier
77 mechanism gives the user complete control over this kind of
80 b. On output, characters 0-255 are converted into bytes 0-255
81 and other characters are converted into `~'.
84 Format used for filenames. This is user-definable via either the
85 `file-name-coding-system' or `pathname-coding-system' (now
89 Format used for the external Unix environment--`argv[]', stuff
90 from `getenv()', stuff from the `/etc/passwd' file, etc.
91 Currently this is the same as Qfile_name. The two should be
92 distinguished for clarity and possible future separation.
95 Compound-text format. This is the standard X11 format used for
96 data stored in properties, selections, and the like. This is an
97 8-bit no-lock-shift ISO2022 coding system. This is a real coding
98 system, unlike Qfile_name, which is user-definable.
100 There are two fundamental macros to convert between external and
103 `TO_INTERNAL_FORMAT' converts external data to internal format, and
104 `TO_EXTERNAL_FORMAT' converts the other way around. The arguments each
105 of these receives are a source type, a source, a sink type, a sink, and
106 a coding system (or a symbol naming a coding system).
108 A typical call looks like
109 TO_EXTERNAL_FORMAT (LISP_STRING, str, C_STRING_MALLOC, ptr, Qfile_name);
111 which means that the contents of the lisp string `str' are written
112 to a malloc'ed memory area which will be pointed to by `ptr', after the
113 function returns. The conversion will be done using the `file-name'
114 coding system, which will be controlled by the user indirectly by
115 setting or binding the variable `file-name-coding-system'.
117 Some sources and sinks require two C variables to specify. We use
118 some preprocessor magic to allow different source and sink types, and
119 even different numbers of arguments to specify different types of
122 So we can have a call that looks like
123 TO_INTERNAL_FORMAT (DATA, (ptr, len),
127 The parenthesized argument pairs are required to make the
128 preprocessor magic work.
130 Here are the different source and sink types:
132 ``DATA, (ptr, len),''
133 input data is a fixed buffer of size LEN at address PTR
135 ``ALLOCA, (ptr, len),''
136 output data is placed in an alloca()ed buffer of size LEN pointed
139 ``MALLOC, (ptr, len),''
140 output data is in a malloc()ed buffer of size LEN pointed to by PTR
142 ``C_STRING_ALLOCA, ptr,''
143 equivalent to `ALLOCA (ptr, len_ignored)' on output.
145 ``C_STRING_MALLOC, ptr,''
146 equivalent to `MALLOC (ptr, len_ignored)' on output
149 equivalent to `DATA, (ptr, strlen (ptr) + 1)' on input
151 ``LISP_STRING, string,''
152 input or output is a Lisp_Object of type string
154 ``LISP_BUFFER, buffer,''
155 output is written to `(point)' in lisp buffer BUFFER
157 ``LISP_LSTREAM, lstream,''
158 input or output is a Lisp_Object of type lstream
160 ``LISP_OPAQUE, object,''
161 input or output is a Lisp_Object of type opaque
163 Often, the data is being converted to a '\0'-byte-terminated string,
164 which is the format required by many external system C APIs. For these
165 purposes, a source type of `C_STRING' or a sink type of
166 `C_STRING_ALLOCA' or `C_STRING_MALLOC' is appropriate. Otherwise, we
167 should try to keep XEmacs '\0'-byte-clean, which means using (ptr, len)
170 The sinks to be specified must be lvalues, unless they are the lisp
171 object types `LISP_LSTREAM' or `LISP_BUFFER'.
173 For the sink types `ALLOCA' and `C_STRING_ALLOCA', the resulting
174 text is stored in a stack-allocated buffer, which is automatically
175 freed on returning from the function. However, the sink types `MALLOC'
176 and `C_STRING_MALLOC' return `xmalloc()'ed memory. The caller is
177 responsible for freeing this memory using `xfree()'.
179 Note that it doesn't make sense for `LISP_STRING' to be a source for
180 `TO_INTERNAL_FORMAT' or a sink for `TO_EXTERNAL_FORMAT'. You'll get an
181 assertion failure if you try.
184 File: internals.info, Node: General Guidelines for Writing Mule-Aware Code, Next: An Example of Mule-Aware Code, Prev: Conversion to and from External Data, Up: Coding for Mule
186 General Guidelines for Writing Mule-Aware Code
187 ----------------------------------------------
189 This section contains some general guidance on how to write
190 Mule-aware code, as well as some pitfalls you should avoid.
192 _Never use `char' and `char *'._
193 In XEmacs, the use of `char' and `char *' is almost always a
194 mistake. If you want to manipulate an Emacs character from "C",
195 use `Emchar'. If you want to examine a specific octet in the
196 internal format, use `Bufbyte'. If you want a Lisp-visible
197 character, use a `Lisp_Object' and `make_char'. If you want a
198 pointer to move through the internal text, use `Bufbyte *'. Also
199 note that you almost certainly do not need `Emchar *'.
201 _Be careful not to confuse `Charcount', `Bytecount', and `Bufpos'._
202 The whole point of using different types is to avoid confusion
203 about the use of certain variables. Lest this effect be
204 nullified, you need to be careful about using the right types.
206 _Always convert external data_
207 It is extremely important to always convert external data, because
208 XEmacs can crash if unexpected 8bit sequences are copied to its
209 internal buffers literally.
211 This means that when a system function, such as `readdir', returns
212 a string, you may need to convert it using one of the conversion
213 macros described in the previous chapter, before passing it
216 Actually, most of the basic system functions that accept
217 '\0'-terminated string arguments, like `stat()' and `open()', have
218 been *encapsulated* so that they are they `always' do internal to
219 external conversion themselves. This means you must pass
220 internally encoded data, typically the `XSTRING_DATA' of a
221 Lisp_String to these functions. This is actually a design bug,
222 since it unexpectedly changes the semantics of the system
223 functions. A better design would be to provide separate versions
224 of these system functions that accepted Lisp_Objects which were
225 lisp strings in place of their current `char *' arguments.
227 int stat_lisp (Lisp_Object path, struct stat *buf); /* Implement me */
229 Also note that many internal functions, such as `make_string',
230 accept Bufbytes, which removes the need for them to convert the
231 data they receive. This increases efficiency because that way
232 external data needs to be decoded only once, when it is read.
233 After that, it is passed around in internal format.
236 File: internals.info, Node: An Example of Mule-Aware Code, Prev: General Guidelines for Writing Mule-Aware Code, Up: Coding for Mule
238 An Example of Mule-Aware Code
239 -----------------------------
241 As an example of Mule-aware code, we will analyze the `string'
242 function, which conses up a Lisp string from the character arguments it
243 receives. Here is the definition, pasted from `alloc.c':
245 DEFUN ("string", Fstring, 0, MANY, 0, /*
246 Concatenate all the argument characters and make the result a string.
248 (int nargs, Lisp_Object *args))
250 Bufbyte *storage = alloca_array (Bufbyte, nargs * MAX_EMCHAR_LEN);
251 Bufbyte *p = storage;
253 for (; nargs; nargs--, args++)
255 Lisp_Object lisp_char = *args;
256 CHECK_CHAR_COERCE_INT (lisp_char);
257 p += set_charptr_emchar (p, XCHAR (lisp_char));
259 return make_string (storage, p - storage);
262 Now we can analyze the source line by line.
264 Obviously, string will be as long as there are arguments to the
265 function. This is why we allocate `MAX_EMCHAR_LEN' * NARGS bytes on
266 the stack, i.e. the worst-case number of bytes for NARGS `Emchar's to
269 Then, the loop checks that each element is a character, converting
270 integers in the process. Like many other functions in XEmacs, this
271 function silently accepts integers where characters are expected, for
272 historical and compatibility reasons. Unless you know what you are
273 doing, `CHECK_CHAR' will also suffice. `XCHAR (lisp_char)' extracts
274 the `Emchar' from the `Lisp_Object', and `set_charptr_emchar' stores it
275 to storage, increasing `p' in the process.
277 Other instructive examples of correct coding under Mule can be found
278 all over the XEmacs code. For starters, I recommend
279 `Fnormalize_menu_item_name' in `menubar.c'. After you have understood
280 this section of the manual and studied the examples, you can proceed
281 writing new Mule-aware code.
284 File: internals.info, Node: Techniques for XEmacs Developers, Prev: Coding for Mule, Up: Rules When Writing New C Code
286 Techniques for XEmacs Developers
287 ================================
289 To make a purified XEmacs, do: `make puremacs'. To make a
290 quantified XEmacs, do: `make quantmacs'.
292 You simply can't dump Quantified and Purified images (unless using
293 the portable dumper). Purify gets confused when xemacs frees memory in
294 one process that was allocated in a _different_ process on a different
295 machine!. Run it like so:
296 temacs -batch -l loadup.el run-temacs XEMACS-ARGS...
298 Before you go through the trouble, are you compiling with all
299 debugging and error-checking off? If not, try that first. Be warned
300 that while Quantify is directly responsible for quite a few
301 optimizations which have been made to XEmacs, doing a run which
302 generates results which can be acted upon is not necessarily a trivial
305 Also, if you're still willing to do some runs make sure you configure
306 with the `--quantify' flag. That will keep Quantify from starting to
307 record data until after the loadup is completed and will shut off
308 recording right before it shuts down (which generates enough bogus data
309 to throw most results off). It also enables three additional elisp
310 commands: `quantify-start-recording-data',
311 `quantify-stop-recording-data' and `quantify-clear-data'.
313 If you want to make XEmacs faster, target your favorite slow
314 benchmark, run a profiler like Quantify, `gprof', or `tcov', and figure
315 out where the cycles are going. Specific projects:
317 * Make the garbage collector faster. Figure out how to write an
318 incremental garbage collector.
320 * Write a compiler that takes bytecode and spits out C code.
321 Unfortunately, you will then need a C compiler and a more fully
322 developed module system.
324 * Speed up redisplay.
326 * Speed up syntax highlighting. Maybe moving some of the syntax
327 highlighting capabilities into C would make a difference.
329 * Implement tail recursion in Emacs Lisp (hard!).
331 Unfortunately, Emacs Lisp is slow, and is going to stay slow.
332 Function calls in elisp are especially expensive. Iterating over a
333 long list is going to be 30 times faster implemented in C than in Elisp.
335 Heavily used small code fragments need to be fast. The traditional
336 way to implement such code fragments in C is with macros. But macros
337 in C are known to be broken.
339 Macro arguments that are repeatedly evaluated may suffer from
340 repeated side effects or suboptimal performance.
342 Variable names used in macros may collide with caller's variables,
343 causing (at least) unwanted compiler warnings.
345 In order to solve these problems, and maintain statement semantics,
346 one should use the `do { ... } while (0)' trick while trying to
347 reference macro arguments exactly once using local variables.
349 Let's take a look at this poor macro definition:
351 #define MARK_OBJECT(obj) \
352 if (!marked_p (obj)) mark_object (obj), did_mark = 1
354 This macro evaluates its argument twice, and also fails if used like
356 if (flag) MARK_OBJECT (obj); else do_something();
358 A much better definition is
360 #define MARK_OBJECT(obj) do { \
361 Lisp_Object mo_obj = (obj); \
362 if (!marked_p (mo_obj)) \
364 mark_object (mo_obj); \
369 Notice the elimination of double evaluation by using the local
370 variable with the obscure name. Writing safe and efficient macros
371 requires great care. The one problem with macros that cannot be
372 portably worked around is, since a C block has no value, a macro used
373 as an expression rather than a statement cannot use the techniques just
374 described to avoid multiple evaluation.
376 In most cases where a macro has function semantics, an inline
377 function is a better implementation technique. Modern compiler
378 optimizers tend to inline functions even if they have no `inline'
379 keyword, and configure magic ensures that the `inline' keyword can be
380 safely used as an additional compiler hint. Inline functions used in a
381 single .c files are easy. The function must already be defined to be
382 `static'. Just add another `inline' keyword to the definition.
385 heavily_used_small_function (int arg)
390 Inline functions in header files are trickier, because we would like
391 to make the following optimization if the function is _not_ inlined
392 (for example, because we're compiling for debugging). We would like the
393 function to be defined externally exactly once, and each calling
394 translation unit would create an external reference to the function,
395 instead of including a definition of the inline function in the object
396 code of every translation unit that uses it. This optimization is
397 currently only available for gcc. But you don't have to worry about the
398 trickiness; just define your inline functions in header files using this
402 i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg);
404 i_used_to_be_a_crufty_macro_but_look_at_me_now (int arg)
409 The declaration right before the definition is to prevent warnings
410 when compiling with `gcc -Wmissing-declarations'. I consider issuing
411 this warning for inline functions a gcc bug, but the gcc maintainers
414 Every header which contains inline functions, either directly by
415 using `INLINE_HEADER' or indirectly by using `DECLARE_LRECORD' must be
416 added to `inline.c''s includes to make the optimization described above
417 work. (Optimization note: if all INLINE_HEADER functions are in fact
418 inlined in all translation units, then the linker can just discard
419 `inline.o', since it contains only unreferenced code).
421 To get started debugging XEmacs, take a look at the `.gdbinit' and
422 `.dbxrc' files in the `src' directory. See the section in the XEmacs
423 FAQ on How to Debug an XEmacs problem with a debugger.
425 After making source code changes, run `make check' to ensure that
426 you haven't introduced any regressions. If you want to make xemacs more
427 reliable, please improve the test suite in `tests/automated'.
429 Did you make sure you didn't introduce any new compiler warnings?
431 Before submitting a patch, please try compiling at least once with
433 configure --with-mule --with-union-type --error-checking=all
435 Here are things to know when you create a new source file:
437 * All `.c' files should `#include <config.h>' first. Almost all
438 `.c' files should `#include "lisp.h"' second.
440 * Generated header files should be included using the `#include
441 <...>' syntax, not the `#include "..."' syntax. The generated
444 `config.h sheap-adjust.h paths.h Emacs.ad.h'
446 The basic rule is that you should assume builds using `--srcdir'
447 and the `#include <...>' syntax needs to be used when the
448 to-be-included generated file is in a potentially different
449 directory _at compile time_. The non-obvious C rule is that
450 `#include "..."' means to search for the included file in the same
451 directory as the including file, _not_ in the current directory.
453 * Header files should _not_ include `<config.h>' and `"lisp.h"'. It
454 is the responsibility of the `.c' files that use it to do so.
457 Here is a checklist of things to do when creating a new lisp object
464 3. add definitions of `syms_of_FOO', etc. to `FOO.c'
466 4. add declarations of `syms_of_FOO', etc. to `symsinit.h'
468 5. add calls to `syms_of_FOO', etc. to `emacs.c'
470 6. add definitions of macros like `CHECK_FOO' and `FOOP' to `FOO.h'
472 7. add the new type index to `enum lrecord_type'
474 8. add a DEFINE_LRECORD_IMPLEMENTATION call to `FOO.c'
476 9. add an INIT_LRECORD_IMPLEMENTATION call to `syms_of_FOO.c'
479 File: internals.info, Node: A Summary of the Various XEmacs Modules, Next: Allocation of Objects in XEmacs Lisp, Prev: Rules When Writing New C Code, Up: Top
481 A Summary of the Various XEmacs Modules
482 ***************************************
484 This is accurate as of XEmacs 20.0.
488 * Low-Level Modules::
489 * Basic Lisp Modules::
490 * Modules for Standard Editing Operations::
491 * Editor-Level Control Flow Modules::
492 * Modules for the Basic Displayable Lisp Objects::
493 * Modules for other Display-Related Lisp Objects::
494 * Modules for the Redisplay Mechanism::
495 * Modules for Interfacing with the File System::
496 * Modules for Other Aspects of the Lisp Interpreter and Object System::
497 * Modules for Interfacing with the Operating System::
498 * Modules for Interfacing with X Windows::
499 * Modules for Internationalization::
502 File: internals.info, Node: Low-Level Modules, Next: Basic Lisp Modules, Prev: A Summary of the Various XEmacs Modules, Up: A Summary of the Various XEmacs Modules
509 This is automatically generated from `config.h.in' based on the
510 results of configure tests and user-selected optional features and
511 contains preprocessor definitions specifying the nature of the
512 environment in which XEmacs is being compiled.
516 This is automatically generated from `paths.h.in' based on supplied
517 configure values, and allows for non-standard installed configurations
518 of the XEmacs directories. It's currently broken, though.
523 `emacs.c' contains `main()' and other code that performs the most
524 basic environment initializations and handles shutting down the XEmacs
525 process (this includes `kill-emacs', the normal way that XEmacs is
526 exited; `dump-emacs', which is used during the build process to write
527 out the XEmacs executable; `run-emacs-from-temacs', which can be used
528 to start XEmacs directly when temacs has finished loading all the Lisp
529 code; and emergency code to handle crashes [XEmacs tries to auto-save
530 all files before it crashes]).
532 Low-level code that directly interacts with the Unix signal
533 mechanism, however, is in `signal.c'. Note that this code does not
534 handle system dependencies in interfacing to signals; that is handled
535 using the `syssignal.h' header file, described in section J below.
555 These modules contain code dumping out the XEmacs executable on
556 various different systems. (This process is highly machine-specific and
557 requires intimate knowledge of the executable format and the memory map
558 of the process.) Only one of these modules is actually used; this is
559 chosen by `configure'.
565 These modules are used in conjunction with the dump mechanism. On
566 some systems, an alternative version of the C startup code (the actual
567 code that receives control from the operating system when the process is
568 started, and which calls `main()') is required so that the dumping
569 process works properly; `crt0.c' provides this.
571 `pre-crt0.c' and `lastfile.c' should be the very first and very last
572 file linked, respectively. (Actually, this is not really true.
573 `lastfile.c' should be after all Emacs modules whose initialized data
574 should be made constant, and before all other Emacs files and all
575 libraries. In particular, the allocation modules `gmalloc.c',
576 `alloca.c', etc. are normally placed past `lastfile.c', and all of the
577 files that implement Xt widget classes _must_ be placed after
578 `lastfile.c' because they contain various structures that must be
579 statically initialized and into which Xt writes at various times.)
580 `pre-crt0.c' and `lastfile.c' contain exported symbols that are used to
581 determine the start and end of XEmacs' initialized data space when
593 These handle basic C allocation of memory. `alloca.c' is an
594 emulation of the stack allocation function `alloca()' on machines that
595 lack this. (XEmacs makes extensive use of `alloca()' in its code.)
597 `gmalloc.c' and `malloc.c' are two implementations of the standard C
598 functions `malloc()', `realloc()' and `free()'. They are often used in
599 place of the standard system-provided `malloc()' because they usually
600 provide a much faster implementation, at the expense of additional
601 memory use. `gmalloc.c' is a newer implementation that is much more
602 memory-efficient for large allocations than `malloc.c', and should
603 always be preferred if it works. (At one point, `gmalloc.c' didn't work
604 on some systems where `malloc.c' worked; but this should be fixed now.)
606 `ralloc.c' is the "relocating allocator". It provides functions
607 similar to `malloc()', `realloc()' and `free()' that allocate memory
608 that can be dynamically relocated in memory. The advantage of this is
609 that allocated memory can be shuffled around to place all the free
610 memory at the end of the heap, and the heap can then be shrunk,
611 releasing the memory back to the operating system. The use of this can
612 be controlled with the configure option `--rel-alloc'; if enabled,
613 memory allocated for buffers will be relocatable, so that if a very
614 large file is visited and the buffer is later killed, the memory can be
615 released to the operating system. (The disadvantage of this mechanism
616 is that it can be very slow. On systems with the `mmap()' system call,
617 the XEmacs version of `ralloc.c' uses this to move memory around
618 without actually having to block-copy it, which can speed things up;
619 but it can still cause noticeable performance degradation.)
621 `free-hook.c' contains some debugging functions for checking for
622 invalid arguments to `free()'.
624 `vm-limit.c' contains some functions that warn the user when memory
625 is getting low. These are callback functions that are called by
626 `gmalloc.c' and `malloc.c' at appropriate times.
628 `getpagesize.h' provides a uniform interface for retrieving the size
629 of a page in virtual memory. `mem-limits.h' provides a uniform
630 interface for retrieving the total amount of available virtual memory.
631 Both are similar in spirit to the `sys*.h' files described in section
638 These implement a couple of basic C data types to facilitate memory
639 allocation. The `Blocktype' type efficiently manages the allocation of
640 fixed-size blocks by minimizing the number of times that `malloc()' and
641 `free()' are called. It allocates memory in large chunks, subdivides
642 the chunks into blocks of the proper size, and returns the blocks as
643 requested. When blocks are freed, they are placed onto a linked list,
644 so they can be efficiently reused. This data type is not much used in
645 XEmacs currently, because it's a fairly new addition.
647 The `Dynarr' type implements a "dynamic array", which is similar to
648 a standard C array but has no fixed limit on the number of elements it
649 can contain. Dynamic arrays can hold elements of any type, and when
650 you add a new element, the array automatically resizes itself if it
651 isn't big enough. Dynarrs are extensively used in the redisplay
656 This module is used in connection with inline functions (available in
657 some compilers). Often, inline functions need to have a corresponding
658 non-inline function that does the same thing. This module is where they
659 reside. It contains no actual code, but defines some special flags that
660 cause inline functions defined in header files to be rendered as actual
661 functions. It then includes all header files that contain any inline
662 function definitions, so that each one gets a real function equivalent.
667 These functions provide a system for doing internal consistency
668 checks during code development. This system is not currently used;
669 instead the simpler `assert()' macro is used along with the various
670 checks provided by the `--error-check-*' configuration options.
674 This is actually the source for a small, self-contained program used
679 This is not currently used.
682 File: internals.info, Node: Basic Lisp Modules, Next: Modules for Standard Editing Operations, Prev: Low-Level Modules, Up: A Summary of the Various XEmacs Modules
694 These are the basic header files for all XEmacs modules. Each module
695 includes `lisp.h', which brings the other header files in. `lisp.h'
696 contains the definitions of the structures and extractor and
697 constructor macros for the basic Lisp objects and various other basic
698 definitions for the Lisp environment, as well as some general-purpose
699 definitions (e.g. `min()' and `max()'). `lisp.h' includes either
700 `lisp-disunion.h' or `lisp-union.h', depending on whether
701 `USE_UNION_TYPE' is defined. These files define the typedef of the
702 Lisp object itself (as described above) and the low-level macros that
703 hide the actual implementation of the Lisp object. All extractor and
704 constructor macros for particular types of Lisp objects are defined in
705 terms of these low-level macros.
707 As a general rule, all typedefs should go into the typedefs section
708 of `lisp.h' rather than into a module-specific header file even if the
709 structure is defined elsewhere. This allows function prototypes that
710 use the typedef to be placed into other header files. Forward structure
711 declarations (i.e. a simple declaration like `struct foo;' where the
712 structure itself is defined elsewhere) should be placed into the
713 typedefs section as necessary.
715 `lrecord.h' contains the basic structures and macros that implement
716 all record-type Lisp objects--i.e. all objects whose type is a field in
717 their C structure, which includes all objects except the few most basic
720 `lisp.h' contains prototypes for most of the exported functions in
721 the various modules. Lisp primitives defined using `DEFUN' that need
722 to be called by C code should be declared using `EXFUN'. Other
723 function prototypes should be placed either into the appropriate
724 section of `lisp.h', or into a module-specific header file, depending
725 on how general-purpose the function is and whether it has
726 special-purpose argument types requiring definitions not in `lisp.h'.)
727 All initialization functions are prototyped in `symsinit.h'.
731 The large module `alloc.c' implements all of the basic allocation and
732 garbage collection for Lisp objects. The most commonly used Lisp
733 objects are allocated in chunks, similar to the Blocktype data type
734 described above; others are allocated in individually `malloc()'ed
735 blocks. This module provides the foundation on which all other aspects
736 of the Lisp environment sit, and is the first module initialized at
739 Note that `alloc.c' provides a series of generic functions that are
740 not dependent on any particular object type, and interfaces to
741 particular types of objects using a standardized interface of
742 type-specific methods. This scheme is a fundamental principle of
743 object-oriented programming and is heavily used throughout XEmacs. The
744 great advantage of this is that it allows for a clean separation of
745 functionality into different modules--new classes of Lisp objects, new
746 event interfaces, new device types, new stream interfaces, etc. can be
747 added transparently without affecting code anywhere else in XEmacs.
748 Because the different subsystems are divided into general and specific
749 code, adding a new subtype within a subsystem will in general not
750 require changes to the generic subsystem code or affect any of the other
751 subtypes in the subsystem; this provides a great deal of robustness to
757 This module contains all of the functions to handle the flow of
758 control. This includes the mechanisms of defining functions, calling
759 functions, traversing stack frames, and binding variables; the control
760 primitives and other special forms such as `while', `if', `eval',
761 `let', `and', `or', `progn', etc.; handling of non-local exits,
762 unwind-protects, and exception handlers; entering the debugger; methods
763 for the subr Lisp object type; etc. It does _not_ include the `read'
764 function, the `print' function, or the handling of symbols and obarrays.
766 `backtrace.h' contains some structures related to stack frames and
771 This module implements the Lisp reader and the `read' function,
772 which converts text into Lisp objects, according to the read syntax of
773 the objects, as described above. This is similar to the parser that is
774 a part of all compilers.
778 This module implements the Lisp print mechanism and the `print'
779 function and related functions. This is the inverse of the Lisp reader
780 - it converts Lisp objects to a printed, textual representation.
781 (Hopefully something that can be read back in using `read' to get an
788 `symbols.c' implements the handling of symbols, obarrays, and
789 retrieving the values of symbols. Much of the code is devoted to
790 handling the special "symbol-value-magic" objects that define special
791 types of variables--this includes buffer-local variables, variable
792 aliases, variables that forward into C variables, etc. This module is
793 initialized extremely early (right after `alloc.c'), because it is here
794 that the basic symbols `t' and `nil' are created, and those symbols are
795 used everywhere throughout XEmacs.
797 `symeval.h' contains the definitions of symbol structures and the
798 `DEFVAR_LISP()' and related macros for declaring variables.
804 These modules implement the methods and standard Lisp primitives for
805 all the basic Lisp object types other than symbols (which are described
806 above). `data.c' contains all the predicates (primitives that return
807 whether an object is of a particular type); the integer arithmetic
808 functions; and the basic accessor and mutator primitives for the various
809 object types. `fns.c' contains all the standard predicates for working
810 with sequences (where, abstractly speaking, a sequence is an ordered set
811 of objects, and can be represented by a list, string, vector, or
812 bit-vector); it also contains `equal', perhaps on the grounds that bulk
813 of the operation of `equal' is comparing sequences. `floatfns.c'
814 contains methods and primitives for floats and floating-point
820 `bytecode.c' implements the byte-code interpreter and
821 compiled-function objects, and `bytecode.h' contains associated
822 structures. Note that the byte-code _compiler_ is written in Lisp.
825 File: internals.info, Node: Modules for Standard Editing Operations, Next: Editor-Level Control Flow Modules, Prev: Basic Lisp Modules, Up: A Summary of the Various XEmacs Modules
827 Modules for Standard Editing Operations
828 =======================================
834 `buffer.c' implements the "buffer" Lisp object type. This includes
835 functions that create and destroy buffers; retrieve buffers by name or
836 by other properties; manipulate lists of buffers (remember that buffers
837 are permanent objects and stored in various ordered lists); retrieve or
838 change buffer properties; etc. It also contains the definitions of all
839 the built-in buffer-local variables (which can be viewed as buffer
840 properties). It does _not_ contain code to manipulate buffer-local
841 variables (that's in `symbols.c', described above); or code to
842 manipulate the text in a buffer.
844 `buffer.h' defines the structures associated with a buffer and the
845 various macros for retrieving text from a buffer and special buffer
846 positions (e.g. `point', the default location for text insertion). It
847 also contains macros for working with buffer positions and converting
848 between their representations as character offsets and as byte offsets
849 (under MULE, they are different, because characters can be multi-byte).
850 It is one of the largest header files.
852 `bufslots.h' defines the fields in the buffer structure that
853 correspond to the built-in buffer-local variables. It is its own
854 header file because it is included many times in `buffer.c', as a way
855 of iterating over all the built-in buffer-local variables.
860 `insdel.c' contains low-level functions for inserting and deleting
861 text in a buffer, keeping track of changed regions for use by
862 redisplay, and calling any before-change and after-change functions
863 that may have been registered for the buffer. It also contains the
864 actual functions that convert between byte offsets and character
867 `insdel.h' contains associated headers.
871 This module implements the "marker" Lisp object type, which
872 conceptually is a pointer to a text position in a buffer that moves
873 around as text is inserted and deleted, so as to remain in the same
874 relative position. This module doesn't actually move the markers around
875 - that's handled in `insdel.c'. This module just creates them and
876 implements the primitives for working with them. As markers are simple
877 objects, this does not entail much.
879 Note that the standard arithmetic primitives (e.g. `+') accept
880 markers in place of integers and automatically substitute the value of
881 `marker-position' for the marker, i.e. an integer describing the
882 current buffer position of the marker.
887 This module implements the "extent" Lisp object type, which is like
888 a marker that works over a range of text rather than a single position.
889 Extents are also much more complex and powerful than markers and have a
890 more efficient (and more algorithmically complex) implementation. The
891 implementation is described in detail in comments in `extents.c'.
893 The code in `extents.c' works closely with `insdel.c' so that
894 extents are properly moved around as text is inserted and deleted.
895 There is also code in `extents.c' that provides information needed by
896 the redisplay mechanism for efficient operation. (Remember that extents
897 can have display properties that affect [sometimes drastically, as in
898 the `invisible' property] the display of the text they cover.)
902 `editfns.c' contains the standard Lisp primitives for working with a
903 buffer's text, and calls the low-level functions in `insdel.c'. It
904 also contains primitives for working with `point' (the default buffer
907 `editfns.c' also contains functions for retrieving various
908 characteristics from the external environment: the current time, the
909 process ID of the running XEmacs process, the name of the user who ran
910 this XEmacs process, etc. It's not clear why this code is in
917 These modules implement the basic "interactive" commands, i.e.
918 user-callable functions. Commands, as opposed to other functions, have
919 special ways of getting their parameters interactively (by querying the
920 user), as opposed to having them passed in a normal function
921 invocation. Many commands are not really meant to be called from other
922 Lisp functions, because they modify global state in a way that's often
923 undesired as part of other Lisp functions.
925 `callint.c' implements the mechanism for querying the user for
926 parameters and calling interactive commands. The bulk of this module is
927 code that parses the interactive spec that is supplied with an
930 `cmds.c' implements the basic, most commonly used editing commands:
931 commands to move around the current buffer and insert and delete
932 characters. These commands are implemented using the Lisp primitives
933 defined in `editfns.c'.
935 `commands.h' contains associated structure definitions and
942 `search.c' implements the Lisp primitives for searching for text in
943 a buffer, and some of the low-level algorithms for doing this. In
944 particular, the fast fixed-string Boyer-Moore search algorithm is
945 implemented in `search.c'. The low-level algorithms for doing
946 regular-expression searching, however, are implemented in `regex.c' and
947 `regex.h'. These two modules are largely independent of XEmacs, and
948 are similar to (and based upon) the regular-expression routines used in
949 `grep' and other GNU utilities.
953 `doprnt.c' implements formatted-string processing, similar to
954 `printf()' command in C.
958 This module implements the undo mechanism for tracking buffer
959 changes. Most of this could be implemented in Lisp.
962 File: internals.info, Node: Editor-Level Control Flow Modules, Next: Modules for the Basic Displayable Lisp Objects, Prev: Modules for Standard Editing Operations, Up: A Summary of the Various XEmacs Modules
964 Editor-Level Control Flow Modules
965 =================================
973 These implement the handling of events (user input and other system
976 `events.c' and `events.h' define the "event" Lisp object type and
977 primitives for manipulating it.
979 `event-stream.c' implements the basic functions for working with
980 event queues, dispatching an event by looking it up in relevant keymaps
981 and such, and handling timeouts; this includes the primitives
982 `next-event' and `dispatch-event', as well as related primitives such
983 as `sit-for', `sleep-for', and `accept-process-output'.
984 (`event-stream.c' is one of the hairiest and trickiest modules in
985 XEmacs. Beware! You can easily mess things up here.)
987 `event-Xt.c' and `event-tty.c' implement the low-level interfaces
988 onto retrieving events from Xt (the X toolkit) and from TTY's (using
989 `read()' and `select()'), respectively. The event interface enforces a
990 clean separation between the specific code for interfacing with the
991 operating system and the generic code for working with events, by
992 defining an API of basic, low-level event methods; `event-Xt.c' and
993 `event-tty.c' are two different implementations of this API. To add
994 support for a new operating system (e.g. NeXTstep), one merely needs to
995 provide another implementation of those API functions.
997 Note that the choice of whether to use `event-Xt.c' or `event-tty.c'
998 is made at compile time! Or at the very latest, it is made at startup
999 time. `event-Xt.c' handles events for _both_ X and TTY frames;
1000 `event-tty.c' is only used when X support is not compiled into XEmacs.
1001 The reason for this is that there is only one event loop in XEmacs:
1002 thus, it needs to be able to receive events from all different kinds of
1008 `keymap.c' and `keymap.h' define the "keymap" Lisp object type and
1009 associated methods and primitives. (Remember that keymaps are objects
1010 that associate event descriptions with functions to be called to
1011 "execute" those events; `dispatch-event' looks up events in the
1016 `keyboard.c' contains functions that implement the actual editor
1017 command loop--i.e. the event loop that cyclically retrieves and
1018 dispatches events. This code is also rather tricky, just like
1024 These two modules contain the basic code for defining keyboard
1025 macros. These functions don't actually do much; most of the code that
1026 handles keyboard macros is mixed in with the event-handling code in
1031 This contains some miscellaneous code related to the minibuffer
1032 (most of the minibuffer code was moved into Lisp by Richard Mlynarik).
1033 This includes the primitives for completion (although filename
1034 completion is in `dired.c'), the lowest-level interface to the
1035 minibuffer (if the command loop were cleaned up, this too could be in
1036 Lisp), and code for dealing with the echo area (this, too, was mostly
1037 moved into Lisp, and the only code remaining is code to call out to
1038 Lisp or provide simple bootstrapping implementations early in temacs,
1039 before the echo-area Lisp code is loaded).
1042 File: internals.info, Node: Modules for the Basic Displayable Lisp Objects, Next: Modules for other Display-Related Lisp Objects, Prev: Editor-Level Control Flow Modules, Up: A Summary of the Various XEmacs Modules
1044 Modules for the Basic Displayable Lisp Objects
1045 ==============================================
1057 These modules implement the "device" Lisp object type. This
1058 abstracts a particular screen or connection on which frames are
1059 displayed. As with Lisp objects, event interfaces, and other
1060 subsystems, the device code is separated into a generic component that
1061 contains a standardized interface (in the form of a set of methods) onto
1062 particular device types.
1064 The device subsystem defines all the methods and provides method
1065 services for not only device operations but also for the frame, window,
1066 menubar, scrollbar, toolbar, and other displayable-object subsystems.
1067 The reason for this is that all of these subsystems have the same
1068 subtypes (X, TTY, NeXTstep, Microsoft Windows, etc.) as devices do.
1077 Each device contains one or more frames in which objects (e.g. text)
1078 are displayed. A frame corresponds to a window in the window system;
1079 usually this is a top-level window but it could potentially be one of a
1080 number of overlapping child windows within a top-level window, using the
1081 MDI (Multiple Document Interface) protocol in Microsoft Windows or a
1084 The `frame-*' files implement the "frame" Lisp object type and
1085 provide the generic and device-type-specific operations on frames (e.g.
1086 raising, lowering, resizing, moving, etc.).
1091 Each frame consists of one or more non-overlapping "windows" (better
1092 known as "panes" in standard window-system terminology) in which a
1093 buffer's text can be displayed. Windows can also have scrollbars
1094 displayed around their edges.
1096 `window.c' and `window.h' implement the "window" Lisp object type
1097 and provide code to manage windows. Since windows have no associated
1098 resources in the window system (the window system knows only about the
1099 frame; no child windows or anything are used for XEmacs windows), there
1100 is no device-type-specific code here; all of that code is part of the
1101 redisplay mechanism or the code for particular object types such as
1105 File: internals.info, Node: Modules for other Display-Related Lisp Objects, Next: Modules for the Redisplay Mechanism, Prev: Modules for the Basic Displayable Lisp Objects, Up: A Summary of the Various XEmacs Modules
1107 Modules for other Display-Related Lisp Objects
1108 ==============================================
1142 This file provides C support for syntax highlighting--i.e.
1143 highlighting different syntactic constructs of a source file in
1144 different colors, for easy reading. The C support is provided so that
1152 These modules decode GIF-format image files, for use with glyphs.
1155 File: internals.info, Node: Modules for the Redisplay Mechanism, Next: Modules for Interfacing with the File System, Prev: Modules for other Display-Related Lisp Objects, Up: A Summary of the Various XEmacs Modules
1157 Modules for the Redisplay Mechanism
1158 ===================================
1166 These files provide the redisplay mechanism. As with many other
1167 subsystems in XEmacs, there is a clean separation between the general
1168 and device-specific support.
1170 `redisplay.c' contains the bulk of the redisplay engine. These
1171 functions update the redisplay structures (which describe how the screen
1172 is to appear) to reflect any changes made to the state of any
1173 displayable objects (buffer, frame, window, etc.) since the last time
1174 that redisplay was called. These functions are highly optimized to
1175 avoid doing more work than necessary (since redisplay is called
1176 extremely often and is potentially a huge time sink), and depend heavily
1177 on notifications from the objects themselves that changes have occurred,
1178 so that redisplay doesn't explicitly have to check each possible object.
1179 The redisplay mechanism also contains a great deal of caching to further
1180 speed things up; some of this caching is contained within the various
1181 displayable objects.
1183 `redisplay-output.c' goes through the redisplay structures and
1184 converts them into calls to device-specific methods to actually output
1187 `redisplay-x.c' and `redisplay-tty.c' are two implementations of
1188 these redisplay output methods, for X frames and TTY frames,
1193 This module contains various functions and Lisp primitives for
1194 converting between buffer positions and screen positions. These
1195 functions call the redisplay mechanism to do most of the work, and then
1196 examine the redisplay structures to get the necessary information. This
1203 These files contain functions for working with the termcap
1204 (BSD-style) and terminfo (System V style) databases of terminal
1205 capabilities and escape sequences, used when XEmacs is displaying in a
1211 These files provide some miscellaneous TTY-output functions and
1212 should probably be merged into `redisplay-tty.c'.