the data stream) or locking (effective until the next designation or
locking) control sequences. An encoding conformant to ISO 2022 is
typically defined by designating the initial contents of the G0-G3
-registers, specifying an 7 or 8 bit environment, and specifying whether
+registers, specifying a 7 or 8 bit environment, and specifying whether
further designations will be recognized.
Some examples of character sets and the registered final characters
* CCL Statements:: Semantics of CCL statements.
* CCL Expressions:: Operators and expressions in CCL.
* Calling CCL:: Running CCL programs.
-* CCL Examples:: The encoding functions for Big5 and KOI-8.
+* CCL Example:: A trivial program to transform the Web's URL encoding.
@end menu
@node CCL Syntax, CCL Statements, , CCL
Shift JIS. CCL_DECODE_SJIS is its inverse.) It is somewhat odd to
represent the SJIS operations in infix form.
-@node Calling CCL, CCL Examples, CCL Expressions, CCL
+@node Calling CCL, CCL Example, CCL Expressions, CCL
@comment Node, Next, Previous, Up
@subsection Calling CCL
Resets the CCL interpreter's internal elapsed time registers.
@end defun
-@node CCL Examples, , Calling CCL, CCL
+@node CCL Example, , Calling CCL, CCL
@comment Node, Next, Previous, Up
-@subsection CCL Examples
+@subsection CCL Example
+
+ In this section, we describe the implementation of a trivial coding
+system to transform from the Web's URL encoding to XEmacs' internal
+coding. Many people will have been first exposed to URL encoding when
+they saw ``%20'' where they expected a space in a file's name on their
+local hard disk; this can happen when a browser saves a file from the
+web and doesn't encode the name, as passed from the server, properly.
+
+ URL encoding itself is underspecified with regard to encodings beyond
+ASCII. The relevant document, RFC 1738, explicitly doesn't give any
+information on how to encode non-ASCII characters, and the ``obvious''
+way---use the %xx values for the octets of the eight bit MIME character
+set in which the page was served---breaks when a user types a character
+outside that character set. Best practice for web development is to
+serve all pages as UTF-8 and treat incoming form data as using that
+coding system. (Oh, and gamble that your clients won't ever want to
+type anything outside Unicode. But that's not so much of a gamble with
+today's client operating systems.) We don't treat non-ASCII in this
+example, as dealing with @samp{(read-multibyte-character ...)} and
+errors therewith would make it much harder to understand.
+
+ Since CCL isn't a very rich language, we move much of the logic that
+would ordinarily be computed from operations like @code{(member ..)},
+@code{(and ...)} and @code{(or ...)} into tables, from which register
+values are read and written, and on which @code{if} statements are
+predicated. Much more of the implementation of this coding system is
+occupied with constructing these tables---in normal Emacs Lisp---than it
+is with actual CCL code.
+
+ All the @code{defvar} statements we deal with in the next few sections
+are surrounded by a @code{(eval-and-compile ...)}, which means that the
+logic which initializes these variables executes at compile time, and if
+XEmacs loads the compiled version of the file, these variables are
+initialized as constants.
- This section is not yet written.
+@menu
+* Four bits to ASCII:: Two tables used for getting hex digits from ASCII.
+* URI Encoding constants:: Useful predefined characters.
+* Numeric to ASCII-hexadecimal conversion:: Trivial in Lisp, not so in CCL.
+* Characters to be preserved:: No transformation needed for these characters.
+* The program to decode to internal format:: .
+* The program to encode from internal format:: .
+* The actual coding system:: .
+@end menu
+
+@node Four bits to ASCII, URI Encoding constants, , CCL Example
+@subsubsection Four bits to ASCII
+
+ The first @code{defvar} is for
+@code{url-coding-high-order-nybble-as-ascii}, a 256-entry table that
+maps from an octet's value to the ASCII encoding for the hex value of
+its most significant four bits. That might sound complex, but it isn't;
+for decimal 65, hex value @samp{#x41}, the entry in the table is the
+ASCII encoding of `4'. For decimal 122, ASCII `z', hex value
+@code{#x7a}, @code{(elt url-coding-high-order-nybble-as-ascii #x7a)}
+after this file is loaded gives the ASCII encoding of 7.
+
+@example
+(defvar url-coding-high-order-nybble-as-ascii
+ (let ((val (make-vector 256 0))
+ (i 0))
+ (while (< i (length val))
+ (aset val i (char-to-int (aref (format "%02X" i) 0)))
+ (setq i (1+ i)))
+ val)
+ "Table to find an ASCII version of an octet's most significant 4 bits.")
+@end example
+
+ The next table, @code{url-coding-low-order-nybble-as-ascii} is almost
+the same thing, but this time it has a map for the hex encoding of the
+low-order four bits. So the sixty-fifth entry (offset @samp{#x41}) is
+the ASCII encoding of `1', the hundred-and-twenty-second (offset
+@samp{#x7a}) is the ASCII encoding of `A'.
+
+@example
+(defvar url-coding-low-order-nybble-as-ascii
+ (let ((val (make-vector 256 0))
+ (i 0))
+ (while (< i (length val))
+ (aset val i (char-to-int (aref (format "%02X" i) 1)))
+ (setq i (1+ i)))
+ val)
+ "Table to find an ASCII version of an octet's least significant 4 bits.")
+@end example
+
+@node URI Encoding constants, Numeric to ASCII-hexadecimal conversion, Four bits to ASCII, CCL Example
+@subsubsection URI Encoding constants
+
+ Next, we have a couple of variables that make the CCL code more
+readable. The first is the ASCII encoding of the percentage sign; this
+character is used as an escape code, to start the encoding of a
+non-printable character. For historical reasons, URL encoding allows
+the space character to be encoded as a plus sign--it does make typing
+URLs like @samp{http://google.com/search?q=XEmacs+home+page} easier--and
+as such, we have to check when decoding for this value, and map it to
+the space character. When doing this in CCL, we use the
+@code{url-coding-escaped-space-code} variable.
+
+@example
+(defvar url-coding-escape-character-code (char-to-int ?%)
+ "The code point for the percentage sign, in ASCII.")
+
+(defvar url-coding-escaped-space-code (char-to-int ?+)
+ "The URL-encoded value of the space character, that is, +.")
+@end example
+
+@node Numeric to ASCII-hexadecimal conversion, Characters to be preserved, URI Encoding constants, CCL Example
+@subsubsection Numeric to ASCII-hexadecimal conversion
+
+ Now, we have a couple of utility tables that wouldn't be necessary in
+a more expressive programming language than is CCL. The first is sixteen
+in length, and maps a hexadecimal number to the ASCII encoding of that
+number; so zero maps to ASCII `0', ten maps to ASCII `A.' The second
+does the reverse; that is, it maps an ASCII character to its value when
+interpreted as a hexadecimal digit. ('A' => 10, 'c' => 12, '2' => 2, as
+a few examples.)
+
+@example
+(defvar url-coding-hex-digit-table
+ (let ((i 0)
+ (val (make-vector 16 0)))
+ (while (< i 16)
+ (aset val i (char-to-int (aref (format "%X" i) 0)))
+ (setq i (1+ i)))
+ val)
+ "A map from a hexadecimal digit's numeric value to its encoding in ASCII.")
+
+(defvar url-coding-latin-1-as-hex-table
+ (let ((val (make-vector 256 0))
+ (i 0))
+ (while (< i (length val))
+ ;; Get a hex val for this ASCII character.
+ (aset val i (string-to-int (format "%c" i) 16))
+ (setq i (1+ i)))
+ val)
+ "A map from Latin 1 code points to their values as hexadecimal digits.")
+@end example
+
+@node Characters to be preserved, The program to decode to internal format, Numeric to ASCII-hexadecimal conversion, CCL Example
+@subsubsection Characters to be preserved
+
+ And finally, the last of these tables. URL encoding says that
+alphanumeric characters, the underscore, hyphen and the full stop
+@footnote{That's what the standards call it, though my North American
+readers will be more familiar with it as the period character.} retain
+their ASCII encoding, and don't undergo transformation.
+@code{url-coding-should-preserve-table} is an array in which the entries
+are one if the corresponding ASCII character should be left as-is, and
+zero if they should be transformed. So the entries for all the control
+and most of the punctuation charcters are zero. Lisp programmers will
+observe that this initialization is particularly inefficient, but
+they'll also be aware that this is a long way from an inner loop where
+every nanosecond counts.
+
+@example
+(defvar url-coding-should-preserve-table
+ (let ((preserve
+ (list ?- ?_ ?. ?a ?b ?c ?d ?e ?f ?g ?h ?i ?j ?k ?l ?m ?n ?o
+ ?p ?q ?r ?s ?t ?u ?v ?w ?x ?y ?z ?A ?B ?C ?D ?E ?F ?G
+ ?H ?I ?J ?K ?L ?M ?N ?O ?P ?Q ?R ?S ?T ?U ?V ?W ?X ?Y
+ ?Z ?0 ?1 ?2 ?3 ?4 ?5 ?6 ?7 ?8 ?9))
+ (i 0)
+ (res (make-vector 256 0)))
+ (while (< i 256)
+ (when (member (int-char i) preserve)
+ (aset res i 1))
+ (setq i (1+ i)))
+ res)
+ "A 256-entry array of flags, indicating whether or not to preserve an
+octet as its ASCII encoding.")
+@end example
+
+@node The program to decode to internal format, The program to encode from internal format, Characters to be preserved, CCL Example
+@subsubsection The program to decode to internal format
+
+ After the almost interminable tables, we get to the CCL. The first
+CCL program, @code{ccl-decode-urlcoding} decodes from the URL coding to
+our internal format; since this version of CCL doesn't have support for
+error checking on the input, we don't do any verification on it.
+
+The buffer magnification--approximate ratio of the size of the output
+buffer to the size of the input buffer--is declared as one, because
+fractional values aren't allowed. (Since all those %20's will map to
+` ', the length of the output text will be less than that of the input
+text.)
+
+So, first we read an octet from the input buffer into register
+@samp{r0}, to set up the loop. Next, we start the loop, with a
+@code{(loop ...)} statement, and we check if the value in @samp{r0} is a
+percentage sign. (Note the comma before
+@code{url-coding-escape-character-code}; since CCL is a Lisp macro
+language, we can break out of the macro evaluation with a comman, and as
+such, ``@code{,url-coding-escape-character-code}'' will be evaluated as a
+literal `37.')
+
+If it is a percentage sign, we read the next two octets into @samp{r2}
+and @samp{r3}, and convert them into their hexadecimal numeric values,
+using the @code{url-coding-latin-1-as-hex-table} array declared above.
+(But again, it'll be interpreted as a literal array.) We then left
+shift the first by four bits, mask the two together, and write the
+result to the output buffer.
+
+If it isn't a percentage sign, and it is a `+' sign, we write a
+space--hexadecimal 20--to the output buffer.
+
+If none of those things are true, we pass the octet to the output buffer
+untransformed. (This could be a place to put error checking, in a more
+expressive language.) We then read one more octet from the input
+buffer, and move to the next iteration of the loop.
+
+@example
+(define-ccl-program ccl-decode-urlcoding
+ `(1
+ ((read r0)
+ (loop
+ (if (r0 == ,url-coding-escape-character-code)
+ ((read r2 r3)
+ ;; Assign the value at offset r2 in the url-coding-hex-digit-table
+ ;; to r3.
+ (r2 = r2 ,url-coding-latin-1-as-hex-table)
+ (r3 = r3 ,url-coding-latin-1-as-hex-table)
+ (r2 <<= 4)
+ (r3 |= r2)
+ (write r3))
+ (if (r0 == ,url-coding-escaped-space-code)
+ (write #x20)
+ (write r0)))
+ (read r0)
+ (repeat))))
+ "CCL program to take URI-encoded ASCII text and transform it to our
+internal encoding. ")
+@end example
+
+@node The program to encode from internal format, The actual coding system, The program to decode to internal format, CCL Example
+@subsubsection The program to encode from internal format
+
+ Next, we see the CCL program to encode ASCII text as URL coded text.
+Here, the buffer magnification is specified as three, to account for ` '
+mapping to %20, etc. As before, we read an octet from the input into
+@samp{r0}, and move into the body of the loop. Next, we check if we
+should preserve the value of this octet, by reading from offset
+@samp{r0} in the @code{url-coding-should-preserve-table} into @samp{r1}.
+Then we have an @samp{if} statement predicated on the value in
+@samp{r1}; for the true branch, we write the input octet directly. For
+the false branch, we write a percentage sign, the ASCII encoding of the
+high four bits in hex, and then the ASCII encoding of the low four bits
+in hex.
+
+We then read an octet from the input into @samp{r0}, and repeat the loop.
+
+@example
+(define-ccl-program ccl-encode-urlcoding
+ `(3
+ ((read r0)
+ (loop
+ (r1 = r0 ,url-coding-should-preserve-table)
+ ;; If we should preserve the value, just write the octet directly.
+ (if r1
+ (write r0)
+ ;; else, write a percentage sign, and the hex value of the octet, in
+ ;; an ASCII-friendly format.
+ ((write ,url-coding-escape-character-code)
+ (write r0 ,url-coding-high-order-nybble-as-ascii)
+ (write r0 ,url-coding-low-order-nybble-as-ascii)))
+ (read r0)
+ (repeat))))
+ "CCL program to encode octets (almost) according to RFC 1738")
+@end example
+
+@node The actual coding system, , The program to encode from internal format, CCL Example
+@subsubsection The actual coding system
+
+To actually create the coding system, we call
+@samp{make-coding-system}. The first argument is the symbol that is to
+be the name of the coding system, in our case @samp{url-coding}. The
+second specifies that the coding system is to be of type
+@samp{ccl}---there are several other coding system types available,
+including, see the documentation for @samp{make-coding-system} for the
+full list. Then there's a documentation string describing the wherefore
+and caveats of the coding system, and the final argument is a property
+list giving information about the CCL programs and the coding system's
+mnemonic.
+
+@example
+(make-coding-system
+ 'url-coding 'ccl
+ "The coding used by application/x-www-form-urlencoded HTTP applications.
+This coding form doesn't specify anything about non-ASCII characters, so
+make sure you've transformed to a seven-bit coding system first."
+ '(decode ccl-decode-urlcoding
+ encode ccl-encode-urlcoding
+ mnemonic "URLenc"))
+@end example
+
+If you're lucky, the @samp{url-coding} coding system describe here
+should be available in the XEmacs package system. Otherwise, downloading
+it from @samp{http://www.parhasard.net/url-coding.el} should work for
+the foreseeable future.
@node Category Tables, , CCL, MULE
@section Category Tables
projects / chise / xemacs-chise.git.1 / blobdiff