]> code.delx.au - gnu-emacs/blob - doc/lispref/nonascii.texi
Merge from origin/emacs-24
[gnu-emacs] / doc / lispref / nonascii.texi
1 @c -*-texinfo-*-
2 @c This is part of the GNU Emacs Lisp Reference Manual.
3 @c Copyright (C) 1998-1999, 2001-2014 Free Software Foundation, Inc.
4 @c See the file elisp.texi for copying conditions.
5 @node Non-ASCII Characters
6 @chapter Non-@acronym{ASCII} Characters
7 @cindex multibyte characters
8 @cindex characters, multi-byte
9 @cindex non-@acronym{ASCII} characters
10
11 This chapter covers the special issues relating to characters and
12 how they are stored in strings and buffers.
13
14 @menu
15 * Text Representations:: How Emacs represents text.
16 * Disabling Multibyte:: Controlling whether to use multibyte characters.
17 * Converting Representations:: Converting unibyte to multibyte and vice versa.
18 * Selecting a Representation:: Treating a byte sequence as unibyte or multi.
19 * Character Codes:: How unibyte and multibyte relate to
20 codes of individual characters.
21 * Character Properties:: Character attributes that define their
22 behavior and handling.
23 * Character Sets:: The space of possible character codes
24 is divided into various character sets.
25 * Scanning Charsets:: Which character sets are used in a buffer?
26 * Translation of Characters:: Translation tables are used for conversion.
27 * Coding Systems:: Coding systems are conversions for saving files.
28 * Input Methods:: Input methods allow users to enter various
29 non-ASCII characters without special keyboards.
30 * Locales:: Interacting with the POSIX locale.
31 @end menu
32
33 @node Text Representations
34 @section Text Representations
35 @cindex text representation
36
37 Emacs buffers and strings support a large repertoire of characters
38 from many different scripts, allowing users to type and display text
39 in almost any known written language.
40
41 @cindex character codepoint
42 @cindex codespace
43 @cindex Unicode
44 To support this multitude of characters and scripts, Emacs closely
45 follows the @dfn{Unicode Standard}. The Unicode Standard assigns a
46 unique number, called a @dfn{codepoint}, to each and every character.
47 The range of codepoints defined by Unicode, or the Unicode
48 @dfn{codespace}, is @code{0..#x10FFFF} (in hexadecimal notation),
49 inclusive. Emacs extends this range with codepoints in the range
50 @code{#x110000..#x3FFFFF}, which it uses for representing characters
51 that are not unified with Unicode and @dfn{raw 8-bit bytes} that
52 cannot be interpreted as characters. Thus, a character codepoint in
53 Emacs is a 22-bit integer.
54
55 @cindex internal representation of characters
56 @cindex characters, representation in buffers and strings
57 @cindex multibyte text
58 To conserve memory, Emacs does not hold fixed-length 22-bit numbers
59 that are codepoints of text characters within buffers and strings.
60 Rather, Emacs uses a variable-length internal representation of
61 characters, that stores each character as a sequence of 1 to 5 8-bit
62 bytes, depending on the magnitude of its codepoint@footnote{
63 This internal representation is based on one of the encodings defined
64 by the Unicode Standard, called @dfn{UTF-8}, for representing any
65 Unicode codepoint, but Emacs extends UTF-8 to represent the additional
66 codepoints it uses for raw 8-bit bytes and characters not unified with
67 Unicode.}. For example, any @acronym{ASCII} character takes up only 1
68 byte, a Latin-1 character takes up 2 bytes, etc. We call this
69 representation of text @dfn{multibyte}.
70
71 Outside Emacs, characters can be represented in many different
72 encodings, such as ISO-8859-1, GB-2312, Big-5, etc. Emacs converts
73 between these external encodings and its internal representation, as
74 appropriate, when it reads text into a buffer or a string, or when it
75 writes text to a disk file or passes it to some other process.
76
77 Occasionally, Emacs needs to hold and manipulate encoded text or
78 binary non-text data in its buffers or strings. For example, when
79 Emacs visits a file, it first reads the file's text verbatim into a
80 buffer, and only then converts it to the internal representation.
81 Before the conversion, the buffer holds encoded text.
82
83 @cindex unibyte text
84 Encoded text is not really text, as far as Emacs is concerned, but
85 rather a sequence of raw 8-bit bytes. We call buffers and strings
86 that hold encoded text @dfn{unibyte} buffers and strings, because
87 Emacs treats them as a sequence of individual bytes. Usually, Emacs
88 displays unibyte buffers and strings as octal codes such as
89 @code{\237}. We recommend that you never use unibyte buffers and
90 strings except for manipulating encoded text or binary non-text data.
91
92 In a buffer, the buffer-local value of the variable
93 @code{enable-multibyte-characters} specifies the representation used.
94 The representation for a string is determined and recorded in the string
95 when the string is constructed.
96
97 @defvar enable-multibyte-characters
98 This variable specifies the current buffer's text representation.
99 If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
100 it contains unibyte encoded text or binary non-text data.
101
102 You cannot set this variable directly; instead, use the function
103 @code{set-buffer-multibyte} to change a buffer's representation.
104 @end defvar
105
106 @defun position-bytes position
107 Buffer positions are measured in character units. This function
108 returns the byte-position corresponding to buffer position
109 @var{position} in the current buffer. This is 1 at the start of the
110 buffer, and counts upward in bytes. If @var{position} is out of
111 range, the value is @code{nil}.
112 @end defun
113
114 @defun byte-to-position byte-position
115 Return the buffer position, in character units, corresponding to given
116 @var{byte-position} in the current buffer. If @var{byte-position} is
117 out of range, the value is @code{nil}. In a multibyte buffer, an
118 arbitrary value of @var{byte-position} can be not at character
119 boundary, but inside a multibyte sequence representing a single
120 character; in this case, this function returns the buffer position of
121 the character whose multibyte sequence includes @var{byte-position}.
122 In other words, the value does not change for all byte positions that
123 belong to the same character.
124 @end defun
125
126 @defun multibyte-string-p string
127 Return @code{t} if @var{string} is a multibyte string, @code{nil}
128 otherwise. This function also returns @code{nil} if @var{string} is
129 some object other than a string.
130 @end defun
131
132 @defun string-bytes string
133 @cindex string, number of bytes
134 This function returns the number of bytes in @var{string}.
135 If @var{string} is a multibyte string, this can be greater than
136 @code{(length @var{string})}.
137 @end defun
138
139 @defun unibyte-string &rest bytes
140 This function concatenates all its argument @var{bytes} and makes the
141 result a unibyte string.
142 @end defun
143
144 @node Disabling Multibyte
145 @section Disabling Multibyte Characters
146 @cindex disabling multibyte
147
148 By default, Emacs starts in multibyte mode: it stores the contents
149 of buffers and strings using an internal encoding that represents
150 non-@acronym{ASCII} characters using multi-byte sequences. Multibyte
151 mode allows you to use all the supported languages and scripts without
152 limitations.
153
154 @cindex turn multibyte support on or off
155 Under very special circumstances, you may want to disable multibyte
156 character support, for a specific buffer.
157 When multibyte characters are disabled in a buffer, we call
158 that @dfn{unibyte mode}. In unibyte mode, each character in the
159 buffer has a character code ranging from 0 through 255 (0377 octal); 0
160 through 127 (0177 octal) represent @acronym{ASCII} characters, and 128
161 (0200 octal) through 255 (0377 octal) represent non-@acronym{ASCII}
162 characters.
163
164 To edit a particular file in unibyte representation, visit it using
165 @code{find-file-literally}. @xref{Visiting Functions}. You can
166 convert a multibyte buffer to unibyte by saving it to a file, killing
167 the buffer, and visiting the file again with
168 @code{find-file-literally}. Alternatively, you can use @kbd{C-x
169 @key{RET} c} (@code{universal-coding-system-argument}) and specify
170 @samp{raw-text} as the coding system with which to visit or save a
171 file. @xref{Text Coding, , Specifying a Coding System for File Text,
172 emacs, GNU Emacs Manual}. Unlike @code{find-file-literally}, finding
173 a file as @samp{raw-text} doesn't disable format conversion,
174 uncompression, or auto mode selection.
175
176 @c See http://debbugs.gnu.org/11226 for lack of unibyte tooltip.
177 @vindex enable-multibyte-characters
178 The buffer-local variable @code{enable-multibyte-characters} is
179 non-@code{nil} in multibyte buffers, and @code{nil} in unibyte ones.
180 The mode line also indicates whether a buffer is multibyte or not.
181 With a graphical display, in a multibyte buffer, the portion of the
182 mode line that indicates the character set has a tooltip that (amongst
183 other things) says that the buffer is multibyte. In a unibyte buffer,
184 the character set indicator is absent. Thus, in a unibyte buffer
185 (when using a graphical display) there is normally nothing before the
186 indication of the visited file's end-of-line convention (colon,
187 backslash, etc.), unless you are using an input method.
188
189 @findex toggle-enable-multibyte-characters
190 You can turn off multibyte support in a specific buffer by invoking the
191 command @code{toggle-enable-multibyte-characters} in that buffer.
192
193 @node Converting Representations
194 @section Converting Text Representations
195
196 Emacs can convert unibyte text to multibyte; it can also convert
197 multibyte text to unibyte, provided that the multibyte text contains
198 only @acronym{ASCII} and 8-bit raw bytes. In general, these
199 conversions happen when inserting text into a buffer, or when putting
200 text from several strings together in one string. You can also
201 explicitly convert a string's contents to either representation.
202
203 Emacs chooses the representation for a string based on the text from
204 which it is constructed. The general rule is to convert unibyte text
205 to multibyte text when combining it with other multibyte text, because
206 the multibyte representation is more general and can hold whatever
207 characters the unibyte text has.
208
209 When inserting text into a buffer, Emacs converts the text to the
210 buffer's representation, as specified by
211 @code{enable-multibyte-characters} in that buffer. In particular, when
212 you insert multibyte text into a unibyte buffer, Emacs converts the text
213 to unibyte, even though this conversion cannot in general preserve all
214 the characters that might be in the multibyte text. The other natural
215 alternative, to convert the buffer contents to multibyte, is not
216 acceptable because the buffer's representation is a choice made by the
217 user that cannot be overridden automatically.
218
219 Converting unibyte text to multibyte text leaves @acronym{ASCII}
220 characters unchanged, and converts bytes with codes 128 through 255 to
221 the multibyte representation of raw eight-bit bytes.
222
223 Converting multibyte text to unibyte converts all @acronym{ASCII}
224 and eight-bit characters to their single-byte form, but loses
225 information for non-@acronym{ASCII} characters by discarding all but
226 the low 8 bits of each character's codepoint. Converting unibyte text
227 to multibyte and back to unibyte reproduces the original unibyte text.
228
229 The next two functions either return the argument @var{string}, or a
230 newly created string with no text properties.
231
232 @defun string-to-multibyte string
233 This function returns a multibyte string containing the same sequence
234 of characters as @var{string}. If @var{string} is a multibyte string,
235 it is returned unchanged. The function assumes that @var{string}
236 includes only @acronym{ASCII} characters and raw 8-bit bytes; the
237 latter are converted to their multibyte representation corresponding
238 to the codepoints @code{#x3FFF80} through @code{#x3FFFFF}, inclusive
239 (@pxref{Text Representations, codepoints}).
240 @end defun
241
242 @defun string-to-unibyte string
243 This function returns a unibyte string containing the same sequence of
244 characters as @var{string}. It signals an error if @var{string}
245 contains a non-@acronym{ASCII} character. If @var{string} is a
246 unibyte string, it is returned unchanged. Use this function for
247 @var{string} arguments that contain only @acronym{ASCII} and eight-bit
248 characters.
249 @end defun
250
251 @c FIXME: Should `@var{character}' be `@var{byte}'?
252 @defun byte-to-string byte
253 @cindex byte to string
254 This function returns a unibyte string containing a single byte of
255 character data, @var{character}. It signals an error if
256 @var{character} is not an integer between 0 and 255.
257 @end defun
258
259 @defun multibyte-char-to-unibyte char
260 This converts the multibyte character @var{char} to a unibyte
261 character, and returns that character. If @var{char} is neither
262 @acronym{ASCII} nor eight-bit, the function returns @minus{}1.
263 @end defun
264
265 @defun unibyte-char-to-multibyte char
266 This convert the unibyte character @var{char} to a multibyte
267 character, assuming @var{char} is either @acronym{ASCII} or raw 8-bit
268 byte.
269 @end defun
270
271 @node Selecting a Representation
272 @section Selecting a Representation
273
274 Sometimes it is useful to examine an existing buffer or string as
275 multibyte when it was unibyte, or vice versa.
276
277 @defun set-buffer-multibyte multibyte
278 Set the representation type of the current buffer. If @var{multibyte}
279 is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
280 is @code{nil}, the buffer becomes unibyte.
281
282 This function leaves the buffer contents unchanged when viewed as a
283 sequence of bytes. As a consequence, it can change the contents
284 viewed as characters; for instance, a sequence of three bytes which is
285 treated as one character in multibyte representation will count as
286 three characters in unibyte representation. Eight-bit characters
287 representing raw bytes are an exception. They are represented by one
288 byte in a unibyte buffer, but when the buffer is set to multibyte,
289 they are converted to two-byte sequences, and vice versa.
290
291 This function sets @code{enable-multibyte-characters} to record which
292 representation is in use. It also adjusts various data in the buffer
293 (including overlays, text properties and markers) so that they cover the
294 same text as they did before.
295
296 This function signals an error if the buffer is narrowed, since the
297 narrowing might have occurred in the middle of multibyte character
298 sequences.
299
300 This function also signals an error if the buffer is an indirect
301 buffer. An indirect buffer always inherits the representation of its
302 base buffer.
303 @end defun
304
305 @defun string-as-unibyte string
306 If @var{string} is already a unibyte string, this function returns
307 @var{string} itself. Otherwise, it returns a new string with the same
308 bytes as @var{string}, but treating each byte as a separate character
309 (so that the value may have more characters than @var{string}); as an
310 exception, each eight-bit character representing a raw byte is
311 converted into a single byte. The newly-created string contains no
312 text properties.
313 @end defun
314
315 @defun string-as-multibyte string
316 If @var{string} is a multibyte string, this function returns
317 @var{string} itself. Otherwise, it returns a new string with the same
318 bytes as @var{string}, but treating each multibyte sequence as one
319 character. This means that the value may have fewer characters than
320 @var{string} has. If a byte sequence in @var{string} is invalid as a
321 multibyte representation of a single character, each byte in the
322 sequence is treated as a raw 8-bit byte. The newly-created string
323 contains no text properties.
324 @end defun
325
326 @node Character Codes
327 @section Character Codes
328 @cindex character codes
329
330 The unibyte and multibyte text representations use different
331 character codes. The valid character codes for unibyte representation
332 range from 0 to @code{#xFF} (255)---the values that can fit in one
333 byte. The valid character codes for multibyte representation range
334 from 0 to @code{#x3FFFFF}. In this code space, values 0 through
335 @code{#x7F} (127) are for @acronym{ASCII} characters, and values
336 @code{#x80} (128) through @code{#x3FFF7F} (4194175) are for
337 non-@acronym{ASCII} characters.
338
339 Emacs character codes are a superset of the Unicode standard.
340 Values 0 through @code{#x10FFFF} (1114111) correspond to Unicode
341 characters of the same codepoint; values @code{#x110000} (1114112)
342 through @code{#x3FFF7F} (4194175) represent characters that are not
343 unified with Unicode; and values @code{#x3FFF80} (4194176) through
344 @code{#x3FFFFF} (4194303) represent eight-bit raw bytes.
345
346 @defun characterp charcode
347 This returns @code{t} if @var{charcode} is a valid character, and
348 @code{nil} otherwise.
349
350 @example
351 @group
352 (characterp 65)
353 @result{} t
354 @end group
355 @group
356 (characterp 4194303)
357 @result{} t
358 @end group
359 @group
360 (characterp 4194304)
361 @result{} nil
362 @end group
363 @end example
364 @end defun
365
366 @cindex maximum value of character codepoint
367 @cindex codepoint, largest value
368 @defun max-char
369 This function returns the largest value that a valid character
370 codepoint can have.
371
372 @example
373 @group
374 (characterp (max-char))
375 @result{} t
376 @end group
377 @group
378 (characterp (1+ (max-char)))
379 @result{} nil
380 @end group
381 @end example
382 @end defun
383
384 @defun get-byte &optional pos string
385 This function returns the byte at character position @var{pos} in the
386 current buffer. If the current buffer is unibyte, this is literally
387 the byte at that position. If the buffer is multibyte, byte values of
388 @acronym{ASCII} characters are the same as character codepoints,
389 whereas eight-bit raw bytes are converted to their 8-bit codes. The
390 function signals an error if the character at @var{pos} is
391 non-@acronym{ASCII}.
392
393 The optional argument @var{string} means to get a byte value from that
394 string instead of the current buffer.
395 @end defun
396
397 @node Character Properties
398 @section Character Properties
399 @cindex character properties
400 A @dfn{character property} is a named attribute of a character that
401 specifies how the character behaves and how it should be handled
402 during text processing and display. Thus, character properties are an
403 important part of specifying the character's semantics.
404
405 @c FIXME: Use the latest URI of this chapter?
406 @c http://www.unicode.org/versions/latest/ch04.pdf
407 On the whole, Emacs follows the Unicode Standard in its implementation
408 of character properties. In particular, Emacs supports the
409 @uref{http://www.unicode.org/reports/tr23/, Unicode Character Property
410 Model}, and the Emacs character property database is derived from the
411 Unicode Character Database (@acronym{UCD}). See the
412 @uref{http://www.unicode.org/versions/Unicode6.2.0/ch04.pdf, Character
413 Properties chapter of the Unicode Standard}, for a detailed
414 description of Unicode character properties and their meaning. This
415 section assumes you are already familiar with that chapter of the
416 Unicode Standard, and want to apply that knowledge to Emacs Lisp
417 programs.
418
419 In Emacs, each property has a name, which is a symbol, and a set of
420 possible values, whose types depend on the property; if a character
421 does not have a certain property, the value is @code{nil}. As a
422 general rule, the names of character properties in Emacs are produced
423 from the corresponding Unicode properties by downcasing them and
424 replacing each @samp{_} character with a dash @samp{-}. For example,
425 @code{Canonical_Combining_Class} becomes
426 @code{canonical-combining-class}. However, sometimes we shorten the
427 names to make their use easier.
428
429 @cindex unassigned character codepoints
430 Some codepoints are left @dfn{unassigned} by the
431 @acronym{UCD}---they don't correspond to any character. The Unicode
432 Standard defines default values of properties for such codepoints;
433 they are mentioned below for each property.
434
435 Here is the full list of value types for all the character
436 properties that Emacs knows about:
437
438 @table @code
439 @item name
440 Corresponds to the @code{Name} Unicode property. The value is a
441 string consisting of upper-case Latin letters A to Z, digits, spaces,
442 and hyphen @samp{-} characters. For unassigned codepoints, the value
443 is @code{nil}.
444
445 @cindex unicode general category
446 @item general-category
447 Corresponds to the @code{General_Category} Unicode property. The
448 value is a symbol whose name is a 2-letter abbreviation of the
449 character's classification. For unassigned codepoints, the value
450 is @code{Cn}.
451
452 @item canonical-combining-class
453 Corresponds to the @code{Canonical_Combining_Class} Unicode property.
454 The value is an integer. For unassigned codepoints, the value
455 is zero.
456
457 @cindex bidirectional class of characters
458 @item bidi-class
459 Corresponds to the Unicode @code{Bidi_Class} property. The value is a
460 symbol whose name is the Unicode @dfn{directional type} of the
461 character. Emacs uses this property when it reorders bidirectional
462 text for display (@pxref{Bidirectional Display}). For unassigned
463 codepoints, the value depends on the code blocks to which the
464 codepoint belongs: most unassigned codepoints get the value of
465 @code{L} (strong L), but some get values of @code{AL} (Arabic letter)
466 or @code{R} (strong R).
467
468 @item decomposition
469 Corresponds to the Unicode properties @code{Decomposition_Type} and
470 @code{Decomposition_Value}. The value is a list, whose first element
471 may be a symbol representing a compatibility formatting tag, such as
472 @code{small}@footnote{The Unicode specification writes these tag names
473 inside @samp{<..>} brackets, but the tag names in Emacs do not include
474 the brackets; e.g., Unicode specifies @samp{<small>} where Emacs uses
475 @samp{small}. }; the other elements are characters that give the
476 compatibility decomposition sequence of this character. For
477 unassigned codepoints, the value is the character itself.
478
479 @item decimal-digit-value
480 Corresponds to the Unicode @code{Numeric_Value} property for
481 characters whose @code{Numeric_Type} is @samp{Decimal}. The value is
482 an integer. For unassigned codepoints, the value is
483 @code{nil}, which means @acronym{NaN}, or ``not-a-number''.
484
485 @item digit-value
486 Corresponds to the Unicode @code{Numeric_Value} property for
487 characters whose @code{Numeric_Type} is @samp{Digit}. The value is an
488 integer. Examples of such characters include compatibility
489 subscript and superscript digits, for which the value is the
490 corresponding number. For unassigned codepoints, the value is
491 @code{nil}, which means @acronym{NaN}.
492
493 @item numeric-value
494 Corresponds to the Unicode @code{Numeric_Value} property for
495 characters whose @code{Numeric_Type} is @samp{Numeric}. The value of
496 this property is a number. Examples of
497 characters that have this property include fractions, subscripts,
498 superscripts, Roman numerals, currency numerators, and encircled
499 numbers. For example, the value of this property for the character
500 @code{U+2155} (@sc{vulgar fraction one fifth}) is @code{0.2}. For
501 unassigned codepoints, the value is @code{nil}, which means
502 @acronym{NaN}.
503
504 @cindex mirroring of characters
505 @item mirrored
506 Corresponds to the Unicode @code{Bidi_Mirrored} property. The value
507 of this property is a symbol, either @code{Y} or @code{N}. For
508 unassigned codepoints, the value is @code{N}.
509
510 @item mirroring
511 Corresponds to the Unicode @code{Bidi_Mirroring_Glyph} property. The
512 value of this property is a character whose glyph represents the
513 mirror image of the character's glyph, or @code{nil} if there's no
514 defined mirroring glyph. All the characters whose @code{mirrored}
515 property is @code{N} have @code{nil} as their @code{mirroring}
516 property; however, some characters whose @code{mirrored} property is
517 @code{Y} also have @code{nil} for @code{mirroring}, because no
518 appropriate characters exist with mirrored glyphs. Emacs uses this
519 property to display mirror images of characters when appropriate
520 (@pxref{Bidirectional Display}). For unassigned codepoints, the value
521 is @code{nil}.
522
523 @item paired-bracket
524 Corresponds to the Unicode @code{Bidi_Paired_Bracket} property. The
525 value of this property is the codepoint of a character's @dfn{paired
526 bracket}, or @code{nil} if the character is not a bracket character.
527 This establishes a mapping between characters that are treated as
528 bracket pairs by the Unicode Bidirectional Algorithm; Emacs uses this
529 property when it decides how to reorder for display parentheses,
530 braces, and other similar characters (@pxref{Bidirectional Display}).
531
532 @item bracket-type
533 Corresponds to the Unicode @code{Bidi_Paired_Bracket_Type} property.
534 For characters whose @code{paired-bracket} property is non-@code{nil},
535 the value of this property is a symbol, either @code{o} (for opening
536 bracket characters) or @code{c} (for closing bracket characters). For
537 characters whose @code{paired-bracket} property is @code{nil}, the
538 value is the symbol @code{n} (None). Like @code{paired-bracket}, this
539 property is used for bidirectional display.
540
541 @item old-name
542 Corresponds to the Unicode @code{Unicode_1_Name} property. The value
543 is a string. Unassigned codepoints, and characters that have no value
544 for this property, the value is @code{nil}.
545
546 @item iso-10646-comment
547 Corresponds to the Unicode @code{ISO_Comment} property. The value is
548 a string. For unassigned codepoints, the value is an empty string.
549
550 @item uppercase
551 Corresponds to the Unicode @code{Simple_Uppercase_Mapping} property.
552 The value of this property is a single character. For unassigned
553 codepoints, the value is @code{nil}, which means the character itself.
554
555 @item lowercase
556 Corresponds to the Unicode @code{Simple_Lowercase_Mapping} property.
557 The value of this property is a single character. For unassigned
558 codepoints, the value is @code{nil}, which means the character itself.
559
560 @item titlecase
561 Corresponds to the Unicode @code{Simple_Titlecase_Mapping} property.
562 @dfn{Title case} is a special form of a character used when the first
563 character of a word needs to be capitalized. The value of this
564 property is a single character. For unassigned codepoints, the value
565 is @code{nil}, which means the character itself.
566 @end table
567
568 @defun get-char-code-property char propname
569 This function returns the value of @var{char}'s @var{propname} property.
570
571 @example
572 @group
573 (get-char-code-property ?\s 'general-category)
574 @result{} Zs
575 @end group
576 @group
577 (get-char-code-property ?1 'general-category)
578 @result{} Nd
579 @end group
580 @group
581 ;; subscript 4
582 (get-char-code-property ?\u2084 'digit-value)
583 @result{} 4
584 @end group
585 @group
586 ;; one fifth
587 (get-char-code-property ?\u2155 'numeric-value)
588 @result{} 0.2
589 @end group
590 @group
591 ;; Roman IV
592 (get-char-code-property ?\u2163 'numeric-value)
593 @result{} 4
594 @end group
595 @group
596 (get-char-code-property ?\( 'paired-bracket)
597 @result{} 41 ;; closing parenthesis
598 @end group
599 @group
600 (get-char-code-property ?\) 'bracket-type)
601 @result{} c
602 @end group
603 @end example
604 @end defun
605
606 @defun char-code-property-description prop value
607 This function returns the description string of property @var{prop}'s
608 @var{value}, or @code{nil} if @var{value} has no description.
609
610 @example
611 @group
612 (char-code-property-description 'general-category 'Zs)
613 @result{} "Separator, Space"
614 @end group
615 @group
616 (char-code-property-description 'general-category 'Nd)
617 @result{} "Number, Decimal Digit"
618 @end group
619 @group
620 (char-code-property-description 'numeric-value '1/5)
621 @result{} nil
622 @end group
623 @end example
624 @end defun
625
626 @defun put-char-code-property char propname value
627 This function stores @var{value} as the value of the property
628 @var{propname} for the character @var{char}.
629 @end defun
630
631 @defvar unicode-category-table
632 The value of this variable is a char-table (@pxref{Char-Tables}) that
633 specifies, for each character, its Unicode @code{General_Category}
634 property as a symbol.
635 @end defvar
636
637 @defvar char-script-table
638 @cindex script symbols
639 The value of this variable is a char-table that specifies, for each
640 character, a symbol whose name is the script to which the character
641 belongs, according to the Unicode Standard classification of the
642 Unicode code space into script-specific blocks. This char-table has a
643 single extra slot whose value is the list of all script symbols.
644 @end defvar
645
646 @defvar char-width-table
647 The value of this variable is a char-table that specifies the width of
648 each character in columns that it will occupy on the screen.
649 @end defvar
650
651 @defvar printable-chars
652 The value of this variable is a char-table that specifies, for each
653 character, whether it is printable or not. That is, if evaluating
654 @code{(aref printable-chars char)} results in @code{t}, the character
655 is printable, and if it results in @code{nil}, it is not.
656 @end defvar
657
658 @node Character Sets
659 @section Character Sets
660 @cindex character sets
661
662 @cindex charset
663 @cindex coded character set
664 An Emacs @dfn{character set}, or @dfn{charset}, is a set of characters
665 in which each character is assigned a numeric code point. (The
666 Unicode Standard calls this a @dfn{coded character set}.) Each Emacs
667 charset has a name which is a symbol. A single character can belong
668 to any number of different character sets, but it will generally have
669 a different code point in each charset. Examples of character sets
670 include @code{ascii}, @code{iso-8859-1}, @code{greek-iso8859-7}, and
671 @code{windows-1255}. The code point assigned to a character in a
672 charset is usually different from its code point used in Emacs buffers
673 and strings.
674
675 @cindex @code{emacs}, a charset
676 @cindex @code{unicode}, a charset
677 @cindex @code{eight-bit}, a charset
678 Emacs defines several special character sets. The character set
679 @code{unicode} includes all the characters whose Emacs code points are
680 in the range @code{0..#x10FFFF}. The character set @code{emacs}
681 includes all @acronym{ASCII} and non-@acronym{ASCII} characters.
682 Finally, the @code{eight-bit} charset includes the 8-bit raw bytes;
683 Emacs uses it to represent raw bytes encountered in text.
684
685 @defun charsetp object
686 Returns @code{t} if @var{object} is a symbol that names a character set,
687 @code{nil} otherwise.
688 @end defun
689
690 @defvar charset-list
691 The value is a list of all defined character set names.
692 @end defvar
693
694 @defun charset-priority-list &optional highestp
695 This function returns a list of all defined character sets ordered by
696 their priority. If @var{highestp} is non-@code{nil}, the function
697 returns a single character set of the highest priority.
698 @end defun
699
700 @defun set-charset-priority &rest charsets
701 This function makes @var{charsets} the highest priority character sets.
702 @end defun
703
704 @defun char-charset character &optional restriction
705 This function returns the name of the character set of highest
706 priority that @var{character} belongs to. @acronym{ASCII} characters
707 are an exception: for them, this function always returns @code{ascii}.
708
709 If @var{restriction} is non-@code{nil}, it should be a list of
710 charsets to search. Alternatively, it can be a coding system, in
711 which case the returned charset must be supported by that coding
712 system (@pxref{Coding Systems}).
713 @end defun
714
715 @c TODO: Explain the properties here and add indexes such as 'charset property'.
716 @defun charset-plist charset
717 This function returns the property list of the character set
718 @var{charset}. Although @var{charset} is a symbol, this is not the
719 same as the property list of that symbol. Charset properties include
720 important information about the charset, such as its documentation
721 string, short name, etc.
722 @end defun
723
724 @defun put-charset-property charset propname value
725 This function sets the @var{propname} property of @var{charset} to the
726 given @var{value}.
727 @end defun
728
729 @defun get-charset-property charset propname
730 This function returns the value of @var{charset}s property
731 @var{propname}.
732 @end defun
733
734 @deffn Command list-charset-chars charset
735 This command displays a list of characters in the character set
736 @var{charset}.
737 @end deffn
738
739 Emacs can convert between its internal representation of a character
740 and the character's codepoint in a specific charset. The following
741 two functions support these conversions.
742
743 @c FIXME: decode-char and encode-char accept and ignore an additional
744 @c argument @var{restriction}. When that argument actually makes a
745 @c difference, it should be documented here.
746 @defun decode-char charset code-point
747 This function decodes a character that is assigned a @var{code-point}
748 in @var{charset}, to the corresponding Emacs character, and returns
749 it. If @var{charset} doesn't contain a character of that code point,
750 the value is @code{nil}. If @var{code-point} doesn't fit in a Lisp
751 integer (@pxref{Integer Basics, most-positive-fixnum}), it can be
752 specified as a cons cell @code{(@var{high} . @var{low})}, where
753 @var{low} are the lower 16 bits of the value and @var{high} are the
754 high 16 bits.
755 @end defun
756
757 @defun encode-char char charset
758 This function returns the code point assigned to the character
759 @var{char} in @var{charset}. If the result does not fit in a Lisp
760 integer, it is returned as a cons cell @code{(@var{high} . @var{low})}
761 that fits the second argument of @code{decode-char} above. If
762 @var{charset} doesn't have a codepoint for @var{char}, the value is
763 @code{nil}.
764 @end defun
765
766 The following function comes in handy for applying a certain
767 function to all or part of the characters in a charset:
768
769 @defun map-charset-chars function charset &optional arg from-code to-code
770 Call @var{function} for characters in @var{charset}. @var{function}
771 is called with two arguments. The first one is a cons cell
772 @code{(@var{from} . @var{to})}, where @var{from} and @var{to}
773 indicate a range of characters contained in charset. The second
774 argument passed to @var{function} is @var{arg}.
775
776 By default, the range of codepoints passed to @var{function} includes
777 all the characters in @var{charset}, but optional arguments
778 @var{from-code} and @var{to-code} limit that to the range of
779 characters between these two codepoints of @var{charset}. If either
780 of them is @code{nil}, it defaults to the first or last codepoint of
781 @var{charset}, respectively.
782 @end defun
783
784 @node Scanning Charsets
785 @section Scanning for Character Sets
786
787 Sometimes it is useful to find out which character set a particular
788 character belongs to. One use for this is in determining which coding
789 systems (@pxref{Coding Systems}) are capable of representing all of
790 the text in question; another is to determine the font(s) for
791 displaying that text.
792
793 @defun charset-after &optional pos
794 This function returns the charset of highest priority containing the
795 character at position @var{pos} in the current buffer. If @var{pos}
796 is omitted or @code{nil}, it defaults to the current value of point.
797 If @var{pos} is out of range, the value is @code{nil}.
798 @end defun
799
800 @defun find-charset-region beg end &optional translation
801 This function returns a list of the character sets of highest priority
802 that contain characters in the current buffer between positions
803 @var{beg} and @var{end}.
804
805 The optional argument @var{translation} specifies a translation table
806 to use for scanning the text (@pxref{Translation of Characters}). If
807 it is non-@code{nil}, then each character in the region is translated
808 through this table, and the value returned describes the translated
809 characters instead of the characters actually in the buffer.
810 @end defun
811
812 @defun find-charset-string string &optional translation
813 This function returns a list of character sets of highest priority
814 that contain characters in @var{string}. It is just like
815 @code{find-charset-region}, except that it applies to the contents of
816 @var{string} instead of part of the current buffer.
817 @end defun
818
819 @node Translation of Characters
820 @section Translation of Characters
821 @cindex character translation tables
822 @cindex translation tables
823
824 A @dfn{translation table} is a char-table (@pxref{Char-Tables}) that
825 specifies a mapping of characters into characters. These tables are
826 used in encoding and decoding, and for other purposes. Some coding
827 systems specify their own particular translation tables; there are
828 also default translation tables which apply to all other coding
829 systems.
830
831 A translation table has two extra slots. The first is either
832 @code{nil} or a translation table that performs the reverse
833 translation; the second is the maximum number of characters to look up
834 for translating sequences of characters (see the description of
835 @code{make-translation-table-from-alist} below).
836
837 @defun make-translation-table &rest translations
838 This function returns a translation table based on the argument
839 @var{translations}. Each element of @var{translations} should be a
840 list of elements of the form @code{(@var{from} . @var{to})}; this says
841 to translate the character @var{from} into @var{to}.
842
843 The arguments and the forms in each argument are processed in order,
844 and if a previous form already translates @var{to} to some other
845 character, say @var{to-alt}, @var{from} is also translated to
846 @var{to-alt}.
847 @end defun
848
849 During decoding, the translation table's translations are applied to
850 the characters that result from ordinary decoding. If a coding system
851 has the property @code{:decode-translation-table}, that specifies the
852 translation table to use, or a list of translation tables to apply in
853 sequence. (This is a property of the coding system, as returned by
854 @code{coding-system-get}, not a property of the symbol that is the
855 coding system's name. @xref{Coding System Basics,, Basic Concepts of
856 Coding Systems}.) Finally, if
857 @code{standard-translation-table-for-decode} is non-@code{nil}, the
858 resulting characters are translated by that table.
859
860 During encoding, the translation table's translations are applied to
861 the characters in the buffer, and the result of translation is
862 actually encoded. If a coding system has property
863 @code{:encode-translation-table}, that specifies the translation table
864 to use, or a list of translation tables to apply in sequence. In
865 addition, if the variable @code{standard-translation-table-for-encode}
866 is non-@code{nil}, it specifies the translation table to use for
867 translating the result.
868
869 @defvar standard-translation-table-for-decode
870 This is the default translation table for decoding. If a coding
871 systems specifies its own translation tables, the table that is the
872 value of this variable, if non-@code{nil}, is applied after them.
873 @end defvar
874
875 @defvar standard-translation-table-for-encode
876 This is the default translation table for encoding. If a coding
877 systems specifies its own translation tables, the table that is the
878 value of this variable, if non-@code{nil}, is applied after them.
879 @end defvar
880
881 @c FIXME: This variable is obsolete since 23.1. We should mention
882 @c that here or simply remove this defvar. --xfq
883 @defvar translation-table-for-input
884 Self-inserting characters are translated through this translation
885 table before they are inserted. Search commands also translate their
886 input through this table, so they can compare more reliably with
887 what's in the buffer.
888
889 This variable automatically becomes buffer-local when set.
890 @end defvar
891
892 @defun make-translation-table-from-vector vec
893 This function returns a translation table made from @var{vec} that is
894 an array of 256 elements to map bytes (values 0 through #xFF) to
895 characters. Elements may be @code{nil} for untranslated bytes. The
896 returned table has a translation table for reverse mapping in the
897 first extra slot, and the value @code{1} in the second extra slot.
898
899 This function provides an easy way to make a private coding system
900 that maps each byte to a specific character. You can specify the
901 returned table and the reverse translation table using the properties
902 @code{:decode-translation-table} and @code{:encode-translation-table}
903 respectively in the @var{props} argument to
904 @code{define-coding-system}.
905 @end defun
906
907 @defun make-translation-table-from-alist alist
908 This function is similar to @code{make-translation-table} but returns
909 a complex translation table rather than a simple one-to-one mapping.
910 Each element of @var{alist} is of the form @code{(@var{from}
911 . @var{to})}, where @var{from} and @var{to} are either characters or
912 vectors specifying a sequence of characters. If @var{from} is a
913 character, that character is translated to @var{to} (i.e., to a
914 character or a character sequence). If @var{from} is a vector of
915 characters, that sequence is translated to @var{to}. The returned
916 table has a translation table for reverse mapping in the first extra
917 slot, and the maximum length of all the @var{from} character sequences
918 in the second extra slot.
919 @end defun
920
921 @node Coding Systems
922 @section Coding Systems
923
924 @cindex coding system
925 When Emacs reads or writes a file, and when Emacs sends text to a
926 subprocess or receives text from a subprocess, it normally performs
927 character code conversion and end-of-line conversion as specified
928 by a particular @dfn{coding system}.
929
930 How to define a coding system is an arcane matter, and is not
931 documented here.
932
933 @menu
934 * Coding System Basics:: Basic concepts.
935 * Encoding and I/O:: How file I/O functions handle coding systems.
936 * Lisp and Coding Systems:: Functions to operate on coding system names.
937 * User-Chosen Coding Systems:: Asking the user to choose a coding system.
938 * Default Coding Systems:: Controlling the default choices.
939 * Specifying Coding Systems:: Requesting a particular coding system
940 for a single file operation.
941 * Explicit Encoding:: Encoding or decoding text without doing I/O.
942 * Terminal I/O Encoding:: Use of encoding for terminal I/O.
943 @end menu
944
945 @node Coding System Basics
946 @subsection Basic Concepts of Coding Systems
947
948 @cindex character code conversion
949 @dfn{Character code conversion} involves conversion between the
950 internal representation of characters used inside Emacs and some other
951 encoding. Emacs supports many different encodings, in that it can
952 convert to and from them. For example, it can convert text to or from
953 encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and
954 several variants of ISO 2022. In some cases, Emacs supports several
955 alternative encodings for the same characters; for example, there are
956 three coding systems for the Cyrillic (Russian) alphabet: ISO,
957 Alternativnyj, and KOI8.
958
959 Every coding system specifies a particular set of character code
960 conversions, but the coding system @code{undecided} is special: it
961 leaves the choice unspecified, to be chosen heuristically for each
962 file, based on the file's data.
963
964 In general, a coding system doesn't guarantee roundtrip identity:
965 decoding a byte sequence using coding system, then encoding the
966 resulting text in the same coding system, can produce a different byte
967 sequence. But some coding systems do guarantee that the byte sequence
968 will be the same as what you originally decoded. Here are a few
969 examples:
970
971 @quotation
972 iso-8859-1, utf-8, big5, shift_jis, euc-jp
973 @end quotation
974
975 Encoding buffer text and then decoding the result can also fail to
976 reproduce the original text. For instance, if you encode a character
977 with a coding system which does not support that character, the result
978 is unpredictable, and thus decoding it using the same coding system
979 may produce a different text. Currently, Emacs can't report errors
980 that result from encoding unsupported characters.
981
982 @cindex EOL conversion
983 @cindex end-of-line conversion
984 @cindex line end conversion
985 @dfn{End of line conversion} handles three different conventions
986 used on various systems for representing end of line in files. The
987 Unix convention, used on GNU and Unix systems, is to use the linefeed
988 character (also called newline). The DOS convention, used on
989 MS-Windows and MS-DOS systems, is to use a carriage-return and a
990 linefeed at the end of a line. The Mac convention is to use just
991 carriage-return. (This was the convention used on the Macintosh
992 system prior to OS X.)
993
994 @cindex base coding system
995 @cindex variant coding system
996 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
997 conversion unspecified, to be chosen based on the data. @dfn{Variant
998 coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
999 @code{latin-1-mac} specify the end-of-line conversion explicitly as
1000 well. Most base coding systems have three corresponding variants whose
1001 names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
1002
1003 @vindex raw-text@r{ coding system}
1004 The coding system @code{raw-text} is special in that it prevents
1005 character code conversion, and causes the buffer visited with this
1006 coding system to be a unibyte buffer. For historical reasons, you can
1007 save both unibyte and multibyte text with this coding system. When
1008 you use @code{raw-text} to encode multibyte text, it does perform one
1009 character code conversion: it converts eight-bit characters to their
1010 single-byte external representation. @code{raw-text} does not specify
1011 the end-of-line conversion, allowing that to be determined as usual by
1012 the data, and has the usual three variants which specify the
1013 end-of-line conversion.
1014
1015 @vindex no-conversion@r{ coding system}
1016 @vindex binary@r{ coding system}
1017 @code{no-conversion} (and its alias @code{binary}) is equivalent to
1018 @code{raw-text-unix}: it specifies no conversion of either character
1019 codes or end-of-line.
1020
1021 @vindex emacs-internal@r{ coding system}
1022 @vindex utf-8-emacs@r{ coding system}
1023 The coding system @code{utf-8-emacs} specifies that the data is
1024 represented in the internal Emacs encoding (@pxref{Text
1025 Representations}). This is like @code{raw-text} in that no code
1026 conversion happens, but different in that the result is multibyte
1027 data. The name @code{emacs-internal} is an alias for
1028 @code{utf-8-emacs}.
1029
1030 @defun coding-system-get coding-system property
1031 This function returns the specified property of the coding system
1032 @var{coding-system}. Most coding system properties exist for internal
1033 purposes, but one that you might find useful is @code{:mime-charset}.
1034 That property's value is the name used in MIME for the character coding
1035 which this coding system can read and write. Examples:
1036
1037 @example
1038 (coding-system-get 'iso-latin-1 :mime-charset)
1039 @result{} iso-8859-1
1040 (coding-system-get 'iso-2022-cn :mime-charset)
1041 @result{} iso-2022-cn
1042 (coding-system-get 'cyrillic-koi8 :mime-charset)
1043 @result{} koi8-r
1044 @end example
1045
1046 The value of the @code{:mime-charset} property is also defined
1047 as an alias for the coding system.
1048 @end defun
1049
1050 @cindex alias, for coding systems
1051 @defun coding-system-aliases coding-system
1052 This function returns the list of aliases of @var{coding-system}.
1053 @end defun
1054
1055 @node Encoding and I/O
1056 @subsection Encoding and I/O
1057
1058 The principal purpose of coding systems is for use in reading and
1059 writing files. The function @code{insert-file-contents} uses a coding
1060 system to decode the file data, and @code{write-region} uses one to
1061 encode the buffer contents.
1062
1063 You can specify the coding system to use either explicitly
1064 (@pxref{Specifying Coding Systems}), or implicitly using a default
1065 mechanism (@pxref{Default Coding Systems}). But these methods may not
1066 completely specify what to do. For example, they may choose a coding
1067 system such as @code{undefined} which leaves the character code
1068 conversion to be determined from the data. In these cases, the I/O
1069 operation finishes the job of choosing a coding system. Very often
1070 you will want to find out afterwards which coding system was chosen.
1071
1072 @defvar buffer-file-coding-system
1073 This buffer-local variable records the coding system used for saving the
1074 buffer and for writing part of the buffer with @code{write-region}. If
1075 the text to be written cannot be safely encoded using the coding system
1076 specified by this variable, these operations select an alternative
1077 encoding by calling the function @code{select-safe-coding-system}
1078 (@pxref{User-Chosen Coding Systems}). If selecting a different encoding
1079 requires to ask the user to specify a coding system,
1080 @code{buffer-file-coding-system} is updated to the newly selected coding
1081 system.
1082
1083 @code{buffer-file-coding-system} does @emph{not} affect sending text
1084 to a subprocess.
1085 @end defvar
1086
1087 @defvar save-buffer-coding-system
1088 This variable specifies the coding system for saving the buffer (by
1089 overriding @code{buffer-file-coding-system}). Note that it is not used
1090 for @code{write-region}.
1091
1092 When a command to save the buffer starts out to use
1093 @code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
1094 and that coding system cannot handle
1095 the actual text in the buffer, the command asks the user to choose
1096 another coding system (by calling @code{select-safe-coding-system}).
1097 After that happens, the command also updates
1098 @code{buffer-file-coding-system} to represent the coding system that
1099 the user specified.
1100 @end defvar
1101
1102 @defvar last-coding-system-used
1103 I/O operations for files and subprocesses set this variable to the
1104 coding system name that was used. The explicit encoding and decoding
1105 functions (@pxref{Explicit Encoding}) set it too.
1106
1107 @strong{Warning:} Since receiving subprocess output sets this variable,
1108 it can change whenever Emacs waits; therefore, you should copy the
1109 value shortly after the function call that stores the value you are
1110 interested in.
1111 @end defvar
1112
1113 The variable @code{selection-coding-system} specifies how to encode
1114 selections for the window system. @xref{Window System Selections}.
1115
1116 @defvar file-name-coding-system
1117 The variable @code{file-name-coding-system} specifies the coding
1118 system to use for encoding file names. Emacs encodes file names using
1119 that coding system for all file operations. If
1120 @code{file-name-coding-system} is @code{nil}, Emacs uses a default
1121 coding system determined by the selected language environment. In the
1122 default language environment, any non-@acronym{ASCII} characters in
1123 file names are not encoded specially; they appear in the file system
1124 using the internal Emacs representation.
1125 @end defvar
1126
1127 @strong{Warning:} if you change @code{file-name-coding-system} (or
1128 the language environment) in the middle of an Emacs session, problems
1129 can result if you have already visited files whose names were encoded
1130 using the earlier coding system and are handled differently under the
1131 new coding system. If you try to save one of these buffers under the
1132 visited file name, saving may use the wrong file name, or it may get
1133 an error. If such a problem happens, use @kbd{C-x C-w} to specify a
1134 new file name for that buffer.
1135
1136 @cindex file-name encoding, MS-Windows
1137 On Windows 2000 and later, Emacs by default uses Unicode APIs to
1138 pass file names to the OS, so the value of
1139 @code{file-name-coding-system} is largely ignored. Lisp applications
1140 that need to encode or decode file names on the Lisp level should use
1141 @code{utf-8} coding-system when @code{system-type} is
1142 @code{windows-nt}; the conversion of UTF-8 encoded file names to the
1143 encoding appropriate for communicating with the OS is performed
1144 internally by Emacs.
1145
1146 @node Lisp and Coding Systems
1147 @subsection Coding Systems in Lisp
1148
1149 Here are the Lisp facilities for working with coding systems:
1150
1151 @cindex list all coding systems
1152 @defun coding-system-list &optional base-only
1153 This function returns a list of all coding system names (symbols). If
1154 @var{base-only} is non-@code{nil}, the value includes only the
1155 base coding systems. Otherwise, it includes alias and variant coding
1156 systems as well.
1157 @end defun
1158
1159 @defun coding-system-p object
1160 This function returns @code{t} if @var{object} is a coding system
1161 name or @code{nil}.
1162 @end defun
1163
1164 @cindex validity of coding system
1165 @cindex coding system, validity check
1166 @defun check-coding-system coding-system
1167 This function checks the validity of @var{coding-system}. If that is
1168 valid, it returns @var{coding-system}. If @var{coding-system} is
1169 @code{nil}, the function return @code{nil}. For any other values, it
1170 signals an error whose @code{error-symbol} is @code{coding-system-error}
1171 (@pxref{Signaling Errors, signal}).
1172 @end defun
1173
1174 @cindex eol type of coding system
1175 @defun coding-system-eol-type coding-system
1176 This function returns the type of end-of-line (a.k.a.@: @dfn{eol})
1177 conversion used by @var{coding-system}. If @var{coding-system}
1178 specifies a certain eol conversion, the return value is an integer 0,
1179 1, or 2, standing for @code{unix}, @code{dos}, and @code{mac},
1180 respectively. If @var{coding-system} doesn't specify eol conversion
1181 explicitly, the return value is a vector of coding systems, each one
1182 with one of the possible eol conversion types, like this:
1183
1184 @lisp
1185 (coding-system-eol-type 'latin-1)
1186 @result{} [latin-1-unix latin-1-dos latin-1-mac]
1187 @end lisp
1188
1189 @noindent
1190 If this function returns a vector, Emacs will decide, as part of the
1191 text encoding or decoding process, what eol conversion to use. For
1192 decoding, the end-of-line format of the text is auto-detected, and the
1193 eol conversion is set to match it (e.g., DOS-style CRLF format will
1194 imply @code{dos} eol conversion). For encoding, the eol conversion is
1195 taken from the appropriate default coding system (e.g.,
1196 default value of @code{buffer-file-coding-system} for
1197 @code{buffer-file-coding-system}), or from the default eol conversion
1198 appropriate for the underlying platform.
1199 @end defun
1200
1201 @cindex eol conversion of coding system
1202 @defun coding-system-change-eol-conversion coding-system eol-type
1203 This function returns a coding system which is like @var{coding-system}
1204 except for its eol conversion, which is specified by @code{eol-type}.
1205 @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
1206 @code{nil}. If it is @code{nil}, the returned coding system determines
1207 the end-of-line conversion from the data.
1208
1209 @var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
1210 @code{dos} and @code{mac}, respectively.
1211 @end defun
1212
1213 @cindex text conversion of coding system
1214 @defun coding-system-change-text-conversion eol-coding text-coding
1215 This function returns a coding system which uses the end-of-line
1216 conversion of @var{eol-coding}, and the text conversion of
1217 @var{text-coding}. If @var{text-coding} is @code{nil}, it returns
1218 @code{undecided}, or one of its variants according to @var{eol-coding}.
1219 @end defun
1220
1221 @cindex safely encode region
1222 @cindex coding systems for encoding region
1223 @defun find-coding-systems-region from to
1224 This function returns a list of coding systems that could be used to
1225 encode a text between @var{from} and @var{to}. All coding systems in
1226 the list can safely encode any multibyte characters in that portion of
1227 the text.
1228
1229 If the text contains no multibyte characters, the function returns the
1230 list @code{(undecided)}.
1231 @end defun
1232
1233 @cindex safely encode a string
1234 @cindex coding systems for encoding a string
1235 @defun find-coding-systems-string string
1236 This function returns a list of coding systems that could be used to
1237 encode the text of @var{string}. All coding systems in the list can
1238 safely encode any multibyte characters in @var{string}. If the text
1239 contains no multibyte characters, this returns the list
1240 @code{(undecided)}.
1241 @end defun
1242
1243 @cindex charset, coding systems to encode
1244 @cindex safely encode characters in a charset
1245 @defun find-coding-systems-for-charsets charsets
1246 This function returns a list of coding systems that could be used to
1247 encode all the character sets in the list @var{charsets}.
1248 @end defun
1249
1250 @defun check-coding-systems-region start end coding-system-list
1251 This function checks whether coding systems in the list
1252 @code{coding-system-list} can encode all the characters in the region
1253 between @var{start} and @var{end}. If all of the coding systems in
1254 the list can encode the specified text, the function returns
1255 @code{nil}. If some coding systems cannot encode some of the
1256 characters, the value is an alist, each element of which has the form
1257 @code{(@var{coding-system1} @var{pos1} @var{pos2} @dots{})}, meaning
1258 that @var{coding-system1} cannot encode characters at buffer positions
1259 @var{pos1}, @var{pos2}, @enddots{}.
1260
1261 @var{start} may be a string, in which case @var{end} is ignored and
1262 the returned value references string indices instead of buffer
1263 positions.
1264 @end defun
1265
1266 @defun detect-coding-region start end &optional highest
1267 This function chooses a plausible coding system for decoding the text
1268 from @var{start} to @var{end}. This text should be a byte sequence,
1269 i.e., unibyte text or multibyte text with only @acronym{ASCII} and
1270 eight-bit characters (@pxref{Explicit Encoding}).
1271
1272 Normally this function returns a list of coding systems that could
1273 handle decoding the text that was scanned. They are listed in order of
1274 decreasing priority. But if @var{highest} is non-@code{nil}, then the
1275 return value is just one coding system, the one that is highest in
1276 priority.
1277
1278 If the region contains only @acronym{ASCII} characters except for such
1279 ISO-2022 control characters ISO-2022 as @code{ESC}, the value is
1280 @code{undecided} or @code{(undecided)}, or a variant specifying
1281 end-of-line conversion, if that can be deduced from the text.
1282
1283 If the region contains null bytes, the value is @code{no-conversion},
1284 even if the region contains text encoded in some coding system.
1285 @end defun
1286
1287 @defun detect-coding-string string &optional highest
1288 This function is like @code{detect-coding-region} except that it
1289 operates on the contents of @var{string} instead of bytes in the buffer.
1290 @end defun
1291
1292 @cindex null bytes, and decoding text
1293 @defvar inhibit-null-byte-detection
1294 If this variable has a non-@code{nil} value, null bytes are ignored
1295 when detecting the encoding of a region or a string. This allows to
1296 correctly detect the encoding of text that contains null bytes, such
1297 as Info files with Index nodes.
1298 @end defvar
1299
1300 @defvar inhibit-iso-escape-detection
1301 If this variable has a non-@code{nil} value, ISO-2022 escape sequences
1302 are ignored when detecting the encoding of a region or a string. The
1303 result is that no text is ever detected as encoded in some ISO-2022
1304 encoding, and all escape sequences become visible in a buffer.
1305 @strong{Warning:} @emph{Use this variable with extreme caution,
1306 because many files in the Emacs distribution use ISO-2022 encoding.}
1307 @end defvar
1308
1309 @cindex charsets supported by a coding system
1310 @defun coding-system-charset-list coding-system
1311 This function returns the list of character sets (@pxref{Character
1312 Sets}) supported by @var{coding-system}. Some coding systems that
1313 support too many character sets to list them all yield special values:
1314 @itemize @bullet
1315 @item
1316 If @var{coding-system} supports all Emacs characters, the value is
1317 @code{(emacs)}.
1318 @item
1319 If @var{coding-system} supports all Unicode characters, the value is
1320 @code{(unicode)}.
1321 @item
1322 If @var{coding-system} supports all ISO-2022 charsets, the value is
1323 @code{iso-2022}.
1324 @item
1325 If @var{coding-system} supports all the characters in the internal
1326 coding system used by Emacs version 21 (prior to the implementation of
1327 internal Unicode support), the value is @code{emacs-mule}.
1328 @end itemize
1329 @end defun
1330
1331 @xref{Coding systems for a subprocess,, Process Information}, in
1332 particular the description of the functions
1333 @code{process-coding-system} and @code{set-process-coding-system}, for
1334 how to examine or set the coding systems used for I/O to a subprocess.
1335
1336 @node User-Chosen Coding Systems
1337 @subsection User-Chosen Coding Systems
1338
1339 @cindex select safe coding system
1340 @defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
1341 This function selects a coding system for encoding specified text,
1342 asking the user to choose if necessary. Normally the specified text
1343 is the text in the current buffer between @var{from} and @var{to}. If
1344 @var{from} is a string, the string specifies the text to encode, and
1345 @var{to} is ignored.
1346
1347 If the specified text includes raw bytes (@pxref{Text
1348 Representations}), @code{select-safe-coding-system} suggests
1349 @code{raw-text} for its encoding.
1350
1351 If @var{default-coding-system} is non-@code{nil}, that is the first
1352 coding system to try; if that can handle the text,
1353 @code{select-safe-coding-system} returns that coding system. It can
1354 also be a list of coding systems; then the function tries each of them
1355 one by one. After trying all of them, it next tries the current
1356 buffer's value of @code{buffer-file-coding-system} (if it is not
1357 @code{undecided}), then the default value of
1358 @code{buffer-file-coding-system} and finally the user's most
1359 preferred coding system, which the user can set using the command
1360 @code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
1361 Coding Systems, emacs, The GNU Emacs Manual}).
1362
1363 If one of those coding systems can safely encode all the specified
1364 text, @code{select-safe-coding-system} chooses it and returns it.
1365 Otherwise, it asks the user to choose from a list of coding systems
1366 which can encode all the text, and returns the user's choice.
1367
1368 @var{default-coding-system} can also be a list whose first element is
1369 t and whose other elements are coding systems. Then, if no coding
1370 system in the list can handle the text, @code{select-safe-coding-system}
1371 queries the user immediately, without trying any of the three
1372 alternatives described above.
1373
1374 The optional argument @var{accept-default-p}, if non-@code{nil},
1375 should be a function to determine whether a coding system selected
1376 without user interaction is acceptable. @code{select-safe-coding-system}
1377 calls this function with one argument, the base coding system of the
1378 selected coding system. If @var{accept-default-p} returns @code{nil},
1379 @code{select-safe-coding-system} rejects the silently selected coding
1380 system, and asks the user to select a coding system from a list of
1381 possible candidates.
1382
1383 @vindex select-safe-coding-system-accept-default-p
1384 If the variable @code{select-safe-coding-system-accept-default-p} is
1385 non-@code{nil}, it should be a function taking a single argument.
1386 It is used in place of @var{accept-default-p}, overriding any
1387 value supplied for this argument.
1388
1389 As a final step, before returning the chosen coding system,
1390 @code{select-safe-coding-system} checks whether that coding system is
1391 consistent with what would be selected if the contents of the region
1392 were read from a file. (If not, this could lead to data corruption in
1393 a file subsequently re-visited and edited.) Normally,
1394 @code{select-safe-coding-system} uses @code{buffer-file-name} as the
1395 file for this purpose, but if @var{file} is non-@code{nil}, it uses
1396 that file instead (this can be relevant for @code{write-region} and
1397 similar functions). If it detects an apparent inconsistency,
1398 @code{select-safe-coding-system} queries the user before selecting the
1399 coding system.
1400 @end defun
1401
1402 Here are two functions you can use to let the user specify a coding
1403 system, with completion. @xref{Completion}.
1404
1405 @defun read-coding-system prompt &optional default
1406 This function reads a coding system using the minibuffer, prompting with
1407 string @var{prompt}, and returns the coding system name as a symbol. If
1408 the user enters null input, @var{default} specifies which coding system
1409 to return. It should be a symbol or a string.
1410 @end defun
1411
1412 @defun read-non-nil-coding-system prompt
1413 This function reads a coding system using the minibuffer, prompting with
1414 string @var{prompt}, and returns the coding system name as a symbol. If
1415 the user tries to enter null input, it asks the user to try again.
1416 @xref{Coding Systems}.
1417 @end defun
1418
1419 @node Default Coding Systems
1420 @subsection Default Coding Systems
1421 @cindex default coding system
1422 @cindex coding system, automatically determined
1423
1424 This section describes variables that specify the default coding
1425 system for certain files or when running certain subprograms, and the
1426 function that I/O operations use to access them.
1427
1428 The idea of these variables is that you set them once and for all to the
1429 defaults you want, and then do not change them again. To specify a
1430 particular coding system for a particular operation in a Lisp program,
1431 don't change these variables; instead, override them using
1432 @code{coding-system-for-read} and @code{coding-system-for-write}
1433 (@pxref{Specifying Coding Systems}).
1434
1435 @cindex file contents, and default coding system
1436 @defopt auto-coding-regexp-alist
1437 This variable is an alist of text patterns and corresponding coding
1438 systems. Each element has the form @code{(@var{regexp}
1439 . @var{coding-system})}; a file whose first few kilobytes match
1440 @var{regexp} is decoded with @var{coding-system} when its contents are
1441 read into a buffer. The settings in this alist take priority over
1442 @code{coding:} tags in the files and the contents of
1443 @code{file-coding-system-alist} (see below). The default value is set
1444 so that Emacs automatically recognizes mail files in Babyl format and
1445 reads them with no code conversions.
1446 @end defopt
1447
1448 @cindex file name, and default coding system
1449 @defopt file-coding-system-alist
1450 This variable is an alist that specifies the coding systems to use for
1451 reading and writing particular files. Each element has the form
1452 @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
1453 expression that matches certain file names. The element applies to file
1454 names that match @var{pattern}.
1455
1456 The @sc{cdr} of the element, @var{coding}, should be either a coding
1457 system, a cons cell containing two coding systems, or a function name (a
1458 symbol with a function definition). If @var{coding} is a coding system,
1459 that coding system is used for both reading the file and writing it. If
1460 @var{coding} is a cons cell containing two coding systems, its @sc{car}
1461 specifies the coding system for decoding, and its @sc{cdr} specifies the
1462 coding system for encoding.
1463
1464 If @var{coding} is a function name, the function should take one
1465 argument, a list of all arguments passed to
1466 @code{find-operation-coding-system}. It must return a coding system
1467 or a cons cell containing two coding systems. This value has the same
1468 meaning as described above.
1469
1470 If @var{coding} (or what returned by the above function) is
1471 @code{undecided}, the normal code-detection is performed.
1472 @end defopt
1473
1474 @defopt auto-coding-alist
1475 This variable is an alist that specifies the coding systems to use for
1476 reading and writing particular files. Its form is like that of
1477 @code{file-coding-system-alist}, but, unlike the latter, this variable
1478 takes priority over any @code{coding:} tags in the file.
1479 @end defopt
1480
1481 @cindex program name, and default coding system
1482 @defvar process-coding-system-alist
1483 This variable is an alist specifying which coding systems to use for a
1484 subprocess, depending on which program is running in the subprocess. It
1485 works like @code{file-coding-system-alist}, except that @var{pattern} is
1486 matched against the program name used to start the subprocess. The coding
1487 system or systems specified in this alist are used to initialize the
1488 coding systems used for I/O to the subprocess, but you can specify
1489 other coding systems later using @code{set-process-coding-system}.
1490 @end defvar
1491
1492 @strong{Warning:} Coding systems such as @code{undecided}, which
1493 determine the coding system from the data, do not work entirely reliably
1494 with asynchronous subprocess output. This is because Emacs handles
1495 asynchronous subprocess output in batches, as it arrives. If the coding
1496 system leaves the character code conversion unspecified, or leaves the
1497 end-of-line conversion unspecified, Emacs must try to detect the proper
1498 conversion from one batch at a time, and this does not always work.
1499
1500 Therefore, with an asynchronous subprocess, if at all possible, use a
1501 coding system which determines both the character code conversion and
1502 the end of line conversion---that is, one like @code{latin-1-unix},
1503 rather than @code{undecided} or @code{latin-1}.
1504
1505 @cindex port number, and default coding system
1506 @cindex network service name, and default coding system
1507 @defvar network-coding-system-alist
1508 This variable is an alist that specifies the coding system to use for
1509 network streams. It works much like @code{file-coding-system-alist},
1510 with the difference that the @var{pattern} in an element may be either a
1511 port number or a regular expression. If it is a regular expression, it
1512 is matched against the network service name used to open the network
1513 stream.
1514 @end defvar
1515
1516 @defvar default-process-coding-system
1517 This variable specifies the coding systems to use for subprocess (and
1518 network stream) input and output, when nothing else specifies what to
1519 do.
1520
1521 The value should be a cons cell of the form @code{(@var{input-coding}
1522 . @var{output-coding})}. Here @var{input-coding} applies to input from
1523 the subprocess, and @var{output-coding} applies to output to it.
1524 @end defvar
1525
1526 @cindex default coding system, functions to determine
1527 @defopt auto-coding-functions
1528 This variable holds a list of functions that try to determine a
1529 coding system for a file based on its undecoded contents.
1530
1531 Each function in this list should be written to look at text in the
1532 current buffer, but should not modify it in any way. The buffer will
1533 contain undecoded text of parts of the file. Each function should
1534 take one argument, @var{size}, which tells it how many characters to
1535 look at, starting from point. If the function succeeds in determining
1536 a coding system for the file, it should return that coding system.
1537 Otherwise, it should return @code{nil}.
1538
1539 If a file has a @samp{coding:} tag, that takes precedence, so these
1540 functions won't be called.
1541 @end defopt
1542
1543 @defun find-auto-coding filename size
1544 This function tries to determine a suitable coding system for
1545 @var{filename}. It examines the buffer visiting the named file, using
1546 the variables documented above in sequence, until it finds a match for
1547 one of the rules specified by these variables. It then returns a cons
1548 cell of the form @code{(@var{coding} . @var{source})}, where
1549 @var{coding} is the coding system to use and @var{source} is a symbol,
1550 one of @code{auto-coding-alist}, @code{auto-coding-regexp-alist},
1551 @code{:coding}, or @code{auto-coding-functions}, indicating which one
1552 supplied the matching rule. The value @code{:coding} means the coding
1553 system was specified by the @code{coding:} tag in the file
1554 (@pxref{Specify Coding,, coding tag, emacs, The GNU Emacs Manual}).
1555 The order of looking for a matching rule is @code{auto-coding-alist}
1556 first, then @code{auto-coding-regexp-alist}, then the @code{coding:}
1557 tag, and lastly @code{auto-coding-functions}. If no matching rule was
1558 found, the function returns @code{nil}.
1559
1560 The second argument @var{size} is the size of text, in characters,
1561 following point. The function examines text only within @var{size}
1562 characters after point. Normally, the buffer should be positioned at
1563 the beginning when this function is called, because one of the places
1564 for the @code{coding:} tag is the first one or two lines of the file;
1565 in that case, @var{size} should be the size of the buffer.
1566 @end defun
1567
1568 @defun set-auto-coding filename size
1569 This function returns a suitable coding system for file
1570 @var{filename}. It uses @code{find-auto-coding} to find the coding
1571 system. If no coding system could be determined, the function returns
1572 @code{nil}. The meaning of the argument @var{size} is like in
1573 @code{find-auto-coding}.
1574 @end defun
1575
1576 @defun find-operation-coding-system operation &rest arguments
1577 This function returns the coding system to use (by default) for
1578 performing @var{operation} with @var{arguments}. The value has this
1579 form:
1580
1581 @example
1582 (@var{decoding-system} . @var{encoding-system})
1583 @end example
1584
1585 The first element, @var{decoding-system}, is the coding system to use
1586 for decoding (in case @var{operation} does decoding), and
1587 @var{encoding-system} is the coding system for encoding (in case
1588 @var{operation} does encoding).
1589
1590 The argument @var{operation} is a symbol; it should be one of
1591 @code{write-region}, @code{start-process}, @code{call-process},
1592 @code{call-process-region}, @code{insert-file-contents}, or
1593 @code{open-network-stream}. These are the names of the Emacs I/O
1594 primitives that can do character code and eol conversion.
1595
1596 The remaining arguments should be the same arguments that might be given
1597 to the corresponding I/O primitive. Depending on the primitive, one
1598 of those arguments is selected as the @dfn{target}. For example, if
1599 @var{operation} does file I/O, whichever argument specifies the file
1600 name is the target. For subprocess primitives, the process name is the
1601 target. For @code{open-network-stream}, the target is the service name
1602 or port number.
1603
1604 Depending on @var{operation}, this function looks up the target in
1605 @code{file-coding-system-alist}, @code{process-coding-system-alist},
1606 or @code{network-coding-system-alist}. If the target is found in the
1607 alist, @code{find-operation-coding-system} returns its association in
1608 the alist; otherwise it returns @code{nil}.
1609
1610 If @var{operation} is @code{insert-file-contents}, the argument
1611 corresponding to the target may be a cons cell of the form
1612 @code{(@var{filename} . @var{buffer})}. In that case, @var{filename}
1613 is a file name to look up in @code{file-coding-system-alist}, and
1614 @var{buffer} is a buffer that contains the file's contents (not yet
1615 decoded). If @code{file-coding-system-alist} specifies a function to
1616 call for this file, and that function needs to examine the file's
1617 contents (as it usually does), it should examine the contents of
1618 @var{buffer} instead of reading the file.
1619 @end defun
1620
1621 @node Specifying Coding Systems
1622 @subsection Specifying a Coding System for One Operation
1623
1624 You can specify the coding system for a specific operation by binding
1625 the variables @code{coding-system-for-read} and/or
1626 @code{coding-system-for-write}.
1627
1628 @defvar coding-system-for-read
1629 If this variable is non-@code{nil}, it specifies the coding system to
1630 use for reading a file, or for input from a synchronous subprocess.
1631
1632 It also applies to any asynchronous subprocess or network stream, but in
1633 a different way: the value of @code{coding-system-for-read} when you
1634 start the subprocess or open the network stream specifies the input
1635 decoding method for that subprocess or network stream. It remains in
1636 use for that subprocess or network stream unless and until overridden.
1637
1638 The right way to use this variable is to bind it with @code{let} for a
1639 specific I/O operation. Its global value is normally @code{nil}, and
1640 you should not globally set it to any other value. Here is an example
1641 of the right way to use the variable:
1642
1643 @example
1644 ;; @r{Read the file with no character code conversion.}
1645 (let ((coding-system-for-read 'no-conversion))
1646 (insert-file-contents filename))
1647 @end example
1648
1649 When its value is non-@code{nil}, this variable takes precedence over
1650 all other methods of specifying a coding system to use for input,
1651 including @code{file-coding-system-alist},
1652 @code{process-coding-system-alist} and
1653 @code{network-coding-system-alist}.
1654 @end defvar
1655
1656 @defvar coding-system-for-write
1657 This works much like @code{coding-system-for-read}, except that it
1658 applies to output rather than input. It affects writing to files,
1659 as well as sending output to subprocesses and net connections.
1660
1661 When a single operation does both input and output, as do
1662 @code{call-process-region} and @code{start-process}, both
1663 @code{coding-system-for-read} and @code{coding-system-for-write}
1664 affect it.
1665 @end defvar
1666
1667 @defopt inhibit-eol-conversion
1668 When this variable is non-@code{nil}, no end-of-line conversion is done,
1669 no matter which coding system is specified. This applies to all the
1670 Emacs I/O and subprocess primitives, and to the explicit encoding and
1671 decoding functions (@pxref{Explicit Encoding}).
1672 @end defopt
1673
1674 @cindex priority order of coding systems
1675 @cindex coding systems, priority
1676 Sometimes, you need to prefer several coding systems for some
1677 operation, rather than fix a single one. Emacs lets you specify a
1678 priority order for using coding systems. This ordering affects the
1679 sorting of lists of coding systems returned by functions such as
1680 @code{find-coding-systems-region} (@pxref{Lisp and Coding Systems}).
1681
1682 @defun coding-system-priority-list &optional highestp
1683 This function returns the list of coding systems in the order of their
1684 current priorities. Optional argument @var{highestp}, if
1685 non-@code{nil}, means return only the highest priority coding system.
1686 @end defun
1687
1688 @defun set-coding-system-priority &rest coding-systems
1689 This function puts @var{coding-systems} at the beginning of the
1690 priority list for coding systems, thus making their priority higher
1691 than all the rest.
1692 @end defun
1693
1694 @defmac with-coding-priority coding-systems &rest body@dots{}
1695 This macro execute @var{body}, like @code{progn} does
1696 (@pxref{Sequencing, progn}), with @var{coding-systems} at the front of
1697 the priority list for coding systems. @var{coding-systems} should be
1698 a list of coding systems to prefer during execution of @var{body}.
1699 @end defmac
1700
1701 @node Explicit Encoding
1702 @subsection Explicit Encoding and Decoding
1703 @cindex encoding in coding systems
1704 @cindex decoding in coding systems
1705
1706 All the operations that transfer text in and out of Emacs have the
1707 ability to use a coding system to encode or decode the text.
1708 You can also explicitly encode and decode text using the functions
1709 in this section.
1710
1711 The result of encoding, and the input to decoding, are not ordinary
1712 text. They logically consist of a series of byte values; that is, a
1713 series of @acronym{ASCII} and eight-bit characters. In unibyte
1714 buffers and strings, these characters have codes in the range 0
1715 through #xFF (255). In a multibyte buffer or string, eight-bit
1716 characters have character codes higher than #xFF (@pxref{Text
1717 Representations}), but Emacs transparently converts them to their
1718 single-byte values when you encode or decode such text.
1719
1720 The usual way to read a file into a buffer as a sequence of bytes, so
1721 you can decode the contents explicitly, is with
1722 @code{insert-file-contents-literally} (@pxref{Reading from Files});
1723 alternatively, specify a non-@code{nil} @var{rawfile} argument when
1724 visiting a file with @code{find-file-noselect}. These methods result in
1725 a unibyte buffer.
1726
1727 The usual way to use the byte sequence that results from explicitly
1728 encoding text is to copy it to a file or process---for example, to write
1729 it with @code{write-region} (@pxref{Writing to Files}), and suppress
1730 encoding by binding @code{coding-system-for-write} to
1731 @code{no-conversion}.
1732
1733 Here are the functions to perform explicit encoding or decoding. The
1734 encoding functions produce sequences of bytes; the decoding functions
1735 are meant to operate on sequences of bytes. All of these functions
1736 discard text properties. They also set @code{last-coding-system-used}
1737 to the precise coding system they used.
1738
1739 @deffn Command encode-coding-region start end coding-system &optional destination
1740 This command encodes the text from @var{start} to @var{end} according
1741 to coding system @var{coding-system}. Normally, the encoded text
1742 replaces the original text in the buffer, but the optional argument
1743 @var{destination} can change that. If @var{destination} is a buffer,
1744 the encoded text is inserted in that buffer after point (point does
1745 not move); if it is @code{t}, the command returns the encoded text as
1746 a unibyte string without inserting it.
1747
1748 If encoded text is inserted in some buffer, this command returns the
1749 length of the encoded text.
1750
1751 The result of encoding is logically a sequence of bytes, but the
1752 buffer remains multibyte if it was multibyte before, and any 8-bit
1753 bytes are converted to their multibyte representation (@pxref{Text
1754 Representations}).
1755
1756 @cindex @code{undecided} coding-system, when encoding
1757 Do @emph{not} use @code{undecided} for @var{coding-system} when
1758 encoding text, since that may lead to unexpected results. Instead,
1759 use @code{select-safe-coding-system} (@pxref{User-Chosen Coding
1760 Systems, select-safe-coding-system}) to suggest a suitable encoding,
1761 if there's no obvious pertinent value for @var{coding-system}.
1762 @end deffn
1763
1764 @defun encode-coding-string string coding-system &optional nocopy buffer
1765 This function encodes the text in @var{string} according to coding
1766 system @var{coding-system}. It returns a new string containing the
1767 encoded text, except when @var{nocopy} is non-@code{nil}, in which
1768 case the function may return @var{string} itself if the encoding
1769 operation is trivial. The result of encoding is a unibyte string.
1770 @end defun
1771
1772 @deffn Command decode-coding-region start end coding-system &optional destination
1773 This command decodes the text from @var{start} to @var{end} according
1774 to coding system @var{coding-system}. To make explicit decoding
1775 useful, the text before decoding ought to be a sequence of byte
1776 values, but both multibyte and unibyte buffers are acceptable (in the
1777 multibyte case, the raw byte values should be represented as eight-bit
1778 characters). Normally, the decoded text replaces the original text in
1779 the buffer, but the optional argument @var{destination} can change
1780 that. If @var{destination} is a buffer, the decoded text is inserted
1781 in that buffer after point (point does not move); if it is @code{t},
1782 the command returns the decoded text as a multibyte string without
1783 inserting it.
1784
1785 If decoded text is inserted in some buffer, this command returns the
1786 length of the decoded text.
1787
1788 This command puts a @code{charset} text property on the decoded text.
1789 The value of the property states the character set used to decode the
1790 original text.
1791 @end deffn
1792
1793 @defun decode-coding-string string coding-system &optional nocopy buffer
1794 This function decodes the text in @var{string} according to
1795 @var{coding-system}. It returns a new string containing the decoded
1796 text, except when @var{nocopy} is non-@code{nil}, in which case the
1797 function may return @var{string} itself if the decoding operation is
1798 trivial. To make explicit decoding useful, the contents of
1799 @var{string} ought to be a unibyte string with a sequence of byte
1800 values, but a multibyte string is also acceptable (assuming it
1801 contains 8-bit bytes in their multibyte form).
1802
1803 If optional argument @var{buffer} specifies a buffer, the decoded text
1804 is inserted in that buffer after point (point does not move). In this
1805 case, the return value is the length of the decoded text.
1806
1807 @cindex @code{charset}, text property
1808 This function puts a @code{charset} text property on the decoded text.
1809 The value of the property states the character set used to decode the
1810 original text:
1811
1812 @example
1813 @group
1814 (decode-coding-string "Gr\374ss Gott" 'latin-1)
1815 @result{} #("Gr@"uss Gott" 0 9 (charset iso-8859-1))
1816 @end group
1817 @end example
1818 @end defun
1819
1820 @defun decode-coding-inserted-region from to filename &optional visit beg end replace
1821 This function decodes the text from @var{from} to @var{to} as if
1822 it were being read from file @var{filename} using @code{insert-file-contents}
1823 using the rest of the arguments provided.
1824
1825 The normal way to use this function is after reading text from a file
1826 without decoding, if you decide you would rather have decoded it.
1827 Instead of deleting the text and reading it again, this time with
1828 decoding, you can call this function.
1829 @end defun
1830
1831 @node Terminal I/O Encoding
1832 @subsection Terminal I/O Encoding
1833
1834 Emacs can use coding systems to decode keyboard input and encode
1835 terminal output. This is useful for terminals that transmit or
1836 display text using a particular encoding, such as Latin-1. Emacs does
1837 not set @code{last-coding-system-used} when encoding or decoding
1838 terminal I/O.
1839
1840 @defun keyboard-coding-system &optional terminal
1841 This function returns the coding system used for decoding keyboard
1842 input from @var{terminal}. A value of @code{no-conversion} means no
1843 decoding is done. If @var{terminal} is omitted or @code{nil}, it
1844 means the selected frame's terminal. @xref{Multiple Terminals}.
1845 @end defun
1846
1847 @deffn Command set-keyboard-coding-system coding-system &optional terminal
1848 This command specifies @var{coding-system} as the coding system to use
1849 for decoding keyboard input from @var{terminal}. If
1850 @var{coding-system} is @code{nil}, that means not to decode keyboard
1851 input. If @var{terminal} is a frame, it means that frame's terminal;
1852 if it is @code{nil}, that means the currently selected frame's
1853 terminal. @xref{Multiple Terminals}.
1854 @end deffn
1855
1856 @defun terminal-coding-system &optional terminal
1857 This function returns the coding system that is in use for encoding
1858 terminal output from @var{terminal}. A value of @code{no-conversion}
1859 means no encoding is done. If @var{terminal} is a frame, it means
1860 that frame's terminal; if it is @code{nil}, that means the currently
1861 selected frame's terminal.
1862 @end defun
1863
1864 @deffn Command set-terminal-coding-system coding-system &optional terminal
1865 This command specifies @var{coding-system} as the coding system to use
1866 for encoding terminal output from @var{terminal}. If
1867 @var{coding-system} is @code{nil}, that means not to encode terminal
1868 output. If @var{terminal} is a frame, it means that frame's terminal;
1869 if it is @code{nil}, that means the currently selected frame's
1870 terminal.
1871 @end deffn
1872
1873 @node Input Methods
1874 @section Input Methods
1875 @cindex input methods
1876
1877 @dfn{Input methods} provide convenient ways of entering non-@acronym{ASCII}
1878 characters from the keyboard. Unlike coding systems, which translate
1879 non-@acronym{ASCII} characters to and from encodings meant to be read by
1880 programs, input methods provide human-friendly commands. (@xref{Input
1881 Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1882 use input methods to enter text.) How to define input methods is not
1883 yet documented in this manual, but here we describe how to use them.
1884
1885 Each input method has a name, which is currently a string;
1886 in the future, symbols may also be usable as input method names.
1887
1888 @defvar current-input-method
1889 This variable holds the name of the input method now active in the
1890 current buffer. (It automatically becomes local in each buffer when set
1891 in any fashion.) It is @code{nil} if no input method is active in the
1892 buffer now.
1893 @end defvar
1894
1895 @defopt default-input-method
1896 This variable holds the default input method for commands that choose an
1897 input method. Unlike @code{current-input-method}, this variable is
1898 normally global.
1899 @end defopt
1900
1901 @deffn Command set-input-method input-method
1902 This command activates input method @var{input-method} for the current
1903 buffer. It also sets @code{default-input-method} to @var{input-method}.
1904 If @var{input-method} is @code{nil}, this command deactivates any input
1905 method for the current buffer.
1906 @end deffn
1907
1908 @defun read-input-method-name prompt &optional default inhibit-null
1909 This function reads an input method name with the minibuffer, prompting
1910 with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1911 by default, if the user enters empty input. However, if
1912 @var{inhibit-null} is non-@code{nil}, empty input signals an error.
1913
1914 The returned value is a string.
1915 @end defun
1916
1917 @defvar input-method-alist
1918 This variable defines all the supported input methods.
1919 Each element defines one input method, and should have the form:
1920
1921 @example
1922 (@var{input-method} @var{language-env} @var{activate-func}
1923 @var{title} @var{description} @var{args}...)
1924 @end example
1925
1926 Here @var{input-method} is the input method name, a string;
1927 @var{language-env} is another string, the name of the language
1928 environment this input method is recommended for. (That serves only for
1929 documentation purposes.)
1930
1931 @var{activate-func} is a function to call to activate this method. The
1932 @var{args}, if any, are passed as arguments to @var{activate-func}. All
1933 told, the arguments to @var{activate-func} are @var{input-method} and
1934 the @var{args}.
1935
1936 @var{title} is a string to display in the mode line while this method is
1937 active. @var{description} is a string describing this method and what
1938 it is good for.
1939 @end defvar
1940
1941 The fundamental interface to input methods is through the
1942 variable @code{input-method-function}. @xref{Reading One Event},
1943 and @ref{Invoking the Input Method}.
1944
1945 @node Locales
1946 @section Locales
1947 @cindex locale
1948
1949 POSIX defines a concept of ``locales'' which control which language
1950 to use in language-related features. These Emacs variables control
1951 how Emacs interacts with these features.
1952
1953 @defvar locale-coding-system
1954 @cindex keyboard input decoding on X
1955 This variable specifies the coding system to use for decoding system
1956 error messages and---on X Window system only---keyboard input, for
1957 encoding the format argument to @code{format-time-string}, and for
1958 decoding the return value of @code{format-time-string}.
1959 @end defvar
1960
1961 @defvar system-messages-locale
1962 This variable specifies the locale to use for generating system error
1963 messages. Changing the locale can cause messages to come out in a
1964 different language or in a different orthography. If the variable is
1965 @code{nil}, the locale is specified by environment variables in the
1966 usual POSIX fashion.
1967 @end defvar
1968
1969 @defvar system-time-locale
1970 This variable specifies the locale to use for formatting time values.
1971 Changing the locale can cause messages to appear according to the
1972 conventions of a different language. If the variable is @code{nil}, the
1973 locale is specified by environment variables in the usual POSIX fashion.
1974 @end defvar
1975
1976 @defun locale-info item
1977 This function returns locale data @var{item} for the current POSIX
1978 locale, if available. @var{item} should be one of these symbols:
1979
1980 @table @code
1981 @item codeset
1982 Return the character set as a string (locale item @code{CODESET}).
1983
1984 @item days
1985 Return a 7-element vector of day names (locale items
1986 @code{DAY_1} through @code{DAY_7});
1987
1988 @item months
1989 Return a 12-element vector of month names (locale items @code{MON_1}
1990 through @code{MON_12}).
1991
1992 @item paper
1993 Return a list @code{(@var{width} @var{height})} for the default paper
1994 size measured in millimeters (locale items @code{PAPER_WIDTH} and
1995 @code{PAPER_HEIGHT}).
1996 @end table
1997
1998 If the system can't provide the requested information, or if
1999 @var{item} is not one of those symbols, the value is @code{nil}. All
2000 strings in the return value are decoded using
2001 @code{locale-coding-system}. @xref{Locales,,, libc, The GNU Libc Manual},
2002 for more information about locales and locale items.
2003 @end defun