]> code.delx.au - gnu-emacs/blob - lispref/nonascii.texi
Fix entry for sysdep.c.
[gnu-emacs] / lispref / nonascii.texi
1 @c -*-texinfo-*-
2 @c This is part of the GNU Emacs Lisp Reference Manual.
3 @c Copyright (C) 1998, 1999, 2002, 2003, 2004,
4 @c 2005 Free Software Foundation, Inc.
5 @c See the file elisp.texi for copying conditions.
6 @setfilename ../info/characters
7 @node Non-ASCII Characters, Searching and Matching, Text, Top
8 @chapter Non-@acronym{ASCII} Characters
9 @cindex multibyte characters
10 @cindex non-@acronym{ASCII} characters
11
12 This chapter covers the special issues relating to non-@acronym{ASCII}
13 characters and how they are stored in strings and buffers.
14
15 @menu
16 * Text Representations:: Unibyte and multibyte representations
17 * Converting Representations:: Converting unibyte to multibyte and vice versa.
18 * Selecting a Representation:: Treating a byte sequence as unibyte or multi.
19 * Character Codes:: How unibyte and multibyte relate to
20 codes of individual characters.
21 * Character Sets:: The space of possible character codes
22 is divided into various character sets.
23 * Chars and Bytes:: More information about multibyte encodings.
24 * Splitting Characters:: Converting a character to its byte sequence.
25 * Scanning Charsets:: Which character sets are used in a buffer?
26 * Translation of Characters:: Translation tables are used for conversion.
27 * Coding Systems:: Coding systems are conversions for saving files.
28 * Input Methods:: Input methods allow users to enter various
29 non-ASCII characters without special keyboards.
30 * Locales:: Interacting with the POSIX locale.
31 @end menu
32
33 @node Text Representations
34 @section Text Representations
35 @cindex text representations
36
37 Emacs has two @dfn{text representations}---two ways to represent text
38 in a string or buffer. These are called @dfn{unibyte} and
39 @dfn{multibyte}. Each string, and each buffer, uses one of these two
40 representations. For most purposes, you can ignore the issue of
41 representations, because Emacs converts text between them as
42 appropriate. Occasionally in Lisp programming you will need to pay
43 attention to the difference.
44
45 @cindex unibyte text
46 In unibyte representation, each character occupies one byte and
47 therefore the possible character codes range from 0 to 255. Codes 0
48 through 127 are @acronym{ASCII} characters; the codes from 128 through 255
49 are used for one non-@acronym{ASCII} character set (you can choose which
50 character set by setting the variable @code{nonascii-insert-offset}).
51
52 @cindex leading code
53 @cindex multibyte text
54 @cindex trailing codes
55 In multibyte representation, a character may occupy more than one
56 byte, and as a result, the full range of Emacs character codes can be
57 stored. The first byte of a multibyte character is always in the range
58 128 through 159 (octal 0200 through 0237). These values are called
59 @dfn{leading codes}. The second and subsequent bytes of a multibyte
60 character are always in the range 160 through 255 (octal 0240 through
61 0377); these values are @dfn{trailing codes}.
62
63 Some sequences of bytes are not valid in multibyte text: for example,
64 a single isolated byte in the range 128 through 159 is not allowed. But
65 character codes 128 through 159 can appear in multibyte text,
66 represented as two-byte sequences. All the character codes 128 through
67 255 are possible (though slightly abnormal) in multibyte text; they
68 appear in multibyte buffers and strings when you do explicit encoding
69 and decoding (@pxref{Explicit Encoding}).
70
71 In a buffer, the buffer-local value of the variable
72 @code{enable-multibyte-characters} specifies the representation used.
73 The representation for a string is determined and recorded in the string
74 when the string is constructed.
75
76 @defvar enable-multibyte-characters
77 This variable specifies the current buffer's text representation.
78 If it is non-@code{nil}, the buffer contains multibyte text; otherwise,
79 it contains unibyte text.
80
81 You cannot set this variable directly; instead, use the function
82 @code{set-buffer-multibyte} to change a buffer's representation.
83 @end defvar
84
85 @defvar default-enable-multibyte-characters
86 This variable's value is entirely equivalent to @code{(default-value
87 'enable-multibyte-characters)}, and setting this variable changes that
88 default value. Setting the local binding of
89 @code{enable-multibyte-characters} in a specific buffer is not allowed,
90 but changing the default value is supported, and it is a reasonable
91 thing to do, because it has no effect on existing buffers.
92
93 The @samp{--unibyte} command line option does its job by setting the
94 default value to @code{nil} early in startup.
95 @end defvar
96
97 @defun position-bytes position
98 @tindex position-bytes
99 Return the byte-position corresponding to buffer position
100 @var{position} in the current buffer. This is 1 at the start of the
101 buffer, and counts upward in bytes. If @var{position} is out of
102 range, the value is @code{nil}.
103 @end defun
104
105 @defun byte-to-position byte-position
106 @tindex byte-to-position
107 Return the buffer position corresponding to byte-position
108 @var{byte-position} in the current buffer. If @var{byte-position} is
109 out of range, the value is @code{nil}.
110 @end defun
111
112 @defun multibyte-string-p string
113 Return @code{t} if @var{string} is a multibyte string.
114 @end defun
115
116 @node Converting Representations
117 @section Converting Text Representations
118
119 Emacs can convert unibyte text to multibyte; it can also convert
120 multibyte text to unibyte, though this conversion loses information. In
121 general these conversions happen when inserting text into a buffer, or
122 when putting text from several strings together in one string. You can
123 also explicitly convert a string's contents to either representation.
124
125 Emacs chooses the representation for a string based on the text that
126 it is constructed from. The general rule is to convert unibyte text to
127 multibyte text when combining it with other multibyte text, because the
128 multibyte representation is more general and can hold whatever
129 characters the unibyte text has.
130
131 When inserting text into a buffer, Emacs converts the text to the
132 buffer's representation, as specified by
133 @code{enable-multibyte-characters} in that buffer. In particular, when
134 you insert multibyte text into a unibyte buffer, Emacs converts the text
135 to unibyte, even though this conversion cannot in general preserve all
136 the characters that might be in the multibyte text. The other natural
137 alternative, to convert the buffer contents to multibyte, is not
138 acceptable because the buffer's representation is a choice made by the
139 user that cannot be overridden automatically.
140
141 Converting unibyte text to multibyte text leaves @acronym{ASCII} characters
142 unchanged, and likewise character codes 128 through 159. It converts
143 the non-@acronym{ASCII} codes 160 through 255 by adding the value
144 @code{nonascii-insert-offset} to each character code. By setting this
145 variable, you specify which character set the unibyte characters
146 correspond to (@pxref{Character Sets}). For example, if
147 @code{nonascii-insert-offset} is 2048, which is @code{(- (make-char
148 'latin-iso8859-1) 128)}, then the unibyte non-@acronym{ASCII} characters
149 correspond to Latin 1. If it is 2688, which is @code{(- (make-char
150 'greek-iso8859-7) 128)}, then they correspond to Greek letters.
151
152 Converting multibyte text to unibyte is simpler: it discards all but
153 the low 8 bits of each character code. If @code{nonascii-insert-offset}
154 has a reasonable value, corresponding to the beginning of some character
155 set, this conversion is the inverse of the other: converting unibyte
156 text to multibyte and back to unibyte reproduces the original unibyte
157 text.
158
159 @defvar nonascii-insert-offset
160 This variable specifies the amount to add to a non-@acronym{ASCII} character
161 when converting unibyte text to multibyte. It also applies when
162 @code{self-insert-command} inserts a character in the unibyte
163 non-@acronym{ASCII} range, 128 through 255. However, the functions
164 @code{insert} and @code{insert-char} do not perform this conversion.
165
166 The right value to use to select character set @var{cs} is @code{(-
167 (make-char @var{cs}) 128)}. If the value of
168 @code{nonascii-insert-offset} is zero, then conversion actually uses the
169 value for the Latin 1 character set, rather than zero.
170 @end defvar
171
172 @defvar nonascii-translation-table
173 This variable provides a more general alternative to
174 @code{nonascii-insert-offset}. You can use it to specify independently
175 how to translate each code in the range of 128 through 255 into a
176 multibyte character. The value should be a char-table, or @code{nil}.
177 If this is non-@code{nil}, it overrides @code{nonascii-insert-offset}.
178 @end defvar
179
180 The next three functions either return the argument @var{string}, or a
181 newly created string with no text properties.
182
183 @defun string-make-unibyte string
184 This function converts the text of @var{string} to unibyte
185 representation, if it isn't already, and returns the result. If
186 @var{string} is a unibyte string, it is returned unchanged. Multibyte
187 character codes are converted to unibyte according to
188 @code{nonascii-translation-table} or, if that is @code{nil}, using
189 @code{nonascii-insert-offset}. If the lookup in the translation table
190 fails, this function takes just the low 8 bits of each character.
191 @end defun
192
193 @defun string-make-multibyte string
194 This function converts the text of @var{string} to multibyte
195 representation, if it isn't already, and returns the result. If
196 @var{string} is a multibyte string or consists entirely of
197 @acronym{ASCII} characters, it is returned unchanged. In particular,
198 if @var{string} is unibyte and entirely @acronym{ASCII}, the returned
199 string is unibyte. (When the characters are all @acronym{ASCII},
200 Emacs primitives will treat the string the same way whether it is
201 unibyte or multibyte.) If @var{string} is unibyte and contains
202 non-@acronym{ASCII} characters, the function
203 @code{unibyte-char-to-multibyte} is used to convert each unibyte
204 character to a multibyte character.
205 @end defun
206
207 @defun string-to-multibyte string
208 This function returns a multibyte string containing the same sequence
209 of character codes as @var{string}. Unlike
210 @code{string-make-multibyte}, this function unconditionally returns a
211 multibyte string. If @var{string} is a multibyte string, it is
212 returned unchanged.
213 @end defun
214
215 @defun multibyte-char-to-unibyte char
216 This convert the multibyte character @var{char} to a unibyte
217 character, based on @code{nonascii-translation-table} and
218 @code{nonascii-insert-offset}.
219 @end defun
220
221 @defun unibyte-char-to-multibyte char
222 This convert the unibyte character @var{char} to a multibyte
223 character, based on @code{nonascii-translation-table} and
224 @code{nonascii-insert-offset}.
225 @end defun
226
227 @node Selecting a Representation
228 @section Selecting a Representation
229
230 Sometimes it is useful to examine an existing buffer or string as
231 multibyte when it was unibyte, or vice versa.
232
233 @defun set-buffer-multibyte multibyte
234 Set the representation type of the current buffer. If @var{multibyte}
235 is non-@code{nil}, the buffer becomes multibyte. If @var{multibyte}
236 is @code{nil}, the buffer becomes unibyte.
237
238 This function leaves the buffer contents unchanged when viewed as a
239 sequence of bytes. As a consequence, it can change the contents viewed
240 as characters; a sequence of two bytes which is treated as one character
241 in multibyte representation will count as two characters in unibyte
242 representation. Character codes 128 through 159 are an exception. They
243 are represented by one byte in a unibyte buffer, but when the buffer is
244 set to multibyte, they are converted to two-byte sequences, and vice
245 versa.
246
247 This function sets @code{enable-multibyte-characters} to record which
248 representation is in use. It also adjusts various data in the buffer
249 (including overlays, text properties and markers) so that they cover the
250 same text as they did before.
251
252 You cannot use @code{set-buffer-multibyte} on an indirect buffer,
253 because indirect buffers always inherit the representation of the
254 base buffer.
255 @end defun
256
257 @defun string-as-unibyte string
258 This function returns a string with the same bytes as @var{string} but
259 treating each byte as a character. This means that the value may have
260 more characters than @var{string} has.
261
262 If @var{string} is already a unibyte string, then the value is
263 @var{string} itself. Otherwise it is a newly created string, with no
264 text properties. If @var{string} is multibyte, any characters it
265 contains of charset @code{eight-bit-control} or @code{eight-bit-graphic}
266 are converted to the corresponding single byte.
267 @end defun
268
269 @defun string-as-multibyte string
270 This function returns a string with the same bytes as @var{string} but
271 treating each multibyte sequence as one character. This means that the
272 value may have fewer characters than @var{string} has.
273
274 If @var{string} is already a multibyte string, then the value is
275 @var{string} itself. Otherwise it is a newly created string, with no
276 text properties. If @var{string} is unibyte and contains any individual
277 8-bit bytes (i.e.@: not part of a multibyte form), they are converted to
278 the corresponding multibyte character of charset @code{eight-bit-control}
279 or @code{eight-bit-graphic}.
280 @end defun
281
282 @node Character Codes
283 @section Character Codes
284 @cindex character codes
285
286 The unibyte and multibyte text representations use different character
287 codes. The valid character codes for unibyte representation range from
288 0 to 255---the values that can fit in one byte. The valid character
289 codes for multibyte representation range from 0 to 524287, but not all
290 values in that range are valid. The values 128 through 255 are not
291 entirely proper in multibyte text, but they can occur if you do explicit
292 encoding and decoding (@pxref{Explicit Encoding}). Some other character
293 codes cannot occur at all in multibyte text. Only the @acronym{ASCII} codes
294 0 through 127 are completely legitimate in both representations.
295
296 @defun char-valid-p charcode &optional genericp
297 This returns @code{t} if @var{charcode} is valid (either for unibyte
298 text or for multibyte text).
299
300 @example
301 (char-valid-p 65)
302 @result{} t
303 (char-valid-p 256)
304 @result{} nil
305 (char-valid-p 2248)
306 @result{} t
307 @end example
308
309 If the optional argument @var{genericp} is non-@code{nil}, this
310 function also returns @code{t} if @var{charcode} is a generic
311 character (@pxref{Splitting Characters}).
312 @end defun
313
314 @node Character Sets
315 @section Character Sets
316 @cindex character sets
317
318 Emacs classifies characters into various @dfn{character sets}, each of
319 which has a name which is a symbol. Each character belongs to one and
320 only one character set.
321
322 In general, there is one character set for each distinct script. For
323 example, @code{latin-iso8859-1} is one character set,
324 @code{greek-iso8859-7} is another, and @code{ascii} is another. An
325 Emacs character set can hold at most 9025 characters; therefore, in some
326 cases, characters that would logically be grouped together are split
327 into several character sets. For example, one set of Chinese
328 characters, generally known as Big 5, is divided into two Emacs
329 character sets, @code{chinese-big5-1} and @code{chinese-big5-2}.
330
331 @acronym{ASCII} characters are in character set @code{ascii}. The
332 non-@acronym{ASCII} characters 128 through 159 are in character set
333 @code{eight-bit-control}, and codes 160 through 255 are in character set
334 @code{eight-bit-graphic}.
335
336 @defun charsetp object
337 Returns @code{t} if @var{object} is a symbol that names a character set,
338 @code{nil} otherwise.
339 @end defun
340
341 @defvar charset-list
342 The value is a list of all defined character set names.
343 @end defvar
344
345 @defun charset-list
346 This function returns the value of @code{charset-list}. It is only
347 provided for backward compatibility.
348 @end defun
349
350 @defun char-charset character
351 This function returns the name of the character set that @var{character}
352 belongs to, or the symbol @code{unknown} if @var{character} is not a
353 valid character.
354 @end defun
355
356 @defun charset-plist charset
357 @tindex charset-plist
358 This function returns the charset property list of the character set
359 @var{charset}. Although @var{charset} is a symbol, this is not the same
360 as the property list of that symbol. Charset properties are used for
361 special purposes within Emacs.
362 @end defun
363
364 @deffn Command list-charset-chars charset
365 This command displays a list of characters in the character set
366 @var{charset}.
367 @end deffn
368
369 @node Chars and Bytes
370 @section Characters and Bytes
371 @cindex bytes and characters
372
373 @cindex introduction sequence
374 @cindex dimension (of character set)
375 In multibyte representation, each character occupies one or more
376 bytes. Each character set has an @dfn{introduction sequence}, which is
377 normally one or two bytes long. (Exception: the @code{ascii} character
378 set and the @code{eight-bit-graphic} character set have a zero-length
379 introduction sequence.) The introduction sequence is the beginning of
380 the byte sequence for any character in the character set. The rest of
381 the character's bytes distinguish it from the other characters in the
382 same character set. Depending on the character set, there are either
383 one or two distinguishing bytes; the number of such bytes is called the
384 @dfn{dimension} of the character set.
385
386 @defun charset-dimension charset
387 This function returns the dimension of @var{charset}; at present, the
388 dimension is always 1 or 2.
389 @end defun
390
391 @defun charset-bytes charset
392 @tindex charset-bytes
393 This function returns the number of bytes used to represent a character
394 in character set @var{charset}.
395 @end defun
396
397 This is the simplest way to determine the byte length of a character
398 set's introduction sequence:
399
400 @example
401 (- (charset-bytes @var{charset})
402 (charset-dimension @var{charset}))
403 @end example
404
405 @node Splitting Characters
406 @section Splitting Characters
407
408 The functions in this section convert between characters and the byte
409 values used to represent them. For most purposes, there is no need to
410 be concerned with the sequence of bytes used to represent a character,
411 because Emacs translates automatically when necessary.
412
413 @defun split-char character
414 Return a list containing the name of the character set of
415 @var{character}, followed by one or two byte values (integers) which
416 identify @var{character} within that character set. The number of byte
417 values is the character set's dimension.
418
419 If @var{character} is invalid as a character code, @code{split-char}
420 returns a list consisting of the symbol @code{unknown} and @var{character}.
421
422 @example
423 (split-char 2248)
424 @result{} (latin-iso8859-1 72)
425 (split-char 65)
426 @result{} (ascii 65)
427 (split-char 128)
428 @result{} (eight-bit-control 128)
429 @end example
430 @end defun
431
432 @defun make-char charset &optional code1 code2
433 This function returns the character in character set @var{charset} whose
434 position codes are @var{code1} and @var{code2}. This is roughly the
435 inverse of @code{split-char}. Normally, you should specify either one
436 or both of @var{code1} and @var{code2} according to the dimension of
437 @var{charset}. For example,
438
439 @example
440 (make-char 'latin-iso8859-1 72)
441 @result{} 2248
442 @end example
443
444 Actually, the eighth bit of both @var{code1} and @var{code2} is zeroed
445 before they are used to index @var{charset}. Thus you may use, for
446 instance, an ISO 8859 character code rather than subtracting 128, as
447 is necessary to index the corresponding Emacs charset.
448 @end defun
449
450 @cindex generic characters
451 If you call @code{make-char} with no @var{byte-values}, the result is
452 a @dfn{generic character} which stands for @var{charset}. A generic
453 character is an integer, but it is @emph{not} valid for insertion in the
454 buffer as a character. It can be used in @code{char-table-range} to
455 refer to the whole character set (@pxref{Char-Tables}).
456 @code{char-valid-p} returns @code{nil} for generic characters.
457 For example:
458
459 @example
460 (make-char 'latin-iso8859-1)
461 @result{} 2176
462 (char-valid-p 2176)
463 @result{} nil
464 (char-valid-p 2176 t)
465 @result{} t
466 (split-char 2176)
467 @result{} (latin-iso8859-1 0)
468 @end example
469
470 The character sets @code{ascii}, @code{eight-bit-control}, and
471 @code{eight-bit-graphic} don't have corresponding generic characters. If
472 @var{charset} is one of them and you don't supply @var{code1},
473 @code{make-char} returns the character code corresponding to the
474 smallest code in @var{charset}.
475
476 @node Scanning Charsets
477 @section Scanning for Character Sets
478
479 Sometimes it is useful to find out which character sets appear in a
480 part of a buffer or a string. One use for this is in determining which
481 coding systems (@pxref{Coding Systems}) are capable of representing all
482 of the text in question.
483
484 @defun charset-after &optional pos
485 This function return the charset of a character in the current buffer
486 at position @var{pos}. If @var{pos} is omitted or @code{nil}, it
487 defauls to the current value of point. If @var{pos} is out of range,
488 the value is @code{nil}.
489 @end defun
490
491 @defun find-charset-region beg end &optional translation
492 This function returns a list of the character sets that appear in the
493 current buffer between positions @var{beg} and @var{end}.
494
495 The optional argument @var{translation} specifies a translation table to
496 be used in scanning the text (@pxref{Translation of Characters}). If it
497 is non-@code{nil}, then each character in the region is translated
498 through this table, and the value returned describes the translated
499 characters instead of the characters actually in the buffer.
500 @end defun
501
502 @defun find-charset-string string &optional translation
503 This function returns a list of the character sets that appear in the
504 string @var{string}. It is just like @code{find-charset-region}, except
505 that it applies to the contents of @var{string} instead of part of the
506 current buffer.
507 @end defun
508
509 @node Translation of Characters
510 @section Translation of Characters
511 @cindex character translation tables
512 @cindex translation tables
513
514 A @dfn{translation table} is a char-table that specifies a mapping
515 of characters into characters. These tables are used in encoding and
516 decoding, and for other purposes. Some coding systems specify their
517 own particular translation tables; there are also default translation
518 tables which apply to all other coding systems.
519
520 For instance, the coding-system @code{utf-8} has a translation table
521 that maps characters of various charsets (e.g.,
522 @code{latin-iso8859-@var{x}}) into Unicode character sets. This way,
523 it can encode Latin-2 characters into UTF-8. Meanwhile,
524 @code{unify-8859-on-decoding-mode} operates by specifying
525 @code{standard-translation-table-for-decode} to translate
526 Latin-@var{x} characters into corresponding Unicode characters.
527
528 @defun make-translation-table &rest translations
529 This function returns a translation table based on the argument
530 @var{translations}. Each element of @var{translations} should be a
531 list of elements of the form @code{(@var{from} . @var{to})}; this says
532 to translate the character @var{from} into @var{to}.
533
534 The arguments and the forms in each argument are processed in order,
535 and if a previous form already translates @var{to} to some other
536 character, say @var{to-alt}, @var{from} is also translated to
537 @var{to-alt}.
538
539 You can also map one whole character set into another character set with
540 the same dimension. To do this, you specify a generic character (which
541 designates a character set) for @var{from} (@pxref{Splitting Characters}).
542 In this case, if @var{to} is also a generic character, its character
543 set should have the same dimension as @var{from}'s. Then the
544 translation table translates each character of @var{from}'s character
545 set into the corresponding character of @var{to}'s character set. If
546 @var{from} is a generic character and @var{to} is an ordinary
547 character, then the translation table translates every character of
548 @var{from}'s character set into @var{to}.
549 @end defun
550
551 In decoding, the translation table's translations are applied to the
552 characters that result from ordinary decoding. If a coding system has
553 property @code{translation-table-for-decode}, that specifies the
554 translation table to use. (This is a property of the coding system,
555 as returned by @code{coding-system-get}, not a property of the symbol
556 that is the coding system's name. @xref{Coding System Basics,, Basic
557 Concepts of Coding Systems}.) Otherwise, if
558 @code{standard-translation-table-for-decode} is non-@code{nil},
559 decoding uses that table.
560
561 In encoding, the translation table's translations are applied to the
562 characters in the buffer, and the result of translation is actually
563 encoded. If a coding system has property
564 @code{translation-table-for-encode}, that specifies the translation
565 table to use. Otherwise the variable
566 @code{standard-translation-table-for-encode} specifies the translation
567 table.
568
569 @defvar standard-translation-table-for-decode
570 This is the default translation table for decoding, for
571 coding systems that don't specify any other translation table.
572 @end defvar
573
574 @defvar standard-translation-table-for-encode
575 This is the default translation table for encoding, for
576 coding systems that don't specify any other translation table.
577 @end defvar
578
579 @defvar translation-table-for-input
580 Self-inserting characters are translated through this translation
581 table before they are inserted. This variable automatically becomes
582 buffer-local when set.
583
584 @code{set-buffer-file-coding-system} sets this variable so that your
585 keyboard input gets translated into the character sets that the buffer
586 is likely to contain.
587 @end defvar
588
589 @node Coding Systems
590 @section Coding Systems
591
592 @cindex coding system
593 When Emacs reads or writes a file, and when Emacs sends text to a
594 subprocess or receives text from a subprocess, it normally performs
595 character code conversion and end-of-line conversion as specified
596 by a particular @dfn{coding system}.
597
598 How to define a coding system is an arcane matter, and is not
599 documented here.
600
601 @menu
602 * Coding System Basics:: Basic concepts.
603 * Encoding and I/O:: How file I/O functions handle coding systems.
604 * Lisp and Coding Systems:: Functions to operate on coding system names.
605 * User-Chosen Coding Systems:: Asking the user to choose a coding system.
606 * Default Coding Systems:: Controlling the default choices.
607 * Specifying Coding Systems:: Requesting a particular coding system
608 for a single file operation.
609 * Explicit Encoding:: Encoding or decoding text without doing I/O.
610 * Terminal I/O Encoding:: Use of encoding for terminal I/O.
611 * MS-DOS File Types:: How DOS "text" and "binary" files
612 relate to coding systems.
613 @end menu
614
615 @node Coding System Basics
616 @subsection Basic Concepts of Coding Systems
617
618 @cindex character code conversion
619 @dfn{Character code conversion} involves conversion between the encoding
620 used inside Emacs and some other encoding. Emacs supports many
621 different encodings, in that it can convert to and from them. For
622 example, it can convert text to or from encodings such as Latin 1, Latin
623 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022. In some
624 cases, Emacs supports several alternative encodings for the same
625 characters; for example, there are three coding systems for the Cyrillic
626 (Russian) alphabet: ISO, Alternativnyj, and KOI8.
627
628 Most coding systems specify a particular character code for
629 conversion, but some of them leave the choice unspecified---to be chosen
630 heuristically for each file, based on the data.
631
632 In general, a coding system doesn't guarantee roundtrip identity:
633 decoding a byte sequence using coding system, then encoding the
634 resulting text in the same coding system, can produce a different byte
635 sequence. However, the following coding systems do guarantee that the
636 byte sequence will be the same as what you originally decoded:
637
638 @quotation
639 chinese-big5 chinese-iso-8bit cyrillic-iso-8bit emacs-mule
640 greek-iso-8bit hebrew-iso-8bit iso-latin-1 iso-latin-2 iso-latin-3
641 iso-latin-4 iso-latin-5 iso-latin-8 iso-latin-9 iso-safe
642 japanese-iso-8bit japanese-shift-jis korean-iso-8bit raw-text
643 @end quotation
644
645 Encoding buffer text and then decoding the result can also fail to
646 reproduce the original text. For instance, if you encode Latin-2
647 characters with @code{utf-8} and decode the result using the same
648 coding system, you'll get Unicode characters (of charset
649 @code{mule-unicode-0100-24ff}). If you encode Unicode characters with
650 @code{iso-latin-2} and decode the result with the same coding system,
651 you'll get Latin-2 characters.
652
653 @cindex end of line conversion
654 @dfn{End of line conversion} handles three different conventions used
655 on various systems for representing end of line in files. The Unix
656 convention is to use the linefeed character (also called newline). The
657 DOS convention is to use a carriage-return and a linefeed at the end of
658 a line. The Mac convention is to use just carriage-return.
659
660 @cindex base coding system
661 @cindex variant coding system
662 @dfn{Base coding systems} such as @code{latin-1} leave the end-of-line
663 conversion unspecified, to be chosen based on the data. @dfn{Variant
664 coding systems} such as @code{latin-1-unix}, @code{latin-1-dos} and
665 @code{latin-1-mac} specify the end-of-line conversion explicitly as
666 well. Most base coding systems have three corresponding variants whose
667 names are formed by adding @samp{-unix}, @samp{-dos} and @samp{-mac}.
668
669 The coding system @code{raw-text} is special in that it prevents
670 character code conversion, and causes the buffer visited with that
671 coding system to be a unibyte buffer. It does not specify the
672 end-of-line conversion, allowing that to be determined as usual by the
673 data, and has the usual three variants which specify the end-of-line
674 conversion. @code{no-conversion} is equivalent to @code{raw-text-unix}:
675 it specifies no conversion of either character codes or end-of-line.
676
677 The coding system @code{emacs-mule} specifies that the data is
678 represented in the internal Emacs encoding. This is like
679 @code{raw-text} in that no code conversion happens, but different in
680 that the result is multibyte data.
681
682 @defun coding-system-get coding-system property
683 This function returns the specified property of the coding system
684 @var{coding-system}. Most coding system properties exist for internal
685 purposes, but one that you might find useful is @code{mime-charset}.
686 That property's value is the name used in MIME for the character coding
687 which this coding system can read and write. Examples:
688
689 @example
690 (coding-system-get 'iso-latin-1 'mime-charset)
691 @result{} iso-8859-1
692 (coding-system-get 'iso-2022-cn 'mime-charset)
693 @result{} iso-2022-cn
694 (coding-system-get 'cyrillic-koi8 'mime-charset)
695 @result{} koi8-r
696 @end example
697
698 The value of the @code{mime-charset} property is also defined
699 as an alias for the coding system.
700 @end defun
701
702 @node Encoding and I/O
703 @subsection Encoding and I/O
704
705 The principal purpose of coding systems is for use in reading and
706 writing files. The function @code{insert-file-contents} uses
707 a coding system for decoding the file data, and @code{write-region}
708 uses one to encode the buffer contents.
709
710 You can specify the coding system to use either explicitly
711 (@pxref{Specifying Coding Systems}), or implicitly using a default
712 mechanism (@pxref{Default Coding Systems}). But these methods may not
713 completely specify what to do. For example, they may choose a coding
714 system such as @code{undefined} which leaves the character code
715 conversion to be determined from the data. In these cases, the I/O
716 operation finishes the job of choosing a coding system. Very often
717 you will want to find out afterwards which coding system was chosen.
718
719 @defvar buffer-file-coding-system
720 This buffer-local variable records the coding system that was used to visit
721 the current buffer. It is used for saving the buffer, and for writing part
722 of the buffer with @code{write-region}. If the text to be written
723 cannot be safely encoded using the coding system specified by this
724 variable, these operations select an alternative encoding by calling
725 the function @code{select-safe-coding-system} (@pxref{User-Chosen
726 Coding Systems}). If selecting a different encoding requires to ask
727 the user to specify a coding system, @code{buffer-file-coding-system}
728 is updated to the newly selected coding system.
729
730 @code{buffer-file-coding-system} does @emph{not} affect sending text
731 to a subprocess.
732 @end defvar
733
734 @defvar save-buffer-coding-system
735 This variable specifies the coding system for saving the buffer (by
736 overriding @code{buffer-file-coding-system}). Note that it is not used
737 for @code{write-region}.
738
739 When a command to save the buffer starts out to use
740 @code{buffer-file-coding-system} (or @code{save-buffer-coding-system}),
741 and that coding system cannot handle
742 the actual text in the buffer, the command asks the user to choose
743 another coding system (by calling @code{select-safe-coding-system}).
744 After that happens, the command also updates
745 @code{buffer-file-coding-system} to represent the coding system that
746 the user specified.
747 @end defvar
748
749 @defvar last-coding-system-used
750 I/O operations for files and subprocesses set this variable to the
751 coding system name that was used. The explicit encoding and decoding
752 functions (@pxref{Explicit Encoding}) set it too.
753
754 @strong{Warning:} Since receiving subprocess output sets this variable,
755 it can change whenever Emacs waits; therefore, you should copy the
756 value shortly after the function call that stores the value you are
757 interested in.
758 @end defvar
759
760 The variable @code{selection-coding-system} specifies how to encode
761 selections for the window system. @xref{Window System Selections}.
762
763 @defvar file-name-coding-system
764 The variable @code{file-name-coding-system} specifies the coding
765 system to use for encoding file names. Emacs encodes file names using
766 that coding system for all file operations. If
767 @code{file-name-coding-system} is @code{nil}, Emacs uses a default
768 coding system determined by the selected language environment. In the
769 default language environment, any non-@acronym{ASCII} characters in
770 file names are not encoded specially; they appear in the file system
771 using the internal Emacs representation.
772 @end defvar
773
774 @strong{Warning:} if you change @code{file-name-coding-system} (or
775 the language environment) in the middle of an Emacs session, problems
776 can result if you have already visited files whose names were encoded
777 using the earlier coding system and are handled differently under the
778 new coding system. If you try to save one of these buffers under the
779 visited file name, saving may use the wrong file name, or it may get
780 an error. If such a problem happens, use @kbd{C-x C-w} to specify a
781 new file name for that buffer.
782
783 @node Lisp and Coding Systems
784 @subsection Coding Systems in Lisp
785
786 Here are the Lisp facilities for working with coding systems:
787
788 @defun coding-system-list &optional base-only
789 This function returns a list of all coding system names (symbols). If
790 @var{base-only} is non-@code{nil}, the value includes only the
791 base coding systems. Otherwise, it includes alias and variant coding
792 systems as well.
793 @end defun
794
795 @defun coding-system-p object
796 This function returns @code{t} if @var{object} is a coding system
797 name or @code{nil}.
798 @end defun
799
800 @defun check-coding-system coding-system
801 This function checks the validity of @var{coding-system}.
802 If that is valid, it returns @var{coding-system}.
803 Otherwise it signals an error with condition @code{coding-system-error}.
804 @end defun
805
806 @defun coding-system-change-eol-conversion coding-system eol-type
807 This function returns a coding system which is like @var{coding-system}
808 except for its eol conversion, which is specified by @code{eol-type}.
809 @var{eol-type} should be @code{unix}, @code{dos}, @code{mac}, or
810 @code{nil}. If it is @code{nil}, the returned coding system determines
811 the end-of-line conversion from the data.
812
813 @var{eol-type} may also be 0, 1 or 2, standing for @code{unix},
814 @code{dos} and @code{mac}, respectively.
815 @end defun
816
817 @defun coding-system-change-text-conversion eol-coding text-coding
818 This function returns a coding system which uses the end-of-line
819 conversion of @var{eol-coding}, and the text conversion of
820 @var{text-coding}. If @var{text-coding} is @code{nil}, it returns
821 @code{undecided}, or one of its variants according to @var{eol-coding}.
822 @end defun
823
824 @defun find-coding-systems-region from to
825 This function returns a list of coding systems that could be used to
826 encode a text between @var{from} and @var{to}. All coding systems in
827 the list can safely encode any multibyte characters in that portion of
828 the text.
829
830 If the text contains no multibyte characters, the function returns the
831 list @code{(undecided)}.
832 @end defun
833
834 @defun find-coding-systems-string string
835 This function returns a list of coding systems that could be used to
836 encode the text of @var{string}. All coding systems in the list can
837 safely encode any multibyte characters in @var{string}. If the text
838 contains no multibyte characters, this returns the list
839 @code{(undecided)}.
840 @end defun
841
842 @defun find-coding-systems-for-charsets charsets
843 This function returns a list of coding systems that could be used to
844 encode all the character sets in the list @var{charsets}.
845 @end defun
846
847 @defun detect-coding-region start end &optional highest
848 This function chooses a plausible coding system for decoding the text
849 from @var{start} to @var{end}. This text should be a byte sequence
850 (@pxref{Explicit Encoding}).
851
852 Normally this function returns a list of coding systems that could
853 handle decoding the text that was scanned. They are listed in order of
854 decreasing priority. But if @var{highest} is non-@code{nil}, then the
855 return value is just one coding system, the one that is highest in
856 priority.
857
858 If the region contains only @acronym{ASCII} characters, the value
859 is @code{undecided} or @code{(undecided)}, or a variant specifying
860 end-of-line conversion, if that can be deduced from the text.
861 @end defun
862
863 @defun detect-coding-string string &optional highest
864 This function is like @code{detect-coding-region} except that it
865 operates on the contents of @var{string} instead of bytes in the buffer.
866 @end defun
867
868 @xref{Coding systems for a subprocess,, Process Information}, in
869 particular the description of the functions
870 @code{process-coding-system} and @code{set-process-coding-system}, for
871 how to examine or set the coding systems used for I/O to a subprocess.
872
873 @node User-Chosen Coding Systems
874 @subsection User-Chosen Coding Systems
875
876 @cindex select safe coding system
877 @defun select-safe-coding-system from to &optional default-coding-system accept-default-p file
878 This function selects a coding system for encoding specified text,
879 asking the user to choose if necessary. Normally the specified text
880 is the text in the current buffer between @var{from} and @var{to}. If
881 @var{from} is a string, the string specifies the text to encode, and
882 @var{to} is ignored.
883
884 If @var{default-coding-system} is non-@code{nil}, that is the first
885 coding system to try; if that can handle the text,
886 @code{select-safe-coding-system} returns that coding system. It can
887 also be a list of coding systems; then the function tries each of them
888 one by one. After trying all of them, it next tries the current
889 buffer's value of @code{buffer-file-coding-system} (if it is not
890 @code{undecided}), then the value of
891 @code{default-buffer-file-coding-system} and finally the user's most
892 preferred coding system, which the user can set using the command
893 @code{prefer-coding-system} (@pxref{Recognize Coding,, Recognizing
894 Coding Systems, emacs, The GNU Emacs Manual}).
895
896 If one of those coding systems can safely encode all the specified
897 text, @code{select-safe-coding-system} chooses it and returns it.
898 Otherwise, it asks the user to choose from a list of coding systems
899 which can encode all the text, and returns the user's choice.
900
901 @var{default-coding-system} can also be a list whose first element is
902 t and whose other elements are coding systems. Then, if no coding
903 system in the list can handle the text, @code{select-safe-coding-system}
904 queries the user immediately, without trying any of the three
905 alternatives described above.
906
907 The optional argument @var{accept-default-p}, if non-@code{nil},
908 should be a function to determine whether a coding system selected
909 without user interaction is acceptable. @code{select-safe-coding-system}
910 calls this function with one argument, the base coding system of the
911 selected coding system. If @var{accept-default-p} returns @code{nil},
912 @code{select-safe-coding-system} rejects the silently selected coding
913 system, and asks the user to select a coding system from a list of
914 possible candidates.
915
916 @vindex select-safe-coding-system-accept-default-p
917 If the variable @code{select-safe-coding-system-accept-default-p} is
918 non-@code{nil}, its value overrides the value of
919 @var{accept-default-p}.
920
921 As a final step, before returning the chosen coding system,
922 @code{select-safe-coding-system} checks whether that coding system is
923 consistent with what would be selected if the contents of the region
924 were read from a file. (If not, this could lead to data corruption in
925 a file subsequently re-visited and edited.) Normally,
926 @code{select-safe-coding-system} uses @code{buffer-file-name} as the
927 file for this purpose, but if @var{file} is non-@code{nil}, it uses
928 that file instead (this can be relevant for @code{write-region} and
929 similar functions). If it detects an apparent inconsistency,
930 @code{select-safe-coding-system} queries the user before selecting the
931 coding system.
932 @end defun
933
934 Here are two functions you can use to let the user specify a coding
935 system, with completion. @xref{Completion}.
936
937 @defun read-coding-system prompt &optional default
938 This function reads a coding system using the minibuffer, prompting with
939 string @var{prompt}, and returns the coding system name as a symbol. If
940 the user enters null input, @var{default} specifies which coding system
941 to return. It should be a symbol or a string.
942 @end defun
943
944 @defun read-non-nil-coding-system prompt
945 This function reads a coding system using the minibuffer, prompting with
946 string @var{prompt}, and returns the coding system name as a symbol. If
947 the user tries to enter null input, it asks the user to try again.
948 @xref{Coding Systems}.
949 @end defun
950
951 @node Default Coding Systems
952 @subsection Default Coding Systems
953
954 This section describes variables that specify the default coding
955 system for certain files or when running certain subprograms, and the
956 function that I/O operations use to access them.
957
958 The idea of these variables is that you set them once and for all to the
959 defaults you want, and then do not change them again. To specify a
960 particular coding system for a particular operation in a Lisp program,
961 don't change these variables; instead, override them using
962 @code{coding-system-for-read} and @code{coding-system-for-write}
963 (@pxref{Specifying Coding Systems}).
964
965 @defvar auto-coding-regexp-alist
966 This variable is an alist of text patterns and corresponding coding
967 systems. Each element has the form @code{(@var{regexp}
968 . @var{coding-system})}; a file whose first few kilobytes match
969 @var{regexp} is decoded with @var{coding-system} when its contents are
970 read into a buffer. The settings in this alist take priority over
971 @code{coding:} tags in the files and the contents of
972 @code{file-coding-system-alist} (see below). The default value is set
973 so that Emacs automatically recognizes mail files in Babyl format and
974 reads them with no code conversions.
975 @end defvar
976
977 @defvar file-coding-system-alist
978 This variable is an alist that specifies the coding systems to use for
979 reading and writing particular files. Each element has the form
980 @code{(@var{pattern} . @var{coding})}, where @var{pattern} is a regular
981 expression that matches certain file names. The element applies to file
982 names that match @var{pattern}.
983
984 The @sc{cdr} of the element, @var{coding}, should be either a coding
985 system, a cons cell containing two coding systems, or a function name (a
986 symbol with a function definition). If @var{coding} is a coding system,
987 that coding system is used for both reading the file and writing it. If
988 @var{coding} is a cons cell containing two coding systems, its @sc{car}
989 specifies the coding system for decoding, and its @sc{cdr} specifies the
990 coding system for encoding.
991
992 If @var{coding} is a function name, the function should take one
993 argument, a list of all arguments passed to
994 @code{find-operation-coding-system}. It must return a coding system
995 or a cons cell containing two coding systems. This value has the same
996 meaning as described above.
997 @end defvar
998
999 @defvar process-coding-system-alist
1000 This variable is an alist specifying which coding systems to use for a
1001 subprocess, depending on which program is running in the subprocess. It
1002 works like @code{file-coding-system-alist}, except that @var{pattern} is
1003 matched against the program name used to start the subprocess. The coding
1004 system or systems specified in this alist are used to initialize the
1005 coding systems used for I/O to the subprocess, but you can specify
1006 other coding systems later using @code{set-process-coding-system}.
1007 @end defvar
1008
1009 @strong{Warning:} Coding systems such as @code{undecided}, which
1010 determine the coding system from the data, do not work entirely reliably
1011 with asynchronous subprocess output. This is because Emacs handles
1012 asynchronous subprocess output in batches, as it arrives. If the coding
1013 system leaves the character code conversion unspecified, or leaves the
1014 end-of-line conversion unspecified, Emacs must try to detect the proper
1015 conversion from one batch at a time, and this does not always work.
1016
1017 Therefore, with an asynchronous subprocess, if at all possible, use a
1018 coding system which determines both the character code conversion and
1019 the end of line conversion---that is, one like @code{latin-1-unix},
1020 rather than @code{undecided} or @code{latin-1}.
1021
1022 @defvar network-coding-system-alist
1023 This variable is an alist that specifies the coding system to use for
1024 network streams. It works much like @code{file-coding-system-alist},
1025 with the difference that the @var{pattern} in an element may be either a
1026 port number or a regular expression. If it is a regular expression, it
1027 is matched against the network service name used to open the network
1028 stream.
1029 @end defvar
1030
1031 @defvar default-process-coding-system
1032 This variable specifies the coding systems to use for subprocess (and
1033 network stream) input and output, when nothing else specifies what to
1034 do.
1035
1036 The value should be a cons cell of the form @code{(@var{input-coding}
1037 . @var{output-coding})}. Here @var{input-coding} applies to input from
1038 the subprocess, and @var{output-coding} applies to output to it.
1039 @end defvar
1040
1041 @defvar auto-coding-functions
1042 This variable holds a list of functions that try to determine a
1043 coding system for a file based on its undecoded contents.
1044
1045 Each function in this list should be written to look at text in the
1046 current buffer, but should not modify it in any way. The buffer will
1047 contain undecoded text of parts of the file. Each function should
1048 take one argument, @var{size}, which tells it how many characters to
1049 look at, starting from point. If the function succeeds in determining
1050 a coding system for the file, it should return that coding system.
1051 Otherwise, it should return @code{nil}.
1052
1053 If a file has a @samp{coding:} tag, that takes precedence, so these
1054 functions won't be called.
1055 @end defvar
1056
1057 @defun find-operation-coding-system operation &rest arguments
1058 This function returns the coding system to use (by default) for
1059 performing @var{operation} with @var{arguments}. The value has this
1060 form:
1061
1062 @example
1063 (@var{decoding-system} . @var{encoding-system})
1064 @end example
1065
1066 The first element, @var{decoding-system}, is the coding system to use
1067 for decoding (in case @var{operation} does decoding), and
1068 @var{encoding-system} is the coding system for encoding (in case
1069 @var{operation} does encoding).
1070
1071 The argument @var{operation} should be a symbol, any one of
1072 @code{insert-file-contents}, @code{write-region},
1073 @code{start-process}, @code{call-process}, @code{call-process-region},
1074 or @code{open-network-stream}. These are the names of the Emacs I/O
1075 primitives that can do coding system conversion.
1076
1077 The remaining arguments should be the same arguments that might be given
1078 to that I/O primitive. Depending on the primitive, one of those
1079 arguments is selected as the @dfn{target}. For example, if
1080 @var{operation} does file I/O, whichever argument specifies the file
1081 name is the target. For subprocess primitives, the process name is the
1082 target. For @code{open-network-stream}, the target is the service name
1083 or port number.
1084
1085 Depending on @var{operation}, this function looks up the target in
1086 @code{file-coding-system-alist}, @code{process-coding-system-alist},
1087 or @code{network-coding-system-alist}.
1088 @end defun
1089
1090 @node Specifying Coding Systems
1091 @subsection Specifying a Coding System for One Operation
1092
1093 You can specify the coding system for a specific operation by binding
1094 the variables @code{coding-system-for-read} and/or
1095 @code{coding-system-for-write}.
1096
1097 @defvar coding-system-for-read
1098 If this variable is non-@code{nil}, it specifies the coding system to
1099 use for reading a file, or for input from a synchronous subprocess.
1100
1101 It also applies to any asynchronous subprocess or network stream, but in
1102 a different way: the value of @code{coding-system-for-read} when you
1103 start the subprocess or open the network stream specifies the input
1104 decoding method for that subprocess or network stream. It remains in
1105 use for that subprocess or network stream unless and until overridden.
1106
1107 The right way to use this variable is to bind it with @code{let} for a
1108 specific I/O operation. Its global value is normally @code{nil}, and
1109 you should not globally set it to any other value. Here is an example
1110 of the right way to use the variable:
1111
1112 @example
1113 ;; @r{Read the file with no character code conversion.}
1114 ;; @r{Assume @acronym{crlf} represents end-of-line.}
1115 (let ((coding-system-for-read 'emacs-mule-dos))
1116 (insert-file-contents filename))
1117 @end example
1118
1119 When its value is non-@code{nil}, @code{coding-system-for-read} takes
1120 precedence over all other methods of specifying a coding system to use for
1121 input, including @code{file-coding-system-alist},
1122 @code{process-coding-system-alist} and
1123 @code{network-coding-system-alist}.
1124 @end defvar
1125
1126 @defvar coding-system-for-write
1127 This works much like @code{coding-system-for-read}, except that it
1128 applies to output rather than input. It affects writing to files,
1129 as well as sending output to subprocesses and net connections.
1130
1131 When a single operation does both input and output, as do
1132 @code{call-process-region} and @code{start-process}, both
1133 @code{coding-system-for-read} and @code{coding-system-for-write}
1134 affect it.
1135 @end defvar
1136
1137 @defvar inhibit-eol-conversion
1138 When this variable is non-@code{nil}, no end-of-line conversion is done,
1139 no matter which coding system is specified. This applies to all the
1140 Emacs I/O and subprocess primitives, and to the explicit encoding and
1141 decoding functions (@pxref{Explicit Encoding}).
1142 @end defvar
1143
1144 @node Explicit Encoding
1145 @subsection Explicit Encoding and Decoding
1146 @cindex encoding text
1147 @cindex decoding text
1148
1149 All the operations that transfer text in and out of Emacs have the
1150 ability to use a coding system to encode or decode the text.
1151 You can also explicitly encode and decode text using the functions
1152 in this section.
1153
1154 The result of encoding, and the input to decoding, are not ordinary
1155 text. They logically consist of a series of byte values; that is, a
1156 series of characters whose codes are in the range 0 through 255. In a
1157 multibyte buffer or string, character codes 128 through 159 are
1158 represented by multibyte sequences, but this is invisible to Lisp
1159 programs.
1160
1161 The usual way to read a file into a buffer as a sequence of bytes, so
1162 you can decode the contents explicitly, is with
1163 @code{insert-file-contents-literally} (@pxref{Reading from Files});
1164 alternatively, specify a non-@code{nil} @var{rawfile} argument when
1165 visiting a file with @code{find-file-noselect}. These methods result in
1166 a unibyte buffer.
1167
1168 The usual way to use the byte sequence that results from explicitly
1169 encoding text is to copy it to a file or process---for example, to write
1170 it with @code{write-region} (@pxref{Writing to Files}), and suppress
1171 encoding by binding @code{coding-system-for-write} to
1172 @code{no-conversion}.
1173
1174 Here are the functions to perform explicit encoding or decoding. The
1175 decoding functions produce sequences of bytes; the encoding functions
1176 are meant to operate on sequences of bytes. All of these functions
1177 discard text properties.
1178
1179 @deffn Command encode-coding-region start end coding-system
1180 This command encodes the text from @var{start} to @var{end} according
1181 to coding system @var{coding-system}. The encoded text replaces the
1182 original text in the buffer. The result of encoding is logically a
1183 sequence of bytes, but the buffer remains multibyte if it was multibyte
1184 before.
1185
1186 This command returns the length of the encoded text.
1187 @end deffn
1188
1189 @defun encode-coding-string string coding-system &optional nocopy
1190 This function encodes the text in @var{string} according to coding
1191 system @var{coding-system}. It returns a new string containing the
1192 encoded text, except when @var{nocopy} is non-@code{nil}, in which
1193 case the function may return @var{string} itself if the encoding
1194 operation is trivial. The result of encoding is a unibyte string.
1195 @end defun
1196
1197 @deffn Command decode-coding-region start end coding-system
1198 This command decodes the text from @var{start} to @var{end} according
1199 to coding system @var{coding-system}. The decoded text replaces the
1200 original text in the buffer. To make explicit decoding useful, the text
1201 before decoding ought to be a sequence of byte values, but both
1202 multibyte and unibyte buffers are acceptable.
1203
1204 This command returns the length of the decoded text.
1205 @end deffn
1206
1207 @defun decode-coding-string string coding-system &optional nocopy
1208 This function decodes the text in @var{string} according to coding
1209 system @var{coding-system}. It returns a new string containing the
1210 decoded text, except when @var{nocopy} is non-@code{nil}, in which
1211 case the function may return @var{string} itself if the decoding
1212 operation is trivial. To make explicit decoding useful, the contents
1213 of @var{string} ought to be a sequence of byte values, but a multibyte
1214 string is acceptable.
1215 @end defun
1216
1217 @defun decode-coding-inserted-region from to filename &optional visit beg end replace
1218 This function decodes the text from @var{from} to @var{to} as if
1219 it were being read from file @var{filename} using @code{insert-file-contents}
1220 using the rest of the arguments provided.
1221
1222 The normal way to use this function is after reading text from a file
1223 without decoding, if you decide you would rather have decoded it.
1224 Instead of deleting the text and reading it again, this time with
1225 decoding, you can call this function.
1226 @end defun
1227
1228 @node Terminal I/O Encoding
1229 @subsection Terminal I/O Encoding
1230
1231 Emacs can decode keyboard input using a coding system, and encode
1232 terminal output. This is useful for terminals that transmit or display
1233 text using a particular encoding such as Latin-1. Emacs does not set
1234 @code{last-coding-system-used} for encoding or decoding for the
1235 terminal.
1236
1237 @defun keyboard-coding-system
1238 This function returns the coding system that is in use for decoding
1239 keyboard input---or @code{nil} if no coding system is to be used.
1240 @end defun
1241
1242 @deffn Command set-keyboard-coding-system coding-system
1243 This command specifies @var{coding-system} as the coding system to
1244 use for decoding keyboard input. If @var{coding-system} is @code{nil},
1245 that means do not decode keyboard input.
1246 @end deffn
1247
1248 @defun terminal-coding-system
1249 This function returns the coding system that is in use for encoding
1250 terminal output---or @code{nil} for no encoding.
1251 @end defun
1252
1253 @deffn Command set-terminal-coding-system coding-system
1254 This command specifies @var{coding-system} as the coding system to use
1255 for encoding terminal output. If @var{coding-system} is @code{nil},
1256 that means do not encode terminal output.
1257 @end deffn
1258
1259 @node MS-DOS File Types
1260 @subsection MS-DOS File Types
1261 @cindex DOS file types
1262 @cindex MS-DOS file types
1263 @cindex Windows file types
1264 @cindex file types on MS-DOS and Windows
1265 @cindex text files and binary files
1266 @cindex binary files and text files
1267
1268 On MS-DOS and Microsoft Windows, Emacs guesses the appropriate
1269 end-of-line conversion for a file by looking at the file's name. This
1270 feature classifies files as @dfn{text files} and @dfn{binary files}. By
1271 ``binary file'' we mean a file of literal byte values that are not
1272 necessarily meant to be characters; Emacs does no end-of-line conversion
1273 and no character code conversion for them. On the other hand, the bytes
1274 in a text file are intended to represent characters; when you create a
1275 new file whose name implies that it is a text file, Emacs uses DOS
1276 end-of-line conversion.
1277
1278 @defvar buffer-file-type
1279 This variable, automatically buffer-local in each buffer, records the
1280 file type of the buffer's visited file. When a buffer does not specify
1281 a coding system with @code{buffer-file-coding-system}, this variable is
1282 used to determine which coding system to use when writing the contents
1283 of the buffer. It should be @code{nil} for text, @code{t} for binary.
1284 If it is @code{t}, the coding system is @code{no-conversion}.
1285 Otherwise, @code{undecided-dos} is used.
1286
1287 Normally this variable is set by visiting a file; it is set to
1288 @code{nil} if the file was visited without any actual conversion.
1289 @end defvar
1290
1291 @defopt file-name-buffer-file-type-alist
1292 This variable holds an alist for recognizing text and binary files.
1293 Each element has the form (@var{regexp} . @var{type}), where
1294 @var{regexp} is matched against the file name, and @var{type} may be
1295 @code{nil} for text, @code{t} for binary, or a function to call to
1296 compute which. If it is a function, then it is called with a single
1297 argument (the file name) and should return @code{t} or @code{nil}.
1298
1299 When running on MS-DOS or MS-Windows, Emacs checks this alist to decide
1300 which coding system to use when reading a file. For a text file,
1301 @code{undecided-dos} is used. For a binary file, @code{no-conversion}
1302 is used.
1303
1304 If no element in this alist matches a given file name, then
1305 @code{default-buffer-file-type} says how to treat the file.
1306 @end defopt
1307
1308 @defopt default-buffer-file-type
1309 This variable says how to handle files for which
1310 @code{file-name-buffer-file-type-alist} says nothing about the type.
1311
1312 If this variable is non-@code{nil}, then these files are treated as
1313 binary: the coding system @code{no-conversion} is used. Otherwise,
1314 nothing special is done for them---the coding system is deduced solely
1315 from the file contents, in the usual Emacs fashion.
1316 @end defopt
1317
1318 @node Input Methods
1319 @section Input Methods
1320 @cindex input methods
1321
1322 @dfn{Input methods} provide convenient ways of entering non-@acronym{ASCII}
1323 characters from the keyboard. Unlike coding systems, which translate
1324 non-@acronym{ASCII} characters to and from encodings meant to be read by
1325 programs, input methods provide human-friendly commands. (@xref{Input
1326 Methods,,, emacs, The GNU Emacs Manual}, for information on how users
1327 use input methods to enter text.) How to define input methods is not
1328 yet documented in this manual, but here we describe how to use them.
1329
1330 Each input method has a name, which is currently a string;
1331 in the future, symbols may also be usable as input method names.
1332
1333 @defvar current-input-method
1334 This variable holds the name of the input method now active in the
1335 current buffer. (It automatically becomes local in each buffer when set
1336 in any fashion.) It is @code{nil} if no input method is active in the
1337 buffer now.
1338 @end defvar
1339
1340 @defopt default-input-method
1341 This variable holds the default input method for commands that choose an
1342 input method. Unlike @code{current-input-method}, this variable is
1343 normally global.
1344 @end defopt
1345
1346 @deffn Command set-input-method input-method
1347 This command activates input method @var{input-method} for the current
1348 buffer. It also sets @code{default-input-method} to @var{input-method}.
1349 If @var{input-method} is @code{nil}, this command deactivates any input
1350 method for the current buffer.
1351 @end deffn
1352
1353 @defun read-input-method-name prompt &optional default inhibit-null
1354 This function reads an input method name with the minibuffer, prompting
1355 with @var{prompt}. If @var{default} is non-@code{nil}, that is returned
1356 by default, if the user enters empty input. However, if
1357 @var{inhibit-null} is non-@code{nil}, empty input signals an error.
1358
1359 The returned value is a string.
1360 @end defun
1361
1362 @defvar input-method-alist
1363 This variable defines all the supported input methods.
1364 Each element defines one input method, and should have the form:
1365
1366 @example
1367 (@var{input-method} @var{language-env} @var{activate-func}
1368 @var{title} @var{description} @var{args}...)
1369 @end example
1370
1371 Here @var{input-method} is the input method name, a string;
1372 @var{language-env} is another string, the name of the language
1373 environment this input method is recommended for. (That serves only for
1374 documentation purposes.)
1375
1376 @var{activate-func} is a function to call to activate this method. The
1377 @var{args}, if any, are passed as arguments to @var{activate-func}. All
1378 told, the arguments to @var{activate-func} are @var{input-method} and
1379 the @var{args}.
1380
1381 @var{title} is a string to display in the mode line while this method is
1382 active. @var{description} is a string describing this method and what
1383 it is good for.
1384 @end defvar
1385
1386 The fundamental interface to input methods is through the
1387 variable @code{input-method-function}. @xref{Reading One Event},
1388 and @ref{Invoking the Input Method}.
1389
1390 @node Locales
1391 @section Locales
1392 @cindex locale
1393
1394 POSIX defines a concept of ``locales'' which control which language
1395 to use in language-related features. These Emacs variables control
1396 how Emacs interacts with these features.
1397
1398 @defvar locale-coding-system
1399 @tindex locale-coding-system
1400 @cindex keyboard input decoding on X
1401 This variable specifies the coding system to use for decoding system
1402 error messages and---on X Window system only---keyboard input, for
1403 encoding the format argument to @code{format-time-string}, and for
1404 decoding the return value of @code{format-time-string}.
1405 @end defvar
1406
1407 @defvar system-messages-locale
1408 @tindex system-messages-locale
1409 This variable specifies the locale to use for generating system error
1410 messages. Changing the locale can cause messages to come out in a
1411 different language or in a different orthography. If the variable is
1412 @code{nil}, the locale is specified by environment variables in the
1413 usual POSIX fashion.
1414 @end defvar
1415
1416 @defvar system-time-locale
1417 @tindex system-time-locale
1418 This variable specifies the locale to use for formatting time values.
1419 Changing the locale can cause messages to appear according to the
1420 conventions of a different language. If the variable is @code{nil}, the
1421 locale is specified by environment variables in the usual POSIX fashion.
1422 @end defvar
1423
1424 @defun locale-info item
1425 This function returns locale data @var{item} for the current POSIX
1426 locale, if available. @var{item} should be one of these symbols:
1427
1428 @table @code
1429 @item codeset
1430 Return the character set as a string (locale item @code{CODESET}).
1431
1432 @item days
1433 Return a 7-element vector of day names (locale items
1434 @code{DAY_1} through @code{DAY_7});
1435
1436 @item months
1437 Return a 12-element vector of month names (locale items @code{MON_1}
1438 through @code{MON_12}).
1439
1440 @item paper
1441 Return a list @code{(@var{width} @var{height})} for the default paper
1442 size measured in millimeters (locale items @code{PAPER_WIDTH} and
1443 @code{PAPER_HEIGHT}).
1444 @end table
1445
1446 If the system can't provide the requested information, or if
1447 @var{item} is not one of those symbols, the value is @code{nil}. All
1448 strings in the return value are decoded using
1449 @code{locale-coding-system}. @xref{Locales,,, libc, The GNU Libc Manual},
1450 for more information about locales and locale items.
1451 @end defun
1452
1453 @ignore
1454 arch-tag: be705bf8-941b-4c35-84fc-ad7d20ddb7cb
1455 @end ignore