Non-ASCII Characters
********************

This chapter covers the special issues relating to non-ASCII characters and how they are stored in strings and buffers.

Text Representations
====================

Emacs has two text representations---two ways to represent text in a string or buffer.  These are called unibyte and multibyte.  Each string, and each buffer, uses one of these two representations.  For most purposes, you can ignore the issue of representations, because Emacs converts text between them as appropriate.  Occasionally in Lisp programming you will need to pay attention to the difference.

In unibyte representation, each character occupies one byte and therefore the possible character codes range from 0 to 255.  Codes 0 through 127 are ASCII characters; the codes from 128 through 255 are used for one non-ASCII character set (you can choose which character set by setting the variable nonascii-insert-offset).

In multibyte representation, a character may occupy more than one byte, and as a result, the full range of Emacs character codes can be stored.  The first byte of a multibyte character is always in the range 128 through 159 (octal 0200 through 0237).  These values are called leading codes.  The second and subsequent bytes of a multibyte character are always in the range 160 through 255 (octal 0240 through 0377); these values are trailing codes.

Some sequences of bytes are not valid in multibyte text: for example, a single isolated byte in the range 128 through 159 is not allowed. But character codes 128 through 159 can appear in multibyte text, represented as two-byte sequences.  None of the character codes 128 through 255 normally appear in ordinary multibyte text, but they do appear in multibyte buffers and strings when you do explicit encoding and decoding (see Explicit Encoding).

In a buffer, the buffer-local value of the variable enable-multibyte-characters specifies the representation used. The representation for a string is determined and recorded in the string when the string is constructed.

 -- Variable: enable-multibyte-characters
     This variable specifies the current buffer's text representation. If it is non-nil, the buffer contains multibyte text; otherwise, it contains unibyte text.

     You cannot set this variable directly; instead, use the function set-buffer-multibyte to change a buffer's representation.

 -- Variable: default-enable-multibyte-characters
     This variable's value is entirely equivalent to (default-value 'enable-multibyte-characters), and setting this variable changes that default value.  Setting the local binding of enable-multibyte-characters in a specific buffer is not allowed, but changing the default value is supported, and it is a reasonable thing to do, because it has no effect on existing buffers.

     The --unibyte command line option does its job by setting the default value to nil early in startup.

 -- Function: position-bytes position
     Return the byte-position corresponding to buffer position POSITION in the current buffer.

 -- Function: byte-to-position byte-position
     Return the buffer position corresponding to byte-position BYTE-POSITION in the current buffer.

 -- Function: multibyte-string-p string
     Return t if STRING is a multibyte string.

Converting Text Representations
===============================

Emacs can convert unibyte text to multibyte; it can also convert multibyte text to unibyte, though this conversion loses information.  In general these conversions happen when inserting text into a buffer, or when putting text from several strings together in one string.  You can also explicitly convert a string's contents to either representation.

Emacs chooses the representation for a string based on the text that it is constructed from.  The general rule is to convert unibyte text to multibyte text when combining it with other multibyte text, because the multibyte representation is more general and can hold whatever characters the unibyte text has.

When inserting text into a buffer, Emacs converts the text to the buffer's representation, as specified by enable-multibyte-characters in that buffer.  In particular, when you insert multibyte text into a unibyte buffer, Emacs converts the text to unibyte, even though this conversion cannot in general preserve all the characters that might be in the multibyte text.  The other natural alternative, to convert the buffer contents to multibyte, is not acceptable because the buffer's representation is a choice made by the user that cannot be overridden automatically.

Converting unibyte text to multibyte text leaves ASCII characters unchanged, and likewise 128 through 159.  It converts the non-ASCII codes 160 through 255 by adding the value nonascii-insert-offset to each character code.  By setting this variable, you specify which character set the unibyte characters correspond to (see Character Sets).  For example, if nonascii-insert-offset is 2048, which is (- (make-char 'latin-iso8859-1) 128), then the unibyte non-ASCII characters correspond to Latin 1.  If it is 2688, which is (- (make-char 'greek-iso8859-7) 128), then they correspond to Greek letters.

Converting multibyte text to unibyte is simpler: it discards all but the low 8 bits of each character code.  If nonascii-insert-offset has a reasonable value, corresponding to the beginning of some character set, this conversion is the inverse of the other: converting unibyte text to multibyte and back to unibyte reproduces the original unibyte text.

 -- Variable: nonascii-insert-offset
     This variable specifies the amount to add to a non-ASCII character when converting unibyte text to multibyte.  It also applies when self-insert-command inserts a character in the unibyte non-ASCII range, 128 through 255.  However, the function insert-char does not perform this conversion.

     The right value to use to select character set CS is (- (make-char CS) 128).  If the value of nonascii-insert-offset is zero, then conversion actually uses the value for the Latin 1 character set, rather than zero.

 -- Variable: nonascii-translation-table
     This variable provides a more general alternative to nonascii-insert-offset.  You can use it to specify independently how to translate each code in the range of 128 through 255 into a multibyte character.  The value should be a vector, or nil. If this is non-nil, it overrides nonascii-insert-offset.

 -- Function: string-make-unibyte string
     This function converts the text of STRING to unibyte representation, if it isn't already, and returns the result.  If STRING is a unibyte string, it is returned unchanged.

 -- Function: string-make-multibyte string
     This function converts the text of STRING to multibyte representation, if it isn't already, and returns the result.  If STRING is a multibyte string, it is returned unchanged.

Selecting a Representation
==========================

Sometimes it is useful to examine an existing buffer or string as multibyte when it was unibyte, or vice versa.

 -- Function: set-buffer-multibyte multibyte
     Set the representation type of the current buffer.  If MULTIBYTE is non-nil, the buffer becomes multibyte.  If MULTIBYTE is nil, the buffer becomes unibyte.

     This function leaves the buffer contents unchanged when viewed as a sequence of bytes.  As a consequence, it can change the contents viewed as characters; a sequence of two bytes which is treated as one character in multibyte representation will count as two characters in unibyte representation.

     This function sets enable-multibyte-characters to record which representation is in use.  It also adjusts various data in the buffer (including overlays, text properties and markers) so that they cover the same text as they did before.

     You cannot use set-buffer-multibyte on an indirect buffer, because indirect buffers always inherit the representation of the base buffer.

 -- Function: string-as-unibyte string
     This function returns a string with the same bytes as STRING but treating each byte as a character.  This means that the value may have more characters than STRING has.

     If STRING is already a unibyte string, then the value is STRING itself.

 -- Function: string-as-multibyte string
     This function returns a string with the same bytes as STRING but treating each multibyte sequence as one character.  This means that the value may have fewer characters than STRING has.

     If STRING is already a multibyte string, then the value is STRING itself.

Character Codes
===============

The unibyte and multibyte text representations use different character codes.  The valid character codes for unibyte representation range from 0 to 255---the values that can fit in one byte.  The valid character codes for multibyte representation range from 0 to 524287, but not all values in that range are valid.  The values 128 through 255 are not really proper in multibyte text, but they can occur if you do explicit encoding and decoding (see Explicit Encoding).  Some other character codes cannot occur at all in multibyte text.  Only the ASCII codes 0 through 127 are truly legitimate in both representations.

 -- Function: char-valid-p charcode
     This returns t if CHARCODE is valid for either one of the two text representations.

          (char-valid-p 65)
               => t
          (char-valid-p 256)
               => nil
          (char-valid-p 2248)
               => t

Character Sets
==============

Emacs classifies characters into various character sets, each of which has a name which is a symbol.  Each character belongs to one and only one character set.

In general, there is one character set for each distinct script.  For example, latin-iso8859-1 is one character set, greek-iso8859-7 is another, and ascii is another.  An Emacs character set can hold at most 9025 characters; therefore, in some cases, characters that would logically be grouped together are split into several character sets.  For example, one set of Chinese characters, generally known as Big 5, is divided into two Emacs character sets, chinese-big5-1 and chinese-big5-2.

 -- Function: charsetp object
     Returns t if OBJECT is a symbol that names a character set, nil otherwise.

 -- Function: charset-list
     This function returns a list of all defined character set names.

 -- Function: char-charset character
     This function returns the name of the character set that CHARACTER belongs to.

 -- Function: charset-plist charset
     This function returns the charset property list of the character set CHARSET.  Although CHARSET is a symbol, this is not the same as the property list of that symbol.  Charset properties are used for special purposes within Emacs; for example, x-charset-registry helps determine which fonts to use (see Font Selection).

Characters and Bytes
====================

In multibyte representation, each character occupies one or more bytes.  Each character set has an introduction sequence, which is normally one or two bytes long.  (Exception: the ASCII character set has a zero-length introduction sequence.)  The introduction sequence is the beginning of the byte sequence for any character in the character set.  The rest of the character's bytes distinguish it from the other characters in the same character set.  Depending on the character set, there are either one or two distinguishing bytes; the number of such bytes is called the dimension of the character set.

 -- Function: charset-dimension charset
     This function returns the dimension of CHARSET; at present, the dimension is always 1 or 2.

 -- Function: charset-bytes charset
     This function returns the number of bytes used to represent a character in character set CHARSET.

This is the simplest way to determine the byte length of a character set's introduction sequence:

     (- (charset-bytes CHARSET)
        (charset-dimension CHARSET))

Splitting Characters
====================

The functions in this section convert between characters and the byte values used to represent them.  For most purposes, there is no need to be concerned with the sequence of bytes used to represent a character, because Emacs translates automatically when necessary.

 -- Function: split-char character
     Return a list containing the name of the character set of CHARACTER, followed by one or two byte values (integers) which identify CHARACTER within that character set.  The number of byte values is the character set's dimension.

          (split-char 2248)
               => (latin-iso8859-1 72)
          (split-char 65)
               => (ascii 65)

     Unibyte non-ASCII characters are considered as part of the ascii character set:

          (split-char 192)
               => (ascii 192)

 -- Function: make-char charset &rest byte-values
     This function returns the character in character set CHARSET identified by BYTE-VALUES.  This is roughly the inverse of split-char.  Normally, you should specify either one or two BYTE-VALUES, according to the dimension of CHARSET.  For example,

          (make-char 'latin-iso8859-1 72)
               => 2248

If you call make-char with no BYTE-VALUES, the result is a generic character which stands for CHARSET.  A generic character is an integer, but it is not valid for insertion in the buffer as a character.  It can be used in char-table-range to refer to the whole character set (see Char-Tables). char-valid-p returns nil for generic characters. For example:

     (make-char 'latin-iso8859-1)
          => 2176
     (char-valid-p 2176)
          => nil
     (split-char 2176)
          => (latin-iso8859-1 0)

Scanning for Character Sets
===========================

Sometimes it is useful to find out which character sets appear in a part of a buffer or a string.  One use for this is in determining which coding systems (see Coding Systems) are capable of representing all of the text in question.

 -- Function: find-charset-region beg end &optional translation
     This function returns a list of the character sets that appear in the current buffer between positions BEG and END.

     The optional argument TRANSLATION specifies a translation table to be used in scanning the text (see Translation of Characters).  If it is non-nil, then each character in the region is translated through this table, and the value returned describes the translated characters instead of the characters actually in the buffer.

     When a buffer contains non-ASCII characters, codes 128 through 255, they are assigned the character set unknown.  See Explicit Encoding.

      -- Function: find-charset-string string &optional translation
          This function returns a list of the character sets that appear in the string STRING.  It is just like find-charset-region, except that it applies to the contents of STRING instead of part of the current buffer.

Translation of Characters
=========================

A translation table specifies a mapping of characters into characters.  These tables are used in encoding and decoding, and for other purposes.  Some coding systems specify their own particular translation tables; there are also default translation tables which apply to all other coding systems.

 -- Function: make-translation-table &rest translations
     This function returns a translation table based on the argument TRANSLATIONS.  Each element of TRANSLATIONS should be a list of the form (FROM . TO); this says to translate the character FROM into TO.

     You can also map one whole character set into another character set with the same dimension.  To do this, you specify a generic character (which designates a character set) for FROM (see Splitting Characters). In this case, TO should also be a generic character, for another character set of the same dimension.  Then the translation table translates each character of FROM's character set into the corresponding character of TO's character set.

In decoding, the translation table's translations are applied to the characters that result from ordinary decoding.  If a coding system has property character-translation-table-for-decode, that specifies the translation table to use.  Otherwise, if standard-translation-table-for-decode is non-nil, decoding uses that table.

In encoding, the translation table's translations are applied to the characters in the buffer, and the result of translation is actually encoded.  If a coding system has property character-translation-table-for-encode, that specifies the translation table to use.  Otherwise the variable standard-translation-table-for-encode specifies the translation table.

 -- Variable: standard-translation-table-for-decode
     This is the default translation table for decoding, for coding systems that don't specify any other translation table.

 -- Variable: standard-translation-table-for-encode
     This is the default translation table for encoding, for coding systems that don't specify any other translation table.

Coding Systems
==============

When Emacs reads or writes a file, and when Emacs sends text to a subprocess or receives text from a subprocess, it normally performs character code conversion and end-of-line conversion as specified by a particular coding system.

How to define a coding system is an arcane matter, and is not documented here.

Basic Concepts of Coding Systems
--------------------------------

Character code conversion involves conversion between the encoding used inside Emacs and some other encoding.  Emacs supports many different encodings, in that it can convert to and from them.  For example, it can convert text to or from encodings such as Latin 1, Latin 2, Latin 3, Latin 4, Latin 5, and several variants of ISO 2022.  In some cases, Emacs supports several alternative encodings for the same characters; for example, there are three coding systems for the Cyrillic (Russian) alphabet: ISO, Alternativnyj, and KOI8.

Most coding systems specify a particular character code for conversion, but some of them leave the choice unspecified---to be chosen heuristically for each file, based on the data.

End of line conversion handles three different conventions used on various systems for representing end of line in files.  The Unix convention is to use the linefeed character (also called newline).  The DOS convention is to use a carriage-return and a linefeed at the end of a line.  The Mac convention is to use just carriage-return.

Base coding systems such as latin-1 leave the end-of-line conversion unspecified, to be chosen based on the data.  Variant coding systems such as latin-1-unix, latin-1-dos and latin-1-mac specify the end-of-line conversion explicitly as well.  Most base coding systems have three corresponding variants whose names are formed by adding -unix, -dos and -mac.

The coding system raw-text is special in that it prevents character code conversion, and causes the buffer visited with that coding system to be a unibyte buffer.  It does not specify the end-of-line conversion, allowing that to be determined as usual by the data, and has the usual three variants which specify the end-of-line conversion.  no-conversion is equivalent to raw-text-unix: it specifies no conversion of either character codes or end-of-line.

The coding system emacs-mule specifies that the data is represented in the internal Emacs encoding.  This is like raw-text in that no code conversion happens, but different in that the result is multibyte data.

 -- Function: coding-system-get coding-system property
     This function returns the specified property of the coding system CODING-SYSTEM.  Most coding system properties exist for internal purposes, but one that you might find useful is mime-charset. That property's value is the name used in MIME for the character coding which this coding system can read and write.  Examples:

          (coding-system-get 'iso-latin-1 'mime-charset)
               => iso-8859-1
          (coding-system-get 'iso-2022-cn 'mime-charset)
               => iso-2022-cn
          (coding-system-get 'cyrillic-koi8 'mime-charset)
               => koi8-r

     The value of the mime-charset property is also defined as an alias for the coding system.

Encoding and I/O
----------------

The principal purpose of coding systems is for use in reading and writing files.  The function insert-file-contents uses a coding system for decoding the file data, and write-region uses one to encode the buffer contents.

You can specify the coding system to use either explicitly (see Specifying Coding Systems), or implicitly using the defaulting mechanism (see Default Coding Systems).  But these methods may not completely specify what to do.  For example, they may choose a coding system such as undefined which leaves the character code conversion to be determined from the data.  In these cases, the I/O operation finishes the job of choosing a coding system.  Very often you will want to find out afterwards which coding system was chosen.

 -- Variable: buffer-file-coding-system
     This variable records the coding system that was used for visiting the current buffer.  It is used for saving the buffer, and for writing part of the buffer with write-region.  When those operations ask the user to specify a different coding system, buffer-file-coding-system is updated to the coding system specified.

     However, buffer-file-coding-system does not affect sending text to a subprocess.

 -- Variable: save-buffer-coding-system
     This variable specifies the coding system for saving the buffer---but it is not used for write-region.

     When a command to save the buffer starts out to use save-buffer-coding-system, and that coding system cannot handle the actual text in the buffer, the command asks the user to choose another coding system.  After that happens, the command also updates save-buffer-coding-system to represent the coding system that the user specified.

 -- Variable: last-coding-system-used
     I/O operations for files and subprocesses set this variable to the coding system name that was used.  The explicit encoding and decoding functions (see Explicit Encoding) set it too.

     Warning: Since receiving subprocess output sets this variable, it can change whenever Emacs waits; therefore, you should copy the value shortly after the function call that stores the value you are interested in.

The variable selection-coding-system specifies how to encode selections for the window system.  See Window System Selections.

Coding Systems in Lisp
----------------------

Here are the Lisp facilities for working with coding systems:

 -- Function: coding-system-list &optional base-only
     This function returns a list of all coding system names (symbols).  If BASE-ONLY is non-nil, the value includes only the base coding systems.  Otherwise, it includes variant coding systems as well.

 -- Function: coding-system-p object
     This function returns t if OBJECT is a coding system name.

 -- Function: check-coding-system coding-system
     This function checks the validity of CODING-SYSTEM. If that is valid, it returns CODING-SYSTEM. Otherwise it signals an error with condition coding-system-error.

 -- Function: coding-system-change-eol-conversion coding-system eol-type
     This function returns a coding system which is like CODING-SYSTEM except for its eol conversion, which is specified by eol-type. EOL-TYPE should be unix, dos, mac, or nil.  If it is nil, the returned coding system determines the end-of-line conversion from the data.

 -- Function: coding-system-change-text-conversion eol-coding text-coding
     This function returns a coding system which uses the end-of-line conversion of EOL-CODING, and the text conversion of TEXT-CODING.  If TEXT-CODING is nil, it returns undecided, or one of its variants according to EOL-CODING.

 -- Function: find-coding-systems-region from to
     This function returns a list of coding systems that could be used to encode a text between FROM and TO.  All coding systems in the list can safely encode any multibyte characters in that portion of the text.

     If the text contains no multibyte characters, the function returns the list (undecided).

 -- Function: find-coding-systems-string string
     This function returns a list of coding systems that could be used to encode the text of STRING.  All coding systems in the list can safely encode any multibyte characters in STRING.  If the text contains no multibyte characters, this returns the list (undecided).

 -- Function: find-coding-systems-for-charsets charsets
     This function returns a list of coding systems that could be used to encode all the character sets in the list CHARSETS.

 -- Function: detect-coding-region start end &optional highest
     This function chooses a plausible coding system for decoding the text from START to END.  This text should be a byte sequence (see Explicit Encoding).

     Normally this function returns a list of coding systems that could handle decoding the text that was scanned.  They are listed in order of decreasing priority.  But if HIGHEST is non-nil, then the return value is just one coding system, the one that is highest in priority.

     If the region contains only ASCII characters, the value is undecided or (undecided).

 -- Function: detect-coding-string string highest
     This function is like detect-coding-region except that it operates on the contents of STRING instead of bytes in the buffer.

See Process Information, for how to examine or set the coding systems used for I/O to a subprocess.

User-Chosen Coding Systems
--------------------------

 -- Function: select-safe-coding-system from to &optional preferred-coding-system
     This function selects a coding system for encoding the text between FROM and TO, asking the user to choose if necessary.

     The optional argument PREFERRED-CODING-SYSTEM specifies a coding system to try first.  If that one can handle the text in the specified region, then it is used.  If this argument is omitted, the current buffer's value of buffer-file-coding-system is tried first.

     If the region contains some multibyte characters that the preferred coding system cannot encode, this function asks the user to choose from a list of coding systems which can encode the text, and returns the user's choice.

     One other kludgy feature: if FROM is a string, the string is the target text, and TO is ignored.

Here are two functions you can use to let the user specify a coding system, with completion.  See Completion.

 -- Function: read-coding-system prompt &optional default
     This function reads a coding system using the minibuffer, prompting with string PROMPT, and returns the coding system name as a symbol.  If the user enters null input, DEFAULT specifies which coding system to return.  It should be a symbol or a string.

 -- Function: read-non-nil-coding-system prompt
     This function reads a coding system using the minibuffer, prompting with string PROMPT, and returns the coding system name as a symbol.  If the user tries to enter null input, it asks the user to try again. See Coding Systems.

Default Coding Systems
----------------------

This section describes variables that specify the default coding system for certain files or when running certain subprograms, and the function that I/O operations use to access them.

The idea of these variables is that you set them once and for all to the defaults you want, and then do not change them again.  To specify a particular coding system for a particular operation in a Lisp program, don't change these variables; instead, override them using coding-system-for-read and coding-system-for-write (see Specifying Coding Systems).

 -- Variable: file-coding-system-alist
     This variable is an alist that specifies the coding systems to use for reading and writing particular files.  Each element has the form (PATTERN . CODING), where PATTERN is a regular expression that matches certain file names.  The element applies to file names that match PATTERN.

     The CDR of the element, CODING, should be either a coding system, a cons cell containing two coding systems, or a function name (a symbol with a function definition).  If CODING is a coding system, that coding system is used for both reading the file and writing it.  If CODING is a cons cell containing two coding systems, its CAR specifies the coding system for decoding, and its CDR specifies the coding system for encoding.

     If CODING is a function name, the function must return a coding system or a cons cell containing two coding systems.  This value is used as described above.

 -- Variable: process-coding-system-alist
     This variable is an alist specifying which coding systems to use for a subprocess, depending on which program is running in the subprocess.  It works like file-coding-system-alist, except that PATTERN is matched against the program name used to start the subprocess.  The coding system or systems specified in this alist are used to initialize the coding systems used for I/O to the subprocess, but you can specify other coding systems later using set-process-coding-system.

Warning: Coding systems such as undecided, which determine the coding system from the data, do not work entirely reliably with asynchronous subprocess output.  This is because Emacs handles asynchronous subprocess output in batches, as it arrives.  If the coding system leaves the character code conversion unspecified, or leaves the end-of-line conversion unspecified, Emacs must try to detect the proper conversion from one batch at a time, and this does not always work.

Therefore, with an asynchronous subprocess, if at all possible, use a coding system which determines both the character code conversion and the end of line conversion---that is, one like latin-1-unix, rather than undecided or latin-1.

 -- Variable: network-coding-system-alist
     This variable is an alist that specifies the coding system to use for network streams.  It works much like file-coding-system-alist, with the difference that the PATTERN in an element may be either a port number or a regular expression.  If it is a regular expression, it is matched against the network service name used to open the network stream.

 -- Variable: default-process-coding-system
     This variable specifies the coding systems to use for subprocess (and network stream) input and output, when nothing else specifies what to do.

     The value should be a cons cell of the form (INPUT-CODING . OUTPUT-CODING).  Here INPUT-CODING applies to input from the subprocess, and OUTPUT-CODING applies to output to it.

 -- Function: find-operation-coding-system operation &rest arguments
     This function returns the coding system to use (by default) for performing OPERATION with ARGUMENTS.  The value has this form:

          (DECODING-SYSTEM ENCODING-SYSTEM)

     The first element, DECODING-SYSTEM, is the coding system to use for decoding (in case OPERATION does decoding), and ENCODING-SYSTEM is the coding system for encoding (in case OPERATION does encoding).

     The argument OPERATION should be a symbol, one of insert-file-contents, write-region, call-process, call-process-region, start-process, or open-network-stream.  These are the names of the Emacs I/O primitives that can do coding system conversion.

     The remaining arguments should be the same arguments that might be given to that I/O primitive.  Depending on the primitive, one of those arguments is selected as the target.  For example, if OPERATION does file I/O, whichever argument specifies the file name is the target.  For subprocess primitives, the process name is the target.  For open-network-stream, the target is the service name or port number.

     This function looks up the target in file-coding-system-alist, process-coding-system-alist, or network-coding-system-alist, depending on OPERATION. See Default Coding Systems.

Specifying a Coding System for One Operation
--------------------------------------------

You can specify the coding system for a specific operation by binding the variables coding-system-for-read and/or coding-system-for-write.

 -- Variable: coding-system-for-read
     If this variable is non-nil, it specifies the coding system to use for reading a file, or for input from a synchronous subprocess.

     It also applies to any asynchronous subprocess or network stream, but in a different way: the value of coding-system-for-read when you start the subprocess or open the network stream specifies the input decoding method for that subprocess or network stream.  It remains in use for that subprocess or network stream unless and until overridden.

     The right way to use this variable is to bind it with let for a specific I/O operation.  Its global value is normally nil, and you should not globally set it to any other value.  Here is an example of the right way to use the variable:

          ;; Read the file with no character code conversion.
          ;; Assume CRLF represents end-of-line.
          (let ((coding-system-for-write 'emacs-mule-dos))
            (insert-file-contents filename))

     When its value is non-nil, coding-system-for-read takes precedence over all other methods of specifying a coding system to use for input, including file-coding-system-alist, process-coding-system-alist and network-coding-system-alist.

 -- Variable: coding-system-for-write
     This works much like coding-system-for-read, except that it applies to output rather than input.  It affects writing to files, as well as sending output to subprocesses and net connections.

     When a single operation does both input and output, as do call-process-region and start-process, both coding-system-for-read and coding-system-for-write affect it.

 -- Variable: inhibit-eol-conversion
     When this variable is non-nil, no end-of-line conversion is done, no matter which coding system is specified.  This applies to all the Emacs I/O and subprocess primitives, and to the explicit encoding and decoding functions (see Explicit Encoding).

Explicit Encoding and Decoding
------------------------------

All the operations that transfer text in and out of Emacs have the ability to use a coding system to encode or decode the text. You can also explicitly encode and decode text using the functions in this section.

The result of encoding, and the input to decoding, are not ordinary text.  They logically consist of a series of byte values; that is, a series of characters whose codes are in the range 0 through 255.  In a multibyte buffer or string, character codes 128 through 159 are represented by multibyte sequences, but this is invisible to Lisp programs.

The usual way to read a file into a buffer as a sequence of bytes, so you can decode the contents explicitly, is with insert-file-contents-literally (see Reading from Files); alternatively, specify a non-nil RAWFILE argument when visiting a file with find-file-noselect.  These methods result in a unibyte buffer.

The usual way to use the byte sequence that results from explicitly encoding text is to copy it to a file or process---for example, to write it with write-region (see Writing to Files), and suppress encoding by binding coding-system-for-write to no-conversion.

Here are the functions to perform explicit encoding or decoding.  The decoding functions produce sequences of bytes; the encoding functions are meant to operate on sequences of bytes.  All of these functions discard text properties.

 -- Function: encode-coding-region start end coding-system
     This function encodes the text from START to END according to coding system CODING-SYSTEM.  The encoded text replaces the original text in the buffer.  The result of encoding is logically a sequence of bytes, but the buffer remains multibyte if it was multibyte before.

 -- Function: encode-coding-string string coding-system
     This function encodes the text in STRING according to coding system CODING-SYSTEM.  It returns a new string containing the encoded text.  The result of encoding is a unibyte string.

 -- Function: decode-coding-region start end coding-system
     This function decodes the text from START to END according to coding system CODING-SYSTEM.  The decoded text replaces the original text in the buffer.  To make explicit decoding useful, the text before decoding ought to be a sequence of byte values, but both multibyte and unibyte buffers are acceptable.

 -- Function: decode-coding-string string coding-system
     This function decodes the text in STRING according to coding system CODING-SYSTEM.  It returns a new string containing the decoded text.  To make explicit decoding useful, the contents of STRING ought to be a sequence of byte values, but a multibyte string is acceptable.

Terminal I/O Encoding
---------------------

Emacs can decode keyboard input using a coding system, and encode terminal output.  This is useful for terminals that transmit or display text using a particular encoding such as Latin-1.  Emacs does not set last-coding-system-used for encoding or decoding for the terminal.

 -- Function: keyboard-coding-system
     This function returns the coding system that is in use for decoding keyboard input---or nil if no coding system is to be used.

 -- Function: set-keyboard-coding-system coding-system
     This function specifies CODING-SYSTEM as the coding system to use for decoding keyboard input.  If CODING-SYSTEM is nil, that means do not decode keyboard input.

 -- Function: terminal-coding-system
     This function returns the coding system that is in use for encoding terminal output---or nil for no encoding.

 -- Function: set-terminal-coding-system coding-system
     This function specifies CODING-SYSTEM as the coding system to use for encoding terminal output.  If CODING-SYSTEM is nil, that means do not encode terminal output.

MS-DOS File Types
-----------------

On MS-DOS and Microsoft Windows, Emacs guesses the appropriate end-of-line conversion for a file by looking at the file's name.  This feature classifies files as text files and binary files.  By ``binary file'' we mean a file of literal byte values that are not necessarily meant to be characters; Emacs does no end-of-line conversion and no character code conversion for them.  On the other hand, the bytes in a text file are intended to represent characters; when you create a new file whose name implies that it is a text file, Emacs uses DOS end-of-line conversion.

 -- Variable: buffer-file-type
     This variable, automatically buffer-local in each buffer, records the file type of the buffer's visited file.  When a buffer does not specify a coding system with buffer-file-coding-system, this variable is used to determine which coding system to use when writing the contents of the buffer.  It should be nil for text, t for binary. If it is t, the coding system is no-conversion. Otherwise, undecided-dos is used.

     Normally this variable is set by visiting a file; it is set to nil if the file was visited without any actual conversion.

 -- User Option: file-name-buffer-file-type-alist
     This variable holds an alist for recognizing text and binary files. Each element has the form (REGEXP . TYPE), where REGEXP is matched against the file name, and TYPE may be nil for text, t for binary, or a function to call to compute which.  If it is a function, then it is called with a single argument (the file name) and should return t or nil.

     When running on MS-DOS or MS-Windows, Emacs checks this alist to decide which coding system to use when reading a file.  For a text file, undecided-dos is used.  For a binary file, no-conversion is used.

     If no element in this alist matches a given file name, then default-buffer-file-type says how to treat the file.

 -- User Option: default-buffer-file-type
     This variable says how to handle files for which file-name-buffer-file-type-alist says nothing about the type.

     If this variable is non-nil, then these files are treated as binary: the coding system no-conversion is used.  Otherwise, nothing special is done for them---the coding system is deduced solely from the file contents, in the usual Emacs fashion.

Input Methods
=============

Input methods provide convenient ways of entering non-ASCII characters from the keyboard.  Unlike coding systems, which translate non-ASCII characters to and from encodings meant to be read by programs, input methods provide human-friendly commands.  (see Input Methods (The GNU Emacs Manual), for information on how users use input methods to enter text.)  How to define input methods is not yet documented in this manual, but here we describe how to use them.

Each input method has a name, which is currently a string; in the future, symbols may also be usable as input method names.

 -- Variable: current-input-method
     This variable holds the name of the input method now active in the current buffer.  (It automatically becomes local in each buffer when set in any fashion.)  It is nil if no input method is active in the buffer now.

 -- Variable: default-input-method
     This variable holds the default input method for commands that choose an input method.  Unlike current-input-method, this variable is normally global.

 -- Function: set-input-method input-method
     This function activates input method INPUT-METHOD for the current buffer.  It also sets default-input-method to INPUT-METHOD. If INPUT-METHOD is nil, this function deactivates any input method for the current buffer.

 -- Function: read-input-method-name prompt &optional default inhibit-null
     This function reads an input method name with the minibuffer, prompting with PROMPT.  If DEFAULT is non-nil, that is returned by default, if the user enters empty input.  However, if INHIBIT-NULL is non-nil, empty input signals an error.

     The returned value is a string.

 -- Variable: input-method-alist
     This variable defines all the supported input methods. Each element defines one input method, and should have the form:

          (INPUT-METHOD LANGUAGE-ENV ACTIVATE-FUNC
           TITLE DESCRIPTION ARGS...)

     Here INPUT-METHOD is the input method name, a string; LANGUAGE-ENV is another string, the name of the language environment this input method is recommended for.  (That serves only for documentation purposes.)

     ACTIVATE-FUNC is a function to call to activate this method.  The ARGS, if any, are passed as arguments to ACTIVATE-FUNC.  All told, the arguments to ACTIVATE-FUNC are INPUT-METHOD and the ARGS.

     TITLE is a string to display in the mode line while this method is active.  DESCRIPTION is a string describing this method and what it is good for.

The fundamental interface to input methods is through the variable input-method-function.  See Reading One Event.

Locales
=======

POSIX defines a concept of ``locales'' which control which language to use in language-related features.  These Emacs variables control how Emacs interacts with these features.

 -- Variable: locale-coding-system
     This variable specifies the coding system to use for decoding system error messages, for encoding the format argument to format-time-string, and for decoding the return value of format-time-string.

 -- Variable: system-messages-locale
     This variable specifies the locale to use for generating system error messages.  Changing the locale can cause messages to come out in a different language or in a different orthography.  If the variable is nil, the locale is specified by environment variables in the usual POSIX fashion.

 -- Variable: system-time-locale
     This variable specifies the locale to use for formatting time values. Changing the locale can cause messages to appear according to the conventions of a different language.  If the variable is nil, the locale is specified by environment variables in the usual POSIX fashion.