International Character Set Support
***********************************

Emacs supports a wide variety of international character sets, including European and Vietnamese variants of the Latin alphabet, as well as Cyrillic, Devanagari (for Hindi and Marathi), Ethiopic, Greek, Han (for Chinese and Japanese), Hangul (for Korean), Hebrew, IPA, Kannada, Lao, Malayalam, Tamil, Thai, Tibetan, and Vietnamese scripts. Emacs also supports various encodings of these characters used by other internationalized software, such as word processors and mailers.

Emacs allows editing text with international characters by supporting all the related activities:

  * You can visit files with non-ASCII characters, save non-ASCII text, and pass non-ASCII text between Emacs and programs it invokes (such as compilers, spell-checkers, and mailers).  Setting your language environment (see Language Environments) takes care of setting up the coding systems and other options for a specific language or culture. Alternatively, you can specify how Emacs should encode or decode text for each command; see Text Coding.

  * You can display non-ASCII characters encoded by the various scripts.  This works by using appropriate fonts on graphics displays (see Defining Fontsets), and by sending special codes to text-only displays (see Terminal Coding).  If some characters are displayed incorrectly, refer to Undisplayable Characters, which describes possible problems and explains how to solve them.

  * You can insert non-ASCII characters or search for them.  To do that, you can specify an input method (see Select Input Method) suitable for your language, or use the default input method set up when you set your language environment.  If your keyboard can produce non-ASCII characters, you can select an appropriate keyboard coding system (see Terminal Coding), and Emacs will accept those characters.  Latin-1 characters can also be input by using the C-x 8 prefix, see Unibyte Mode.

     On X Window systems, your locale should be set to an appropriate value to make sure Emacs interprets keyboard input correctly; see locales.

The rest of this chapter describes these issues in detail.

Introduction to International Character Sets
============================================

The users of international character sets and scripts have established many more-or-less standard coding systems for storing files.  Emacs internally uses a single multibyte character encoding, so that it can intermix characters from all these scripts in a single buffer or string.  This encoding represents each non-ASCII character as a sequence of bytes in the range 0200 through 0377. Emacs translates between the multibyte character encoding and various other coding systems when reading and writing files, when exchanging data with subprocesses, and (in some cases) in the C-q command (see Multibyte Conversion).

The command C-h h (view-hello-file) displays the file etc/HELLO, which shows how to say ``hello'' in many languages. This illustrates various scripts.  If some characters can't be displayed on your terminal, they appear as ? or as hollow boxes (see Undisplayable Characters).

Keyboards, even in the countries where these character sets are used, generally don't have keys for all the characters in them.  So Emacs supports various input methods, typically one for each script or language, to make it convenient to type them.

The prefix key C-x RET is used for commands that pertain to multibyte characters, coding systems, and input methods.

Enabling Multibyte Characters
=============================

By default, Emacs starts in multibyte mode, because that allows you to use all the supported languages and scripts without limitations.

You can enable or disable multibyte character support, either for Emacs as a whole, or for a single buffer.  When multibyte characters are disabled in a buffer, we call that unibyte mode.  Then each byte in that buffer represents a character, even codes 0200 through 0377.

The old features for supporting the European character sets, ISO Latin-1 and ISO Latin-2, work in unibyte mode as they did in Emacs 19 and also work for the other ISO 8859 character sets.  However, there is no need to turn off multibyte character support to use ISO Latin; the Emacs multibyte character set includes all the characters in these character sets, and Emacs can translate automatically to and from the ISO codes.

To edit a particular file in unibyte representation, visit it using find-file-literally.  See Visiting.  To convert a buffer in multibyte representation into a single-byte representation of the same characters, the easiest way is to save the contents in a file, kill the buffer, and find the file again with find-file-literally.  You can also use C-x RET c (universal-coding-system-argument) and specify raw-text as the coding system with which to find or save a file.  See Text Coding.  Finding a file as raw-text doesn't disable format conversion, uncompression and auto mode selection as find-file-literally does.

To turn off multibyte character support by default, start Emacs with the --unibyte option (see Initial Options), or set the environment variable EMACS_UNIBYTE.  You can also customize enable-multibyte-characters or, equivalently, directly set the variable default-enable-multibyte-characters to nil in your init file to have basically the same effect as --unibyte.

To convert a unibyte session to a multibyte session, set default-enable-multibyte-characters to t.  Buffers which were created in the unibyte session before you turn on multibyte support will stay unibyte.  You can turn on multibyte support in a specific buffer by invoking the command toggle-enable-multibyte-characters in that buffer.

With --unibyte, multibyte strings are not created during initialization from the values of environment variables, /etc/passwd entries etc. that contain non-ASCII 8-bit characters.

Emacs normally loads Lisp files as multibyte, regardless of whether you used --unibyte.  This includes the Emacs initialization file, .emacs, and the initialization files of Emacs packages such as Gnus.  However, you can specify unibyte loading for a particular Lisp file, by putting -*-unibyte: t;-*- in a comment on the first line (see File Variables).  Then that file is always loaded as unibyte text, even if you did not start Emacs with --unibyte. The motivation for these conventions is that it is more reliable to always load any particular Lisp file in the same way.  However, you can load a Lisp file as unibyte, on any one occasion, by typing C-x RET c raw-text RET immediately before loading it.

The mode line indicates whether multibyte character support is enabled in the current buffer.  If it is, there are two or more characters (most often two dashes) before the colon near the beginning of the mode line. When multibyte characters are not enabled, nothing precedes the colon except a single dash.

Language Environments
=====================

All supported character sets are supported in Emacs buffers whenever multibyte characters are enabled; there is no need to select a particular language in order to display its characters in an Emacs buffer.  However, it is important to select a language environment in order to set various defaults.  The language environment really represents a choice of preferred script (more or less) rather than a choice of language.

The language environment controls which coding systems to recognize when reading text (see Recognize Coding).  This applies to files, incoming mail, netnews, and any other text you read into Emacs.  It may also specify the default coding system to use when you create a file. Each language environment also specifies a default input method.

To select a language environment, you can customize the variable current-language-environment or use the command M-x set-language-environment.  It makes no difference which buffer is current when you use this command, because the effects apply globally to the Emacs session.  The supported language environments include:

     Belarusian, Brazilian Portuguese, Bulgarian, Chinese-BIG5, Chinese-CNS, Chinese-EUC-TW, Chinese-GB, Croatian, Cyrillic-ALT, Cyrillic-ISO, Cyrillic-KOI8, Czech, Devanagari, Dutch, English, Ethiopic, French, Georgian, German, Greek, Hebrew, IPA, Italian, Japanese, Kannada, Korean, Lao, Latin-1, Latin-2, Latin-3, Latin-4, Latin-5, Latin-6, Latin-7, Latin-8 (Celtic), Latin-9 (updated Latin-1 with the Euro sign), Latvian, Lithuanian, Malayalam, Polish, Romanian, Russian, Slovak, Slovenian, Spanish, Swedish, Tajik, Tamil, Thai, Tibetan, Turkish, UTF-8 (for a setup which prefers Unicode characters and files encoded in UTF-8), Ukrainian, Vietnamese, Welsh, and Windows-1255 (for a setup which prefers Cyrillic characters and files encoded in Windows-1255).

To display the script(s) used by your language environment on a graphical display, you need to have a suitable font.  If some of the characters appear as empty boxes, you should install the GNU Intlfonts
package, which includes fonts for most supported scripts. (If
you run Emacs on X, you need to inform the X server about the location
of the newly installed fonts with the following commands:)

      xset fp+ /usr/local/share/emacs/fonts
      xset fp rehash

See Fontsets, for more details about setting up your fonts.

Some operating systems let you specify the character-set locale you are using by setting the locale environment variables LC_ALL, LC_CTYPE, or LANG. (If more than one of these is set, the first one that is nonempty specifies your locale for this purpose.)  During startup, Emacs looks up your character-set locale's name in the system locale alias table, matches its canonical name against entries in the value of the variables locale-charset-language-names and locale-language-names, and selects the corresponding language environment if a match is found. (The former variable overrides the latter.)  It also adjusts the display table and terminal coding system, the locale coding system, the preferred coding system as needed for the locale, and---last but not least---the way Emacs decodes non-ASCII characters sent by your keyboard.

If you modify the LC_ALL, LC_CTYPE, or LANG environment variables while running Emacs, you may want to invoke the set-locale-environment function afterwards to readjust the language environment from the new locale.

The set-locale-environment function normally uses the preferred coding system established by the language environment to decode system messages.  But if your locale matches an entry in the variable locale-preferred-coding-systems, Emacs uses the corresponding coding system instead.  For example, if the locale ja_JP.PCK matches japanese-shift-jis in locale-preferred-coding-systems, Emacs uses that encoding even though it might normally use japanese-iso-8bit.

You can override the language environment chosen at startup with explicit use of the command set-language-environment, or with customization of current-language-environment in your init file.

To display information about the effects of a certain language environment LANG-ENV, use the command C-h L LANG-ENV RET (describe-language-environment).  This tells you which languages this language environment is useful for, and lists the character sets, coding systems, and input methods that go with it.  It also shows some sample text to illustrate scripts used in this language environment.  If you give an empty input for LANG-ENV, this command describes the chosen language environment.

You can customize any language environment with the normal hook set-language-environment-hook.  The command set-language-environment runs that hook after setting up the new language environment.  The hook functions can test for a specific language environment by checking the variable current-language-environment.  This hook is where you should put non-default settings for specific language environment, such as coding systems for keyboard input and terminal output, the default input method, etc.

Before it starts to set up the new language environment, set-language-environment first runs the hook exit-language-environment-hook.  This hook is useful for undoing customizations that were made with set-language-environment-hook. For instance, if you set up a special key binding in a specific language environment using set-language-environment-hook, you should set up exit-language-environment-hook to restore the normal binding for that key.

Input Methods
=============

An input method is a kind of character conversion designed specifically for interactive input.  In Emacs, typically each language has its own input method; sometimes several languages which use the same characters can share one input method.  A few languages support several input methods.

The simplest kind of input method works by mapping ASCII letters into another alphabet; this allows you to use one other alphabet instead of ASCII.  The Greek and Russian input methods work this way.

A more powerful technique is composition: converting sequences of characters into one letter.  Many European input methods use composition to produce a single non-ASCII letter from a sequence that consists of a letter followed by accent characters (or vice versa).  For example, some methods convert the sequence a' into a single accented letter. These input methods have no special commands of their own; all they do is compose sequences of printing characters.

The input methods for syllabic scripts typically use mapping followed by composition.  The input methods for Thai and Korean work this way. First, letters are mapped into symbols for particular sounds or tone marks; then, sequences of these which make up a whole syllable are mapped into one syllable sign.

Chinese and Japanese require more complex methods.  In Chinese input methods, first you enter the phonetic spelling of a Chinese word (in input method chinese-py, among others), or a sequence of portions of the character (input methods chinese-4corner and chinese-sw, and others).  One input sequence typically corresponds to many possible Chinese characters.  You select the one you mean using keys such as C-f, C-b, C-n, C-p, and digits, which have special meanings in this situation.

The possible characters are conceptually arranged in several rows, with each row holding up to 10 alternatives.  Normally, Emacs displays just one row at a time, in the echo area; (I/J) appears at the beginning, to indicate that this is the Ith row out of a total of J rows.  Type C-n or C-p to display the next row or the previous row.

Type C-f and C-b to move forward and backward among the alternatives in the current row.  As you do this, Emacs highlights the current alternative with a special color; type C-SPC to select the current alternative and use it as input.  The alternatives in the row are also numbered; the number appears before the alternative.  Typing a digit N selects the Nth alternative of the current row and uses it as input.

TAB in these Chinese input methods displays a buffer showing all the possible characters at once; then clicking Mouse-2 on one of them selects that alternative.  The keys C-f, C-b, C-n, C-p, and digits continue to work as usual, but they do the highlighting in the buffer showing the possible characters, rather than in the echo area.

In Japanese input methods, first you input a whole word using phonetic spelling; then, after the word is in the buffer, Emacs converts it into one or more characters using a large dictionary.  One phonetic spelling corresponds to a number of different Japanese words; to select one of them, use C-n and C-p to cycle through the alternatives.

Sometimes it is useful to cut off input method processing so that the characters you have just entered will not combine with subsequent characters.  For example, in input method latin-1-postfix, the sequence e ' combines to form an e with an accent.  What if you want to enter them as separate characters?

One way is to type the accent twice; this is a special feature for entering the separate letter and accent.  For example, e ' ' gives you the two characters e'.  Another way is to type another letter after the e---something that won't combine with that---and immediately delete it.  For example, you could type e e DEL ' to get separate e and '.

Another method, more general but not quite as easy to type, is to use C-\ C-\ between two characters to stop them from combining.  This is the command C-\ (toggle-input-method) used twice. See Select Input Method.

C-\ C-\ is especially useful inside an incremental search, because it stops waiting for more characters to combine, and starts searching for what you have already entered.

To find out how to input the character after point using the current input method, type C-u C-x =.  See Position Info.

The variables input-method-highlight-flag and input-method-verbose-flag control how input methods explain what is happening.  If input-method-highlight-flag is non-nil, the partial sequence is highlighted in the buffer (for most input methods---some disable this feature).  If input-method-verbose-flag is non-nil, the list of possible characters to type next is displayed in the echo area (but not when you are in the minibuffer).

Selecting an Input Method
=========================

C-\
     Enable or disable use of the selected input method.

C-x RET C-\ METHOD RET
     Select a new input method for the current buffer.

C-h I METHOD RET
C-h C-\ METHOD RET
     Describe the input method METHOD (describe-input-method). By default, it describes the current input method (if any).  This description should give you the full details of how to use any particular input method.

M-x list-input-methods
     Display a list of all the supported input methods.

To choose an input method for the current buffer, use C-x RET C-\ (set-input-method).  This command reads the input method name from the minibuffer; the name normally starts with the language environment that it is meant to be used with.  The variable current-input-method records which input method is selected.

Input methods use various sequences of ASCII characters to stand for non-ASCII characters.  Sometimes it is useful to turn off the input method temporarily.  To do this, type C-\ (toggle-input-method).  To reenable the input method, type C-\ again.

If you type C-\ and you have not yet selected an input method, it prompts for you to specify one.  This has the same effect as using C-x RET C-\ to specify an input method.

When invoked with a numeric argument, as in C-u C-\, toggle-input-method always prompts you for an input method, suggesting the most recently selected one as the default.

Selecting a language environment specifies a default input method for use in various buffers.  When you have a default input method, you can select it in the current buffer by typing C-\.  The variable default-input-method specifies the default input method (nil means there is none).

In some language environments, which support several different input methods, you might want to use an input method different from the default chosen by set-language-environment.  You can instruct Emacs to select a different default input method for a certain language environment, if you wish, by using set-language-environment-hook (see set-language-environment-hook).  For example:

     (defun my-chinese-setup ()
       "Set up my private Chinese environment."
       (if (equal current-language-environment "Chinese-GB")
           (setq default-input-method "chinese-tonepy")))
     (add-hook 'set-language-environment-hook 'my-chinese-setup)

This sets the default input method to be chinese-tonepy whenever you choose a Chinese-GB language environment.

Some input methods for alphabetic scripts work by (in effect) remapping the keyboard to emulate various keyboard layouts commonly used for those scripts.  How to do this remapping properly depends on your actual keyboard layout.  To specify which layout your keyboard has, use the command M-x quail-set-keyboard-layout.

You can use the command M-x quail-show-key to show what key (or key sequence) to type in order to input the character following point, using the selected keyboard layout.  The command C-u C-x = also shows that information in addition to the other information about the character.

To display a list of all the supported input methods, type M-x list-input-methods.  The list gives information about each input method, including the string that stands for it in the mode line.

Unibyte and Multibyte Non-ASCII characters
==========================================

When multibyte characters are enabled, character codes 0240 (octal) through 0377 (octal) are not really legitimate in the buffer.  The valid non-ASCII printing characters have codes that start from 0400.

If you type a self-inserting character in the range 0240 through 0377, or if you use C-q to insert one, Emacs assumes you intended to use one of the ISO Latin-N character sets, and converts it to the Emacs code representing that Latin-N character.  You select which ISO Latin character set to use through your choice of language environment (see Language Environments). If you do not specify a choice, the default is Latin-1.

If you insert a character in the range 0200 through 0237, which forms the eight-bit-control character set, it is inserted literally.  You should normally avoid doing this since buffers containing such characters have to be written out in either the emacs-mule or raw-text coding system, which is usually not what you want.

Coding Systems
==============

Users of various languages have established many more-or-less standard coding systems for representing them.  Emacs does not use these coding systems internally; instead, it converts from various coding systems to its own system when reading data, and converts the internal coding system to other coding systems when writing data.  Conversion is possible in reading or writing files, in sending or receiving from the terminal, and in exchanging data with subprocesses.

Emacs assigns a name to each coding system.  Most coding systems are used for one language, and the name of the coding system starts with the language name.  Some coding systems are used for several languages; their names usually start with iso.  There are also special coding systems no-conversion, raw-text and emacs-mule which do not convert printing characters at all.

A special class of coding systems, collectively known as codepages, is designed to support text encoded by MS-Windows and MS-DOS software.  The names of these coding systems are cpNNNN, where NNNN is a 3- or 4-digit number of the codepage.  You can use these encodings just like any other coding system; for example, to visit a file encoded in codepage 850, type C-x RET c cp850 RET C-x C-f FILENAME RET (In the MS-DOS port of Emacs, you need to create a cpNNN coding system with M-x codepage-setup, before you can use it. See MS-DOS and MULE (Specialized Emacs Features).).

In addition to converting various representations of non-ASCII characters, a coding system can perform end-of-line conversion.  Emacs handles three different conventions for how to separate lines in a file: newline, carriage-return linefeed, and just carriage-return.

C-h C CODING RET
     Describe coding system CODING.

C-h C RET
     Describe the coding systems currently in use.

M-x list-coding-systems
     Display a list of all the supported coding systems.

The command C-h C (describe-coding-system) displays information about particular coding systems.  You can specify a coding system name as the argument; alternatively, with an empty argument, it describes the coding systems currently selected for various purposes, both in the current buffer and as the defaults, and the priority list for recognizing coding systems (see Recognize Coding).

To display a list of all the supported coding systems, type M-x list-coding-systems.  The list gives information about each coding system, including the letter that stands for it in the mode line (see Mode Line).

Each of the coding systems that appear in this list---except for no-conversion, which means no conversion of any kind---specifies how and whether to convert printing characters, but leaves the choice of end-of-line conversion to be decided based on the contents of each file. For example, if the file appears to use the sequence carriage-return linefeed to separate lines, DOS end-of-line conversion will be used.

Each of the listed coding systems has three variants which specify exactly what to do for end-of-line conversion:

...-unix
     Don't do any end-of-line conversion; assume the file uses newline to separate lines.  (This is the convention normally used on Unix and GNU systems.)

...-dos
     Assume the file uses carriage-return linefeed to separate lines, and do the appropriate conversion.  (This is the convention normally used on Microsoft systems. (It is also specified for MIME text/* bodies and in other network transport contexts.  It is different from the SGML reference syntax record-start/record-end format which Emacs doesn't support directly.))

...-mac
     Assume the file uses carriage-return to separate lines, and do the appropriate conversion.  (This is the convention normally used on the Macintosh system.)

These variant coding systems are omitted from the list-coding-systems display for brevity, since they are entirely predictable.  For example, the coding system iso-latin-1 has variants iso-latin-1-unix, iso-latin-1-dos and iso-latin-1-mac.

The coding system raw-text is good for a file which is mainly ASCII text, but may contain byte values above 127 which are not meant to encode non-ASCII characters.  With raw-text, Emacs copies those byte values unchanged, and sets enable-multibyte-characters to nil in the current buffer so that they will be interpreted properly.  raw-text handles end-of-line conversion in the usual way, based on the data encountered, and has the usual three variants to specify the kind of end-of-line conversion to use.

In contrast, the coding system no-conversion specifies no character code conversion at all---none for non-ASCII byte values and none for end of line.  This is useful for reading or writing binary files, tar files, and other files that must be examined verbatim.  It, too, sets enable-multibyte-characters to nil.

The easiest way to edit a file with no conversion of any kind is with the M-x find-file-literally command.  This uses no-conversion, and also suppresses other Emacs features that might convert the file contents before you see them.  See Visiting.

The coding system emacs-mule means that the file contains non-ASCII characters stored with the internal Emacs encoding.  It handles end-of-line conversion based on the data encountered, and has the usual three variants to specify the kind of end-of-line conversion.

Recognizing Coding Systems
==========================

Emacs tries to recognize which coding system to use for a given text as an integral part of reading that text.  (This applies to files being read, output from subprocesses, text from X selections, etc.) Emacs can select the right coding system automatically most of the time---once you have specified your preferences.

Some coding systems can be recognized or distinguished by which byte sequences appear in the data.  However, there are coding systems that cannot be distinguished, not even potentially.  For example, there is no way to distinguish between Latin-1 and Latin-2; they use the same byte values with different meanings.

Emacs handles this situation by means of a priority list of coding systems.  Whenever Emacs reads a file, if you do not specify the coding system to use, Emacs checks the data against each coding system, starting with the first in priority and working down the list, until it finds a coding system that fits the data.  Then it converts the file contents assuming that they are represented in this coding system.

The priority list of coding systems depends on the selected language environment (see Language Environments).  For example, if you use French, you probably want Emacs to prefer Latin-1 to Latin-2; if you use Czech, you probably want Latin-2 to be preferred.  This is one of the reasons to specify a language environment.

However, you can alter the coding system priority list in detail with the command M-x prefer-coding-system.  This command reads the name of a coding system from the minibuffer, and adds it to the front of the priority list, so that it is preferred to all others.  If you use this command several times, each use adds one element to the front of the priority list.

If you use a coding system that specifies the end-of-line conversion type, such as iso-8859-1-dos, what this means is that Emacs should attempt to recognize iso-8859-1 with priority, and should use DOS end-of-line conversion when it does recognize iso-8859-1.

Sometimes a file name indicates which coding system to use for the file.  The variable file-coding-system-alist specifies this correspondence.  There is a special function modify-coding-system-alist for adding elements to this list.  For example, to read and write all .txt files using the coding system china-iso-8bit, you can execute this Lisp expression:

     (modify-coding-system-alist 'file "\\.txt\\'" 'chinese-iso-8bit)

The first argument should be file, the second argument should be a regular expression that determines which files this applies to, and the third argument says which coding system to use for these files.

Emacs recognizes which kind of end-of-line conversion to use based on the contents of the file: if it sees only carriage-returns, or only carriage-return linefeed sequences, then it chooses the end-of-line conversion accordingly.  You can inhibit the automatic use of end-of-line conversion by setting the variable inhibit-eol-conversion to non-nil.  If you do that, DOS-style files will be displayed with the ^M characters visible in the buffer; some people prefer this to the more subtle (DOS) end-of-line type indication near the left edge of the mode line (see eol-mnemonic).

By default, the automatic detection of coding system is sensitive to escape sequences.  If Emacs sees a sequence of characters that begin with an escape character, and the sequence is valid as an ISO-2022 code, that tells Emacs to use one of the ISO-2022 encodings to decode the file.

However, there may be cases that you want to read escape sequences in a file as is.  In such a case, you can set the variable inhibit-iso-escape-detection to non-nil.  Then the code detection ignores any escape sequences, and never uses an ISO-2022 encoding.  The result is that all escape sequences become visible in the buffer.

The default value of inhibit-iso-escape-detection is nil.  We recommend that you not change it permanently, only for one specific operation.  That's because many Emacs Lisp source files in the Emacs distribution contain non-ASCII characters encoded in the coding system iso-2022-7bit, and they won't be decoded correctly when you visit those files if you suppress the escape sequence detection.

You can specify the coding system for a particular file using the -*-...-*- construct at the beginning of a file, or a local variables list at the end (see File Variables).  You do this by defining a value for the ``variable'' named coding.  Emacs does not really have a variable coding; instead of setting a variable, this uses the specified coding system for the file.  For example, -*-mode: C; coding: latin-1;-*- specifies use of the Latin-1 coding system, as well as C mode.  When you specify the coding explicitly in the file, that overrides file-coding-system-alist.

The variables auto-coding-alist, auto-coding-regexp-alist and auto-coding-functions are the strongest way to specify the coding system for certain patterns of file names, or for files containing certain patterns; these variables even override -*-coding:-*- tags in the file itself.  Emacs uses auto-coding-alist for tar and archive files, to prevent it from being confused by a -*-coding:-*- tag in a member of the archive and thinking it applies to the archive file as a whole. Likewise, Emacs uses auto-coding-regexp-alist to ensure that RMAIL files, whose names in general don't match any particular pattern, are decoded correctly.  One of the builtin auto-coding-functions detects the encoding for XML files.

If Emacs recognizes the encoding of a file incorrectly, you can reread the file using the correct coding system by typing C-x RET r CODING-SYSTEM RET.  To see what coding system Emacs actually used to decode the file, look at the coding system mnemonic letter near the left edge of the mode line (see Mode Line), or type C-h C RET.

The command unify-8859-on-decoding-mode enables a mode that ``unifies'' the Latin alphabets when decoding text.  This works by converting all non-ASCII Latin-N characters to either Latin-1 or Unicode characters.  This way it is easier to use various Latin-N alphabets together.  In a future Emacs version we hope to move towards full Unicode support and complete unification of character sets.

Once Emacs has chosen a coding system for a buffer, it stores that coding system in buffer-file-coding-system and uses that coding system, by default, for operations that write from this buffer into a file.  This includes the commands save-buffer and write-region.  If you want to write files from this buffer using a different coding system, you can specify a different coding system for the buffer using set-buffer-file-coding-system (see Text Coding).

You can insert any possible character into any Emacs buffer, but most coding systems can only handle some of the possible characters. This means that it is possible for you to insert characters that cannot be encoded with the coding system that will be used to save the buffer.  For example, you could start with an ASCII file and insert a few Latin-1 characters into it, or you could edit a text file in Polish encoded in iso-8859-2 and add some Russian words to it. When you save the buffer, Emacs cannot use the current value of buffer-file-coding-system, because the characters you added cannot be encoded by that coding system.

When that happens, Emacs tries the most-preferred coding system (set by M-x prefer-coding-system or M-x set-language-environment), and if that coding system can safely encode all of the characters in the buffer, Emacs uses it, and stores its value in buffer-file-coding-system.  Otherwise, Emacs displays a list of coding systems suitable for encoding the buffer's contents, and asks you to choose one of those coding systems.

If you insert the unsuitable characters in a mail message, Emacs behaves a bit differently.  It additionally checks whether the most-preferred coding system is recommended for use in MIME messages; if not, Emacs tells you that the most-preferred coding system is not recommended and prompts you for another coding system.  This is so you won't inadvertently send a message encoded in a way that your recipient's mail software will have difficulty decoding.  (If you do want to use the most-preferred coding system, you can still type its name in response to the question.)

When you send a message with Mail mode (see Sending Mail), Emacs has four different ways to determine the coding system to use for encoding the message text.  It tries the buffer's own value of buffer-file-coding-system, if that is non-nil.  Otherwise, it uses the value of sendmail-coding-system, if that is non-nil.  The third way is to use the default coding system for new files, which is controlled by your choice of language environment, if that is non-nil.  If all of these three values are nil, Emacs encodes outgoing mail using the Latin-1 coding system.

When you get new mail in Rmail, each message is translated automatically from the coding system it is written in, as if it were a separate file.  This uses the priority list of coding systems that you have specified.  If a MIME message specifies a character set, Rmail obeys that specification, unless rmail-decode-mime-charset is nil.

For reading and saving Rmail files themselves, Emacs uses the coding system specified by the variable rmail-file-coding-system.  The default value is nil, which means that Rmail files are not translated (they are read and written in the Emacs internal character code).

Specifying a Coding System for File Text
========================================

In cases where Emacs does not automatically choose the right coding system for a file's contents, you can use these commands to specify one:

C-x RET f CODING RET
     Use coding system CODING for saving or revisiting the visited file in the current buffer.

C-x RET c CODING RET
     Specify coding system CODING for the immediately following command.

C-x RET r CODING RET
     Revisit the current file using the coding system CODING.

M-x recode-region RET RIGHT RET WRONG RET
     Convert a region that was decoded using coding system WRONG, decoding it using coding system RIGHT instead.

The command C-x RET f (set-buffer-file-coding-system) sets the file coding system for the current buffer---in other words, it says which coding system to use when saving or reverting the visited file.  You specify which coding system using the minibuffer.  If you specify a coding system that cannot handle all of the characters in the buffer, Emacs warns you about the troublesome characters when you actually save the buffer.

Another way to specify the coding system for a file is when you visit the file.  First use the command C-x RET c (universal-coding-system-argument); this command uses the minibuffer to read a coding system name.  After you exit the minibuffer, the specified coding system is used for the immediately following command.

So if the immediately following command is C-x C-f, for example, it reads the file using that coding system (and records the coding system for when you later save the file).  Or if the immediately following command is C-x C-w, it writes the file using that coding system. When you specify the coding system for saving in this way, instead of with C-x RET f, there is no warning if the buffer contains characters that the coding system cannot handle.

Other file commands affected by a specified coding system include C-x C-i and C-x C-v, as well as the other-window variants of C-x C-f.  C-x RET c also affects commands that start subprocesses, including M-x shell (see Shell).  If the immediately following command does not use the coding system, then C-x RET c ultimately has no effect.

An easy way to visit a file with no conversion is with the M-x find-file-literally command.  See Visiting.

The variable default-buffer-file-coding-system specifies the choice of coding system to use when you create a new file.  It applies when you find a new file, and when you create a buffer and then save it in a file.  Selecting a language environment typically sets this variable to a good choice of default coding system for that language environment.

If you visit a file with a wrong coding system, you can correct this with C-x RET r (revert-buffer-with-coding-system). This visits the current file again, using a coding system you specify.

If a piece of text has already been inserted into a buffer using the wrong coding system, you can redo the decoding of it using M-x recode-region.  This prompts you for the proper coding system, then for the wrong coding system that was actually used, and does the conversion.  It first encodes the region using the wrong coding system, then decodes it again using the proper coding system.

Coding Systems for Interprocess Communication
=============================================

This section explains how to specify coding systems for use in communication with other processes.

C-x RET x CODING RET
     Use coding system CODING for transferring selections to and from other window-based applications.

C-x RET X CODING RET
     Use coding system CODING for transferring one selection---the next one---to or from another window-based application.

C-x RET p INPUT-CODING RET OUTPUT-CODING RET
     Use coding systems INPUT-CODING and OUTPUT-CODING for subprocess input and output in the current buffer.

C-x RET c CODING RET
     Specify coding system CODING for the immediately following command.

The command C-x RET x (set-selection-coding-system) specifies the coding system for sending selected text to other windowing applications, and for receiving the text of selections made in other applications.  This command applies to all subsequent selections, until you override it by using the command again.  The command C-x RET X (set-next-selection-coding-system) specifies the coding system for the next selection made in Emacs or read by Emacs.

The command C-x RET p (set-buffer-process-coding-system) specifies the coding system for input and output to a subprocess.  This command applies to the current buffer; normally, each subprocess has its own buffer, and thus you can use this command to specify translation to and from a particular subprocess by giving the command in the corresponding buffer.

You can also use C-x RET c just before the command that runs or starts a subprocess, to specify the coding system to use for communication with that subprocess.

The default for translation of process input and output depends on the current language environment.

The variable locale-coding-system specifies a coding system to use when encoding and decoding system strings such as system error messages and format-time-string formats and time stamps.  That coding system is also used for decoding non-ASCII keyboard input on X Window systems.  You should choose a coding system that is compatible with the underlying system's text representation, which is normally specified by one of the environment variables LC_ALL, LC_CTYPE, and LANG.  (The first one, in the order specified above, whose value is nonempty is the one that determines the text representation.)

Coding Systems for File Names
=============================

C-x RET F CODING RET
     Use coding system CODING for encoding and decoding file names.

The variable file-name-coding-system specifies a coding system to use for encoding file names.  It has no effect on reading and writing the contents of files.

If you set the variable to a coding system name (as a Lisp symbol or a string), Emacs encodes file names using that coding system for all file operations.  This makes it possible to use non-ASCII characters in file names---or, at least, those non-ASCII characters which the specified coding system can encode.  Use C-x RET F (set-file-name-coding-system) to specify this interactively.

If file-name-coding-system is nil, Emacs uses a default coding system determined by the selected language environment. In the default language environment, any non-ASCII characters in file names are not encoded specially; they appear in the file system using the internal Emacs representation.

Warning: if you change file-name-coding-system (or the language environment) in the middle of an Emacs session, problems can result if you have already visited files whose names were encoded using the earlier coding system and cannot be encoded (or are encoded differently) under the new coding system.  If you try to save one of these buffers under the visited file name, saving may use the wrong file name, or it may get an error.  If such a problem happens, use C-x C-w to specify a new file name for that buffer.

If a mistake occurs when encoding a file name, use the command M-x recode-file-name to change the file name's coding system.  This prompts for an existing file name, its old coding system, and the coding system to which you wish to convert.

Coding Systems for Terminal I/O
===============================

C-x RET k CODING RET
     Use coding system CODING for keyboard input.

C-x RET t CODING RET
     Use coding system CODING for terminal output.

The command C-x RET t (set-terminal-coding-system) specifies the coding system for terminal output.  If you specify a character code for terminal output, all characters output to the terminal are translated into that coding system.

This feature is useful for certain character-only terminals built to support specific languages or character sets---for example, European terminals that support one of the ISO Latin character sets.  You need to specify the terminal coding system when using multibyte text, so that Emacs knows which characters the terminal can actually handle.

By default, output to the terminal is not translated at all, unless Emacs can deduce the proper coding system from your terminal type or your locale specification (see Language Environments).

The command C-x RET k (set-keyboard-coding-system) or the variable keyboard-coding-system specifies the coding system for keyboard input.  Character-code translation of keyboard input is useful for terminals with keys that send non-ASCII graphic characters---for example, some terminals designed for ISO Latin-1 or subsets of it.

By default, keyboard input is translated based on your system locale setting.  If your terminal does not really support the encoding implied by your locale (for example, if you find it inserts a non-ASCII character if you type M-i), you will need to set keyboard-coding-system to nil to turn off encoding. You can do this by putting

     (set-keyboard-coding-system nil)

in your ~/.emacs file.

There is a similarity between using a coding system translation for keyboard input, and using an input method: both define sequences of keyboard input that translate into single characters.  However, input methods are designed to be convenient for interactive use by humans, and the sequences that are translated are typically sequences of ASCII printing characters.  Coding systems typically translate sequences of non-graphic characters.

Fontsets
========

A font typically defines shapes for a single alphabet or script. Therefore, displaying the entire range of scripts that Emacs supports requires a collection of many fonts.  In Emacs, such a collection is called a fontset.  A fontset is defined by a list of fonts, each assigned to handle a range of character codes.

Each fontset has a name, like a font.  However, while fonts are stored in the system and the available font names are defined by the system, fontsets are defined within Emacs itself.  Once you have defined a fontset, you can use it within Emacs by specifying its name, anywhere that you could use a single font.  Of course, Emacs fontsets can use only the fonts that the system supports; if certain characters appear on the screen as hollow boxes, this means that the fontset in use for them has no font for those characters. (The Emacs installation instructions have information on additional font support.)

Emacs creates two fontsets automatically: the standard fontset and the startup fontset.  The standard fontset is most likely to have fonts for a wide variety of non-ASCII characters; however, this is not the default for Emacs to use.  (By default, Emacs tries to find a font that has bold and italic variants.)  You can specify use of the standard fontset with the -fn option.  For example,

     emacs -fn fontset-standard

You can also specify a fontset with the Font resource (see X Resources).

A fontset does not necessarily specify a font for every character code.  If a fontset specifies no font for a certain character, or if it specifies a font that does not exist on your system, then it cannot display that character properly.  It will display that character as an empty box instead.

Defining fontsets
=================

Emacs creates a standard fontset automatically according to the value of standard-fontset-spec.  This fontset's name is

     -*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard

or just fontset-standard for short.

Bold, italic, and bold-italic variants of the standard fontset are created automatically.  Their names have bold instead of medium, or i instead of r, or both.

If you specify a default ASCII font with the Font resource or the -fn argument, Emacs generates a fontset from it automatically.  This is the startup fontset and its name is fontset-startup.  It does this by replacing the FOUNDRY, FAMILY, ADD_STYLE, and AVERAGE_WIDTH fields of the font name with *, replacing CHARSET_REGISTRY field with fontset, and replacing CHARSET_ENCODING field with startup, then using the resulting string to specify a fontset.

For instance, if you start Emacs this way,

     emacs -fn "*courier-medium-r-normal--14-140-*-iso8859-1"

Emacs generates the following fontset and uses it for the initial X window frame:

     -*-*-medium-r-normal-*-14-140-*-*-*-*-fontset-startup

With the X resource Emacs.Font, you can specify a fontset name just like an actual font name.  But be careful not to specify a fontset name in a wildcard resource like Emacs*Font---that wildcard specification matches various other resources, such as for menus, and menus cannot handle fontsets.

You can specify additional fontsets using X resources named Fontset-N, where N is an integer starting from 0. The resource value should have this form:

     FONTPATTERN, [CHARSETNAME:FONTNAME]...

FONTPATTERN should have the form of a standard X font name, except for the last two fields.  They should have the form fontset-ALIAS.

The fontset has two names, one long and one short.  The long name is FONTPATTERN.  The short name is fontset-ALIAS.  You can refer to the fontset by either name.

The construct CHARSET:FONT specifies which font to use (in this fontset) for one particular character set.  Here, CHARSET is the name of a character set, and FONT is the font to use for that character set.  You can use this construct any number of times in defining one fontset.

For the other character sets, Emacs chooses a font based on FONTPATTERN.  It replaces fontset-ALIAS with values that describe the character set.  For the ASCII character font, fontset-ALIAS is replaced with ISO8859-1.

In addition, when several consecutive fields are wildcards, Emacs collapses them into a single wildcard.  This is to prevent use of auto-scaled fonts.  Fonts made by scaling larger fonts are not usable for editing, and scaling a smaller font is not useful because it is better to use the smaller font in its own size, which is what Emacs does.

Thus if FONTPATTERN is this,

     -*-fixed-medium-r-normal-*-24-*-*-*-*-*-fontset-24

the font specification for ASCII characters would be this:

     -*-fixed-medium-r-normal-*-24-*-ISO8859-1

and the font specification for Chinese GB2312 characters would be this:

     -*-fixed-medium-r-normal-*-24-*-gb2312*-*

You may not have any Chinese font matching the above font specification.  Most X distributions include only Chinese fonts that have song ti or fangsong ti in FAMILY field.  In such a case, Fontset-N can be specified as below:

     Emacs.Fontset-0: -*-fixed-medium-r-normal-*-24-*-*-*-*-*-fontset-24,\
             chinese-gb2312:-*-*-medium-r-normal-*-24-*-gb2312*-*

Then, the font specifications for all but Chinese GB2312 characters have fixed in the FAMILY field, and the font specification for Chinese GB2312 characters has a wild card * in the FAMILY field.

The function that processes the fontset resource value to create the fontset is called create-fontset-from-fontset-spec.  You can also call this function explicitly to create a fontset.

See Font X, for more information about font naming in X.

Undisplayable Characters
========================

There may be a some non-ASCII characters that your terminal cannot display.  Most text-only terminals support just a single character set (use the variable default-terminal-coding-system (see Terminal Coding) to tell Emacs which one); characters which can't be encoded in that coding system are displayed as ? by default.

Graphical displays can display a broader range of characters, but you may not have fonts installed for all of them; characters that have no font appear as a hollow box.

If you use Latin-1 characters but your terminal can't display Latin-1, you can arrange to display mnemonic ASCII sequences instead, e.g. "o for o-umlaut.  Load the library iso-ascii to do this.

If your terminal can display Latin-1, you can display characters from other European character sets using a mixture of equivalent Latin-1 characters and ASCII mnemonics.  Customize the variable latin1-display to enable this.  The mnemonic ASCII sequences mostly correspond to those of the prefix input methods.

Unibyte Editing Mode
====================

The ISO 8859 Latin-N character sets define character codes in the range 0240 to 0377 octal (160 to 255 decimal) to handle the accented letters and punctuation needed by various European languages (and some non-European ones).  If you disable multibyte characters, Emacs can still handle one of these character codes at a time. To specify which of these codes to use, invoke M-x set-language-environment and specify a suitable language environment such as Latin-N.

For more information about unibyte operation, see Enabling Multibyte.  Note particularly that you probably want to ensure that your initialization files are read as unibyte if they contain non-ASCII characters.

Emacs can also display those characters, provided the terminal or font in use supports them.  This works automatically.  Alternatively, on a graphical display, Emacs can also display single-byte characters through fontsets, in effect by displaying the equivalent multibyte characters according to the current language environment.  To request this, set the variable unibyte-display-via-language-environment to a non-nil value.

If your terminal does not support display of the Latin-1 character set, Emacs can display these characters as ASCII sequences which at least give you a clear idea of what the characters are.  To do this, load the library iso-ascii.  Similar libraries for other Latin-N character sets could be implemented, but we don't have them yet.

Normally non-ISO-8859 characters (decimal codes between 128 and 159 inclusive) are displayed as octal escapes.  You can change this for non-standard ``extended'' versions of ISO-8859 character sets by using the function standard-display-8bit in the disp-table library.

There are two ways to input single-byte non-ASCII characters:

  * You can use an input method for the selected language environment. See Input Methods.  When you use an input method in a unibyte buffer, the non-ASCII character you specify with it is converted to unibyte.

  * If your keyboard can generate character codes 128 (decimal) and up, representing non-ASCII characters, you can type those character codes directly.

     On a graphical display, you should not need to do anything special to use these keys; they should simply work.  On a text-only terminal, you should use the command M-x set-keyboard-coding-system or the variable keyboard-coding-system to specify which coding system your keyboard uses (see Terminal Coding).  Enabling this feature will probably require you to use ESC to type Meta characters; however, on a console terminal or in xterm, you can arrange for Meta to be converted to ESC and still be able type 8-bit characters present directly on the keyboard or using Compose or AltGr keys.  See User Input.

  * For Latin-1 only, you can use the key C-x 8 as a ``compose character'' prefix for entry of non-ASCII Latin-1 printing characters.  C-x 8 is good for insertion (in the minibuffer as well as other buffers), for searching, and in any other context where a key sequence is allowed.

     C-x 8 works by loading the iso-transl library.  Once that library is loaded, the ALT modifier key, if the keyboard has one, serves the same purpose as C-x 8: use ALT together with an accent character to modify the following letter.  In addition, if the keyboard has keys for the Latin-1 ``dead accent characters,'' they too are defined to compose with the following character, once iso-transl is loaded.

     Use C-x 8 C-h to list all the available C-x 8 translations.

Charsets
========

Emacs groups all supported characters into disjoint charsets. Each character code belongs to one and only one charset.  For historical reasons, Emacs typically divides an 8-bit character code for an extended version of ASCII into two charsets: ASCII, which covers the codes 0 through 127, plus another charset which covers the ``right-hand part'' (the codes 128 and up). For instance, the characters of Latin-1 include the Emacs charset ascii plus the Emacs charset latin-iso8859-1.

Emacs characters belonging to different charsets may look the same, but they are still different characters.  For example, the letter o with acute accent in charset latin-iso8859-1, used for Latin-1, is different from the letter o with acute accent in charset latin-iso8859-2, used for Latin-2.

There are two commands for obtaining information about Emacs charsets.  The command M-x list-charset-chars prompts for a name of a character set, and displays all the characters in that character set.  The command M-x describe-character-set prompts for a charset name and displays information about that charset, including its internal representation within Emacs.

To find out which charset a character in the buffer belongs to, put point before it and type C-u C-x =.