1 -*-mode: text; coding: utf-8;-*-
3 Copyright (C) 2002-2016 Free Software Foundation, Inc.
4 See the end of the file for license conditions.
6 Importing a new Unicode Standard version into Emacs
7 -------------------------------------------------------------
9 Emacs uses the following files from the Unicode Character Database
18 First, these files need to be copied into admin/unidata/, and then
19 Emacs should be rebuilt for them to take effect. Rebuilding Emacs
20 updates several derived files elsewhere in the Emacs source tree,
21 mainly in lisp/international/.
23 When Emacs is rebuilt for the first time after importing the new
24 files, pay attention to any warning or error messages. In particular,
25 admin/unidata/unidata-gen.el will complain if UnicodeData.txt defines
26 new bidirectional attributes of characters, because unidata-gen.el,
27 bidi.c and dispextern.h need to be updated in that case; failure to do
28 so will cause aborts in redisplay.
30 Next, review the changes in UnicodeData.txt vs the previous version
31 used by Emacs. Any changes, be it introduction of new scripts or
32 addition of codepoints to existing scripts, might need corresponding
33 changes in the data used for filling the category-table, case-table,
34 and char-width-table. The additional scripts should cause automatic
35 updates in charscript.el, but it is a good idea to look at the results
36 and see if any changes in admin/unidata/blocks.awk are required.
38 Any new scripts added by UnicodeData.txt will also need updates to
39 script-representative-chars defined in fontset.el. Other databases in
40 fontset.el might also need to be updated as needed.
42 The function 'ucs-names', defined in lisp/international/mule-cmds.el,
43 might need to be updated because it knows about used and unused ranges
44 of Unicode codepoints, which a new release of the Unicode Standard
47 Problems, fixmes and other unicode-related issues
48 -------------------------------------------------------------
50 Notes by fx to record various things of variable importance. Handa
51 needs to check them -- don't take too seriously, especially with
52 regard to completeness.
54 * SINGLE_BYTE_CHAR_P returns true for Latin-1 characters, which has
55 undesirable effects. E.g.:
56 (multibyte-string-p (let ((s "x")) (aset s 0 ?£) s)) => nil
57 (multibyte-string-p (concat [?£])) => nil
58 (text-char-description ?£) => "M-#"
60 These examples are all fixed by the change of 2002-10-14, but
61 there still exist questionable SINGLE_BYTE_CHAR_P in the
62 code (keymap.c and print.c).
64 * Rationalize character syntax and its relationship to the Unicode
65 database. (Applies mainly to symbol an punctuation syntax.)
67 * Fontset handling and customization needs work. We want to relate
68 fonts to scripts, probably based on the Unicode blocks. The
69 presence of small-repertoire 10646-encoded fonts in XFree 4 is a
70 pain, not currently worked round.
72 With the change on 2002-07-26, multiple fonts can be
73 specified in a fontset for a specific range of characters.
74 Each range can also be specified by script. Before using
75 ISO10646 fonts, Emacs checks their repertories to avoid such
76 fonts that don't have a glyph for a specific character.
78 fx has worked on fontset customization, but was stymied by
79 basic problems with the way the default face is dealt with
80 (and something else, I think). This needs revisiting.
82 * Work is also needed on charset and coding system priorities.
84 * The relevant bits of latin1-disp.el need porting (and probably
85 re-naming/updating). See also cyril-util.el.
87 * Quail files need more work now the encoding is largely irrelevant.
89 * What to do with the old coding categories stuff?
91 * The preferred-coding-system property of charsets should probably be
92 junked unless it can be made more useful now.
94 * find-multibyte-characters needs looking at.
96 * Implement Korean cp949/UHC, BIG5-HKSCS and any other important missing
99 * Lazy-load tables for unify-charset somehow?
101 Actually, Emacs clears out all charset maps and unify-map just
102 before dumping, and they are loaded again on demand by the
103 dumped emacs. But, those maps (char tables) generated while
104 temacs is running can't be removed from the dumped emacs.
106 * iso-2022 charsets get unified on i/o.
108 With the change on 2003-01-06, decoding routines put the 'charset'
109 property onto decoded text, and iso-2022 encoder pay attention
110 to it. Thus, for instance, reading and writing by
111 iso-2022-7bit preserve the original designation sequences.
112 The property name 'preferred-charset' may be better?
114 We may have to utilize this property to decide a font.
116 * Revisit locale processing: look at treating the language and
117 charset parts separately. (Language should affect things like
118 spelling and calendar, but that's not a Unicode issue.)
120 * Handle Unicode combining characters usefully, e.g. diacritics, and
121 handle more scripts specifically (à la Devanagari). There are
122 issues with canonicalization.
124 * We need tabular input methods, e.g. for maths symbols. (Not
125 specific to Unicode.)
127 * Need multibyte text in menus, e.g. for the above. (Not specific to
128 Unicode -- see Emacs etc/TODO, but now mostly works with gtk.)
130 * There's currently no support for Unicode normalization.
132 * Populate char-width-table correctly for Unicode characters and
133 worry about what happens when double-width charsets covering
134 non-CJK characters are unified.
136 * There are type errors lurking, e.g. in
137 Fcheck_coding_systems_region. Define ENABLE_CHECKING to find them.
139 * Old auto-save files, and similar files, such as Gnus drafts,
140 containing non-ASCII characters probably won't be re-read correctly.
146 Most Emacs source files are encoded in UTF-8 (or in ASCII, which is a
147 subset), but there are a few exceptions, listed below. Perhaps
148 someday many of these files will be converted to UTF-8, for
149 convenience when using tools like 'grep -r', but this might need
150 nontrivial changes to the build process.
154 These are verbatim copies of files taken from external sources.
155 They haven't been converted to UTF-8.
157 leim/CXTERM-DIC/4Corner.tit
158 leim/CXTERM-DIC/ARRAY30.tit
159 leim/CXTERM-DIC/ECDICT.tit
160 leim/CXTERM-DIC/ETZY.tit
161 leim/CXTERM-DIC/PY-b5.tit
162 leim/CXTERM-DIC/Punct-b5.tit
163 leim/CXTERM-DIC/QJ-b5.tit
164 leim/CXTERM-DIC/ZOZY.tit
165 leim/MISC-DIC/CTLau-b5.html
166 leim/MISC-DIC/cangjie-table.b5
170 These are verbatim copies of files taken from external sources.
171 They haven't been converted to UTF-8.
173 leim/CXTERM-DIC/CCDOSPY.tit
174 leim/CXTERM-DIC/Punct.tit
175 leim/CXTERM-DIC/QJ.tit
176 leim/CXTERM-DIC/SW.tit
177 leim/CXTERM-DIC/TONEPY.tit
178 leim/MISC-DIC/CTLau.html
179 leim/MISC-DIC/pinyin.map
180 leim/MISC-DIC/ziranma.cin
184 This file contains non-ASCII characters in unibyte strings. When
185 editing a keyboard layout it's more convenient to see 'é' than
186 '\202', and the MS-DOS compiler requires the single byte if a
187 backslash escape is not being used.
193 This file is externally generated from leim/MISC-DIC/cangjie-table.b5
194 by Big5->CNS converter. It hasn't been converted to UTF-8.
196 leim/MISC-DIC/cangjie-table.cns
200 SKK-JISYO.L is a verbatim copy of a file taken from an external source.
201 It hasn't been converted to UTF-8.
203 leim/SKK-DIC/SKK-JISYO.L
207 This is a verbatim copy of a file taken from an external source.
208 It hasn't been converted to UTF-8.
210 admin/charsets/mapfiles/cns2ucsdkw.txt
214 This file switches between CJK charsets, which is not encoded in UTF-8.
218 Each of these files contains just one CJK charset, but Emacs
219 currently has no easy way to specify set-charset-priority on a
220 per-file basis, so converting any of these files to UTF-8 might
221 change the file's appearance when viewed by an Emacs that is
222 operating in some other language environment.
224 etc/tutorials/TUTORIAL.ja
225 lisp/international/ja-dic-cnv.el
226 lisp/international/ja-dic-utl.el
227 lisp/international/kinsoku.el
228 lisp/international/kkc.el
229 lisp/international/titdic-cnv.el
230 lisp/language/japan-util.el
231 lisp/language/japanese.el
232 lisp/leim/quail/cyril-jis.el
233 lisp/leim/quail/hanja-jis.el
234 lisp/leim/quail/japanese.el
235 lisp/leim/quail/py-punct.el
236 lisp/leim/quail/pypunct-b5.el
238 This file contains just Chinese characters, and has same problem.
239 Also, it contains characters that cannot be encoded in UTF-8.
241 lisp/international/titdic-cnv.el
245 These files contain characters that cannot be encoded in UTF-8.
247 lisp/language/ethio-util.el
248 lisp/language/ethiopic.el
249 lisp/language/ind-util.el
250 lisp/language/tibet-util.el
251 lisp/language/tibetan.el
252 lisp/leim/quail/ethiopic.el
253 lisp/leim/quail/tibetan.el
257 These files contain binary data, and are not text files.
258 Some of the entries in this list are patterns, and stand for any
259 files with the listed extension.
269 etc/package-keyring.gpg
271 nextstep/GNUstep/Emacs.base/Resources/emacs.tiff
275 This file is part of GNU Emacs.
277 GNU Emacs is free software: you can redistribute it and/or modify
278 it under the terms of the GNU General Public License as published by
279 the Free Software Foundation, either version 3 of the License, or
280 (at your option) any later version.
282 GNU Emacs is distributed in the hope that it will be useful,
283 but WITHOUT ANY WARRANTY; without even the implied warranty of
284 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
285 GNU General Public License for more details.
287 You should have received a copy of the GNU General Public License
288 along with GNU Emacs. If not, see <http://www.gnu.org/licenses/>.