2 @setfilename ../../info/url.info
3 @settitle URL Programmer's Manual
9 @c @setchapternewpage odd
14 %\global\baselineskip 30pt % for printing in double space
16 @dircategory Emacs lisp libraries
18 * URL: (url). URL loading package.
22 This is the manual for the @code{url} Emacs Lisp library.
24 Copyright @copyright{} 1993--1999, 2002, 2004--2016 Free Software
28 Permission is granted to copy, distribute and/or modify this document
29 under the terms of the GNU Free Documentation License, Version 1.3 or
30 any later version published by the Free Software Foundation; with no
31 Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,''
32 and with the Back-Cover Texts as in (a) below. A copy of the license
33 is included in the section entitled ``GNU Free Documentation License''.
35 (a) The FSF's Back-Cover Text is: ``You have the freedom to copy and
36 modify this GNU manual.''
42 @title URL Programmer's Manual
43 @subtitle First Edition, URL Version 2.0
44 @author William M. Perry @email{wmperry@@gnu.org}
45 @author David Love @email{fx@@gnu.org}
47 @vskip 0pt plus 1filll
61 * Introduction:: About the @code{url} library.
62 * URI Parsing:: Parsing (and unparsing) URIs.
63 * Retrieving URLs:: How to use this package to retrieve a URL.
64 * Supported URL Types:: Descriptions of URL types currently supported.
65 * General Facilities:: URLs can be cached, accessed via a gateway
66 and tracked in a history list.
67 * Customization:: Variables you can alter.
68 * GNU Free Documentation License:: The license for this documentation.
78 @cindex uniform resource identifier
79 @cindex uniform resource locator
81 A @dfn{Uniform Resource Identifier} (URI) is a specially-formatted
82 name, such as an Internet address, that identifies some name or
83 resource. The format of URIs is described in RFC 3986, which updates
84 and replaces the earlier RFCs 2732, 2396, 1808, and 1738. A
85 @dfn{Uniform Resource Locator} (URL) is an older but still-common
86 term, which basically refers to a URI corresponding to a resource that
87 can be accessed (usually over a network) in a specific way.
89 Here are some examples of URIs (taken from RFC 3986):
92 ftp://ftp.is.co.za/rfc/rfc1808.txt
93 http://www.ietf.org/rfc/rfc2396.txt
94 ldap://[2001:db8::7]/c=GB?objectClass?one
95 mailto:John.Doe@@example.com
96 news:comp.infosystems.www.servers.unix
98 telnet://192.0.2.16:80/
99 urn:oasis:names:specification:docbook:dtd:xml:4.1.2
102 This manual describes the @code{url} library, an Emacs Lisp library
103 for parsing URIs and retrieving the resources to which they refer.
104 (The library is so-named for historical reasons; nowadays, the ``URI''
105 terminology is regarded as the more general one, and ``URL'' is
106 technically obsolete despite its widespread vernacular usage.)
111 A URI consists of several @dfn{components}, each having a different
112 meaning. For example, the URI
115 http://www.gnu.org/software/emacs/
119 specifies the scheme component @samp{http}, the hostname component
120 @samp{www.gnu.org}, and the path component @samp{/software/emacs/}.
123 The format of URIs is specified by RFC 3986. The @code{url} library
124 provides the Lisp function @code{url-generic-parse-url}, a (mostly)
125 standard-compliant URI parser, as well as function
126 @code{url-recreate-url}, which converts a parsed URI back into a URI
129 @defun url-generic-parse-url uri-string
130 This function returns a parsed version of the string @var{uri-string}.
133 @defun url-recreate-url uri-obj
134 @cindex unparsing URLs
135 Given a parsed URI, this function returns the corresponding URI string.
139 The return value of @code{url-generic-parse-url}, and the argument
140 expected by @code{url-recreate-url}, is a @dfn{parsed URI}: a CL
141 structure whose slots hold the various components of the URI@.
142 @xref{Top,the CL Manual,,cl,GNU Emacs Common Lisp Emulation}, for
143 details about CL structures. Most of the other functions in the
144 @code{url} library act on parsed URIs.
147 * Parsed URIs:: Format of parsed URI structures.
148 * URI Encoding:: Non-@acronym{ASCII} characters in URIs.
152 @section Parsed URI structures
154 Each parsed URI structure contains the following slots:
158 The URI scheme (a string, e.g., @code{http}). @xref{Supported URL
159 Types}, for a list of schemes that the @code{url} library knows how to
160 process. This slot can also be @code{nil}, if the URI is not fully
164 The user name (a string), or @code{nil}.
167 The user password (a string), or @code{nil}. The use of this URI
168 component is strongly discouraged; nowadays, passwords are transmitted
169 by other means, not as part of a URI.
172 The host name (a string), or @code{nil}. If present, this is
173 typically a domain name or IP address.
176 The port number (an integer), or @code{nil}. Omitting this component
177 usually means to use the ``standard'' port associated with the URI
181 The combination of the ``path'' and ``query'' components of the URI (a
182 string), or @code{nil}. If the query component is present, it is the
183 substring following the first @samp{?} character, and the path
184 component is the substring before the @samp{?}. The meaning of these
185 components is scheme-dependent; they do not necessarily refer to a
189 The fragment component (a string), or @code{nil}. The fragment
190 component specifies a ``secondary resource'', such as a section of a
194 This is @code{t} if the URI is fully specified, i.e., the
195 hierarchical components of the URI (the hostname and/or username
196 and/or password) are preceded by @samp{//}.
206 @findex url-attributes
208 These slots have accessors named @code{url-@var{part}}, where
209 @var{part} is the slot name. For example, the accessor for the
210 @code{host} slot is the function @code{url-host}. The @code{url-port}
211 accessor returns the default port for the URI scheme if the parsed
212 URI's @var{port} slot is @code{nil}.
214 The slots can be set using @code{setf}. For example:
217 (setf (url-port url) 80)
221 @section URI Encoding
223 @cindex percent encoding
224 The @code{url-generic-parse-url} parser does not obey RFC 3986 in
225 one respect: it allows non-@acronym{ASCII} characters in URI strings.
227 Strictly speaking, RFC 3986 compatible URIs may only consist of
228 @acronym{ASCII} characters; non-@acronym{ASCII} characters are
229 represented by converting them to UTF-8 byte sequences, and performing
230 @dfn{percent encoding} on the bytes. For example, the o-umlaut
231 character is converted to the UTF-8 byte sequence @samp{\xD3\xA7},
232 then percent encoded to @samp{%D3%A7}. (Certain ``reserved''
233 @acronym{ASCII} characters must also be percent encoded when they
234 appear in URI components.)
236 The function @code{url-encode-url} can be used to convert a URI
237 string containing arbitrary characters to one that is properly
238 percent-encoded in accordance with RFC 3986.
240 @defun url-encode-url url-string
241 This function return a properly URI-encoded version of
242 @var{url-string}. It also performs @dfn{URI normalization},
243 e.g., converting the scheme component to lowercase if it was
244 previously uppercase.
247 To convert between a string containing arbitrary characters and a
248 percent-encoded all-@acronym{ASCII} string, use the functions
249 @code{url-hexify-string} and @code{url-unhex-string}:
251 @defun url-hexify-string string &optional allowed-chars
252 This function performs percent-encoding on @var{string}, and returns
255 If @var{string} is multibyte, it is first converted to a UTF-8 byte
256 string. Each byte corresponding to an allowed character is left
257 as-is, while all other bytes are converted to a three-character
258 sequence: @samp{%} followed by two upper-case hex digits.
260 @vindex url-unreserved-chars
261 @cindex unreserved characters
262 The allowed characters are specified by @var{allowed-chars}. If this
263 argument is @code{nil}, the allowed characters are those specified as
264 @dfn{unreserved characters} by RFC 3986 (see the variable
265 @code{url-unreserved-chars}). Otherwise, @var{allowed-chars} should
266 be a vector whose @var{n}-th element is non-@code{nil} if character
270 @defun url-unhex-string string &optional allow-newlines
271 This function replaces percent-encoding sequences in @var{string} with
272 their character equivalents, and returns the resulting string.
274 If @var{allow-newlines} is non-@code{nil}, it allows the decoding of
275 carriage returns and line feeds, which are normally forbidden in URIs.
278 @node Retrieving URLs
279 @chapter Retrieving URLs
281 The @code{url} library defines the following three functions for
282 retrieving the data specified by a URL@. The actual retrieval protocol
283 depends on the URL's URI scheme, and is performed by lower-level
284 scheme-specific functions. (Those lower-level functions are not
285 documented here, and generally should not be called directly.)
287 In each of these functions, the @var{url} argument can be either a
288 string or a parsed URL structure. If it is a string, that string is
289 passed through @code{url-encode-url} before using it, to ensure that
290 it is properly URI-encoded (@pxref{URI Encoding}).
292 @defun url-retrieve-synchronously url
293 This function synchronously retrieves the data specified by @var{url},
294 and returns a buffer containing the data. The return value is
295 @code{nil} if there is no data associated with the URL (as is the case
296 for @code{dired}, @code{info}, and @code{mailto} URLs).
299 @defun url-retrieve url callback &optional cbargs silent no-cookies
300 This function retrieves @var{url} asynchronously, calling the function
301 @var{callback} when the object has been completely retrieved. The
302 return value is the buffer into which the data will be inserted, or
303 @code{nil} if the process has already completed.
305 The callback function is called this way:
308 (apply @var{callback} @var{status} @var{cbargs})
312 where @var{status} is a plist representing what happened during the
313 retrieval, with most recent events first, or an empty list if no
314 events have occurred. Each pair in the plist is one of:
317 @item (:redirect @var{redirected-to})
318 This means that the request was redirected to the URL
321 @item (:error (@var{error-symbol} . @var{data}))
322 This means that an error occurred. If so desired, the error can be
323 signaled with @code{(signal @var{error-symbol} @var{data})}.
326 When the callback function is called, the current buffer is the one
327 containing the retrieved data (if any). The buffer also contains any
328 MIME headers associated with the data retrieval.
330 If the optional argument @var{silent} is non-@code{nil}, progress
331 messages are suppressed. If the optional argument @var{no-cookies} is
332 non-@code{nil}, cookies are not stored or sent.
335 @defun url-queue-retrieve url callback &optional cbargs silent no-cookies
336 This function acts like @code{url-retrieve}, but with limits on the
337 number of concurrently-running network processes. The option
338 @code{url-queue-parallel-processes} controls the number of concurrent
339 processes, and the option @code{url-queue-timeout} sets a timeout in
342 To use this function, you must @code{(require 'url-queue)}.
345 @vindex url-queue-parallel-processes
346 @defopt url-queue-parallel-processes
347 The value of this option is an integer specifying the maximum number
348 of concurrent @code{url-queue-retrieve} network processes. If the
349 number of @code{url-queue-retrieve} calls is larger than this number,
350 later ones are queued until earlier ones are finished.
353 @vindex url-queue-timeout
354 @defopt url-queue-timeout
355 The value of this option is a number specifying the maximum lifetime
356 of a @code{url-queue-retrieve} network process, once it is started.
357 If a process is not finished by then, it is killed and removed from
361 @node Supported URL Types
362 @chapter Supported URL Types
364 This chapter describes functions and variables affecting URL retrieval
365 for specific schemes.
368 * http/https:: Hypertext Transfer Protocol.
369 * file/ftp:: Local files and FTP archives.
370 * info:: Emacs "Info" pages.
371 * mailto:: Sending email.
372 * news/nntp/snews:: Usenet news.
373 * rlogin/telnet/tn3270:: Remote host connectivity.
374 * irc:: Internet Relay Chat.
375 * data:: Embedded data URLs.
376 * nfs:: Networked File System.
377 * ldap:: Lightweight Directory Access Protocol.
378 * man:: Unix man pages.
379 * Tramp:: Schemes supported via Tramp.
383 @section @code{http} and @code{https}
385 The @code{http} scheme refers to the Hypertext Transfer Protocol. The
386 @code{url} library supports HTTP version 1.1, specified in RFC 2616.
387 Its default port is 80.
389 The @code{https} scheme is a secure version of @code{http}, with
390 transmission via SSL@. It is defined in RFC 2069, and its default port
391 is 443. When using @code{https}, the @code{url} library performs SSL
392 encryption via the @code{ssl} library, by forcing the @code{ssl}
393 gateway method to be used. @xref{Gateways in general}.
395 @defopt url-honor-refresh-requests
396 If this option is non-@code{nil} (the default), the @code{url} library
397 honors the HTTP @samp{Refresh} header, which is used by servers to
398 direct clients to reload documents from the same URL or a or different
399 one. If the value is @code{nil}, the @samp{Refresh} header is
400 ignored; any other value means to ask the user on each request.
405 * HTTP language/coding::
407 * Dealing with HTTP documents::
413 @findex url-cookie-delete
414 @defun url-cookie-list
415 This command creates a @file{*url cookies*} buffer listing the current
416 cookies, if there are any. You can remove a cookie using the
417 @kbd{C-k} (@code{url-cookie-delete}) command.
420 @defopt url-cookie-file
421 The file in which cookies are stored, defaulting to @file{cookies} in
422 the directory specified by @code{url-configuration-directory}.
425 @defopt url-cookie-confirmation
426 Specifies whether confirmation is required to accept cookies.
429 @defopt url-cookie-multiple-line
430 Specifies whether to put all cookies for the server on one line in the
431 HTTP request to satisfy broken servers like
432 @url{http://www.hotmail.com}.
435 @defopt url-cookie-trusted-urls
436 A list of regular expressions matching URLs from which to accept
440 @defopt url-cookie-untrusted-urls
441 A list of regular expressions matching URLs from which to reject
445 @defopt url-cookie-save-interval
446 The number of seconds between automatic saves of cookies to disk.
451 @node HTTP language/coding
452 @subsection Language and Encoding Preferences
454 HTTP allows clients to express preferences for the language and
455 encoding of documents which servers may honor. For each of these
456 variables, the value is a string; it can specify a single choice, or
457 it can be a comma-separated list.
459 Normally, this list is ordered by descending preference. However, each
460 element can be followed by @samp{;q=@var{priority}} to specify its
461 preference level, a decimal number from 0 to 1; e.g., for
462 @code{url-mime-language-string}, @w{@code{"de, en-gb;q=0.8,
463 en;q=0.7"}}. An element that has no @samp{;q} specification has
466 @defopt url-mime-charset-string
467 @cindex character sets
468 @cindex coding systems
469 This variable specifies a preference for character sets when documents
470 can be served in more than one encoding.
472 HTTP allows specifying a series of MIME charsets which indicate your
473 preferred character set encodings, e.g., Latin-9 or Big5, and these
474 can be weighted. The default series is generated automatically from
475 the associated MIME types of all defined coding systems, sorted by the
476 coding system priority specified in Emacs. @xref{Recognize Coding, ,
477 Recognizing Coding Systems, emacs, The GNU Emacs Manual}.
480 @defopt url-mime-language-string
481 @cindex language preferences
482 A string specifying the preferred language when servers can serve
483 files in several languages. Use RFC 1766 abbreviations, e.g.,
484 @samp{en} for English, @samp{de} for German.
486 The string can be @code{"*"} to get the first available language (as
487 opposed to the default).
490 @node HTTP URL Options
491 @subsection HTTP URL Options
493 HTTP supports an @samp{OPTIONS} method describing things supported by
496 @defun url-http-options url
497 Returns a property list describing options available for URL@. The
498 property list members are:
502 A list of symbols specifying what HTTP methods the resource
507 A list of numbers specifying what DAV protocol/schema versions are
512 A list of supported DASL search types supported (string form).
515 A list of the units available for use in partial document fetches.
519 The @dfn{Platform For Privacy Protection} description for the resource.
520 Currently this is just the raw header contents.
525 @node Dealing with HTTP documents
526 @subsection Dealing with HTTP documents
528 HTTP URLs are retrieved into a buffer containing the HTTP headers
529 followed by the body. Since the headers are quasi-MIME, they may be
530 processed using the MIME library. @xref{Top,, Emacs MIME,
531 emacs-mime, The Emacs MIME Manual}.
534 @section file and ftp
537 @cindex File Transfer Protocol
538 @cindex compressed files
541 The @code{ftp} and @code{file} schemes are defined in RFC 1808. The
542 @code{url} library treats @samp{ftp:} and @samp{file:} as synonymous.
543 Such URLs have the form
546 ftp://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
547 file://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
551 If the URL specifies a local file, it is retrieved by reading the file
552 contents in the usual way. If it specifies a remote file, it is
553 retrieved using either the Tramp or the Ange-FTP package.
554 @xref{Remote Files,,, emacs, The GNU Emacs Manual}.
556 When retrieving a compressed file, it is automatically uncompressed
557 if it has the file suffix @file{.z}, @file{.gz}, @file{.Z},
558 @file{.bz2}, or @file{.xz}. (The list of supported suffixes is
559 hard-coded, and cannot be altered by customizing
560 @code{jka-compr-compression-info-list}.)
562 @defopt url-directory-index-file
563 This option specifies the filename to look for when a @code{file} or
564 @code{ftp} URL specifies a directory. The default is
565 @file{index.html}. If this file exists and is readable, it is viewed.
566 Otherwise, Emacs visits the directory using Dired.
573 @findex Info-goto-node
575 The @code{info} scheme is non-standard. Such URLs have the form
578 info:@var{file}#@var{node}
582 and are retrieved by invoking @code{Info-goto-node} with argument
583 @samp{(@var{file})@var{node}}. If @samp{#@var{node}} is omitted, the
584 @samp{Top} node is opened.
591 A @code{mailto} URL specifies an email message to be sent to a given
592 email address. For example, @samp{mailto:foo@@bar.com} specifies
593 sending a message to @samp{foo@@bar.com}. The ``retrieval method''
594 for such URLs is to open a mail composition buffer in which the
595 appropriate content (e.g., the recipient address) has been filled in.
597 As defined in RFC 6068, a @code{mailto} URL can have the form
600 @samp{mailto:@var{mailbox}[?@var{header}=@var{contents}[&@var{header}=@var{contents}]]}
604 where an arbitrary number of @var{header}s can be added. If the
605 @var{header} is @samp{body}, then @var{contents} is put in the message
606 body; otherwise, a @var{header} header field is created with
607 @var{contents} as its contents. Note that the @code{url} library does
608 not perform any checking of @var{header} or @var{contents}, so you
609 should check them before sending the message.
611 @defopt url-mail-command
612 @vindex mail-user-agent
613 The value of this variable is the function called whenever url needs
614 to send mail. This should normally be left its default, which is the
615 standard mail-composition command @code{compose-mail}. @xref{Sending
616 Mail,,, emacs, The GNU Emacs Manual}.
619 If the document containing the @code{mailto} URL itself possessed a
620 known URL, Emacs automatically inserts an @samp{X-Url-From} header
621 field into the mail buffer, specifying that URL.
623 @node news/nntp/snews
624 @section @code{news}, @code{nntp} and @code{snews}
631 The @code{news}, @code{nntp}, and @code{snews} schemes, defined in RFC
632 1738, are used for reading Usenet newsgroups. For compatibility with
633 non-standard-compliant news clients, the @code{url} library allows
634 host and port fields to be included in @code{news} URLs, even though
635 they are properly only allowed for @code{nntp} and @code{snews}.
637 @code{news} and @code{nntp} URLs have the following form:
640 @item news:@var{newsgroup}
641 Retrieves a list of messages in @var{newsgroup};
642 @item news:@var{message-id}
643 Retrieves the message with the given @var{message-id};
645 Retrieves a list of all available newsgroups;
646 @item nntp://@var{host}:@var{port}/@var{newsgroup}
647 @itemx nntp://@var{host}:@var{port}/@var{message-id}
648 @itemx nntp://@var{host}:@var{port}/*
649 Similar to the @samp{news} versions.
652 The default port for @code{nntp} (and @code{news}) is 119. The
653 difference between an @code{nntp} URL and a @code{news} URL is that an
654 @code{nttp} URL may specify an article by its number. The
655 @samp{snews} scheme is the same as @samp{nntp}, except that it is
656 tunneled through SSL and has default port 563.
658 These URLs are retrieved via the Gnus package.
660 @cindex environment variable
662 @defopt url-news-server
663 This variable specifies the default news server from which to fetch
664 news, if no server was specified in the URL@. The default value,
665 @code{nil}, means to use the server specified by the standard
666 environment variable @samp{NNTPSERVER}, or @samp{news} if that
667 environment variable is unset.
670 @node rlogin/telnet/tn3270
671 @section rlogin, telnet and tn3270
675 @cindex terminal emulation
676 @findex terminal-emulator
678 These URL schemes are defined in RFC 1738, and are used for logging in
679 via a terminal emulator. They have the form
682 telnet://@var{user}:@var{password}@@@var{host}:@var{port}
686 but the @var{password} component is ignored. By default, the
687 @code{telnet} scheme is handled via Tramp (@pxref{Tramp}).
689 To handle rlogin, telnet and tn3270 URLs, a @code{rlogin},
690 @code{telnet} or @code{tn3270} (the program names and arguments are
691 hardcoded) session is run in a @code{terminal-emulator} buffer.
692 Well-known ports are used if the URL does not specify a port.
697 @cindex Internet Relay Chat
702 The @code{irc} scheme is defined in the Internet Draft at
703 @url{http://www.w3.org/Addressing/draft-mirashi-url-irc-01.txt} (which
704 was never approved as an RFC). Such URLs have the form
707 irc://@var{host}:@var{port}/@var{target},@var{needpass}
711 and are retrieved by opening an @acronym{IRC} session using the
712 function specified by @code{url-irc-function}.
714 @defopt url-irc-function
715 The value of this option is a function, which is called to open an IRC
716 connection for @code{irc} URLs. This function must take five
717 arguments, @var{host}, @var{port}, @var{channel}, @var{user} and
718 @var{password}. The @var{channel} argument specifies the channel to
719 join immediately, and may be @code{nil}.
721 The default is @code{url-irc-rcirc}, which uses the Rcirc package.
722 Other options are @code{url-irc-erc} (which uses ERC) and
723 @code{url-irc-zenirc} (which uses ZenIRC).
730 The @code{data} scheme, defined in RFC 2397, contains MIME data in
731 the URL itself. Such URLs have the form
734 data:@r{[}@var{media-type}@r{]}@r{[};@var{base64}@r{]},@var{data}
738 @var{media-type} is a MIME @samp{Content-Type} string, possibly
739 including parameters. It defaults to
740 @samp{text/plain;charset=US-ASCII}. The @samp{text/plain} can be
741 omitted but the charset parameter supplied. If @samp{;base64} is
742 present, the @var{data} are base64-encoded.
747 @cindex Network File System
750 The @code{nfs} scheme, defined in RFC 2224, is similar to @code{ftp}
751 except that it points to a file on a remote host that is handled by an
752 NFS automounter on the local host. Such URLs have the form
755 nfs://@var{user}:@var{password}@@@var{host}:@var{port}/@var{file}
758 @defvar url-nfs-automounter-directory-spec
760 A string saying how to invoke the NFS automounter. Certain @samp{%}
761 sequences are recognized:
765 The hostname of the NFS server;
767 The port number of the NFS server;
769 The username to use to authenticate;
771 The password to use to authenticate;
773 The filename on the remote server;
778 Each can be used any number of times.
783 @cindex Lightweight Directory Access Protocol
785 The LDAP scheme is defined in RFC 2255.
789 @cindex @command{man}
790 @cindex Unix man pages
793 The @code{man} scheme is a non-standard one. Such URLs have the form
796 @samp{man:@var{page-spec}}
800 and are retrieved by passing @var{page-spec} to the Lisp function
804 @section URL Types Supported via Tramp
806 @vindex url-tramp-protocols
807 Some additional URL types are supported by passing them to Tramp
808 (@pxref{Top, The Tramp Manual,, tramp, The Tramp Manual}). These
809 protocols are listed in the @code{url-tramp-protocols} variable, which
810 you can customize. The default value includes the following
815 The file transfer protocol. @xref{file/ftp}.
819 The secure shell protocol. @xref{Inline Methods,,, tramp, The Tramp
824 The secure file copy protocol. @xref{External Methods,,, tramp, The
829 The remote sync protocol.
835 @node General Facilities
836 @chapter General Facilities
841 * Gateways in general::
846 @section Disk Caching
848 @cindex Persistent Cache
851 The disk cache stores retrieved documents locally, whence they can be
852 retrieved more quickly. When requesting a URL that is in the cache,
853 the library checks to see if the page has changed since it was last
854 retrieved from the remote machine. If not, the local copy is used,
855 saving the transmission over the network.
856 @cindex Cleaning the cache
857 @cindex Clearing the cache
858 @cindex Cache cleaning
859 Currently the cache isn't cleared automatically.
860 @c Running the @code{clean-cache} shell script
861 @c fist is recommended, to allow for future cleaning of the cache. This
862 @c shell script will remove all files that have not been accessed since it
863 @c was last run. To keep the cache pared down, it is recommended that this
864 @c script be run from @i{at} or @i{cron} (see the manual pages for
865 @c crontab(5) or at(1) for more information)
867 @defopt url-automatic-caching
868 Setting this variable non-@code{nil} causes documents to be cached
872 @defopt url-cache-directory
873 This variable specifies the
874 directory to store the cache files. It defaults to sub-directory
875 @file{cache} of @code{url-configuration-directory}.
878 @defopt url-cache-creation-function
879 The cache relies on a scheme for mapping URLs to files in the cache.
880 This variable names a function which sets the type of cache to use.
881 It takes a URL as argument and returns the absolute file name of the
882 corresponding cache file. The two supplied possibilities are
883 @code{url-cache-create-filename-using-md5} and
884 @code{url-cache-create-filename-human-readable}.
887 @defun url-cache-create-filename-using-md5 url
888 Creates a cache file name from @var{url} using MD5 hashing.
889 This is creates entries with very few cache collisions and is fast.
892 (url-cache-create-filename-using-md5 "http://www.example.com/foo/bar")
893 @result{} "/home/fx/.url/cache/fx/http/com/example/www/b8a35774ad20db71c7c3409a5410e74f"
897 @defun url-cache-create-filename-human-readable url
898 Creates a cache file name from @var{url} more obviously connected to
899 @var{url} than for @code{url-cache-create-filename-using-md5}, but
900 more likely to conflict with other files.
902 (url-cache-create-filename-human-readable "http://www.example.com/foo/bar")
903 @result{} "/home/fx/.url/cache/fx/http/com/example/www/foo/bar"
907 @defun url-cache-expired
908 This function returns non-@code{nil} if a cache entry has expired (or is absent).
909 The arguments are a URL and optional expiration delay in seconds
910 (default @var{url-cache-expire-time}).
913 @defopt url-cache-expire-time
914 This variable is the default number of seconds to use for the
915 expire-time argument of the function @code{url-cache-expired}.
918 @defun url-fetch-from-cache
919 This function takes a URL as its argument and returns a buffer
920 containing the data cached for that URL.
923 @c Fixme: never actually used currently?
924 @c @defopt url-standalone-mode
925 @c @cindex Relying on cache
926 @c @cindex Cache only mode
927 @c @cindex Standalone mode
928 @c If this variable is non-@code{nil}, the library relies solely on the
929 @c cache for fetching documents and avoids checking if they have changed
930 @c on remote servers.
933 @c With a large cache of documents on the local disk, it can be very handy
934 @c when traveling, or any other time the network connection is not active
935 @c (a laptop with a dial-on-demand PPP connection, etc.). Emacs/W3 can rely
936 @c solely on its cache, and avoid checking to see if the page has changed
937 @c on the remote server. In the case of a dial-on-demand PPP connection,
938 @c this will keep the phone line free as long as possible, only bringing up
939 @c the PPP connection when asking for a page that is not located in the
940 @c cache. This is very useful for demonstrations as well.
943 @section Proxies and Gatewaying
945 @c fixme: check/document url-ns stuff
946 @cindex proxy servers
948 @cindex environment variables
950 Proxy servers are commonly used to provide gateways through firewalls
951 or as caches serving some more-or-less local network. Each protocol
952 (HTTP, FTP, etc.)@: can have a different gateway server. Proxying is
953 conventionally configured commonly amongst different programs through
954 environment variables of the form @code{@var{protocol}_proxy}, where
955 @var{protocol} is one of the supported network protocols (@code{http},
956 @code{ftp} etc.). The library recognizes such variables in either
957 upper or lower case. Their values are of one of the forms:
959 @item @code{@var{host}:@var{port}}
961 @item Simply a host name.
965 The @code{NO_PROXY} environment variable specifies URLs that should be
966 excluded from proxying (on servers that should be contacted directly).
967 This should be a comma-separated list of hostnames, domain names, or a
968 mixture of both. Asterisks can be used as wildcards, but other
969 clients may not support that. Domain names may be indicated by a
970 leading dot. For example:
972 NO_PROXY="*.aventail.com,home.com,.seanet.com"
974 @noindent says to contact all machines in the @samp{aventail.com} and
975 @samp{seanet.com} domains directly, as well as the machine named
976 @samp{home.com}. If @code{NO_PROXY} isn't defined, @code{no_PROXY}
977 and @code{no_proxy} are also tried, in that order.
979 Proxies may also be specified directly in Lisp.
981 @defopt url-proxy-services
982 This variable is an alist of URL schemes and proxy servers that
983 gateway them. The items are of the form @w{@code{(@var{scheme}
984 . @var{host}:@var{portnumber})}}, says that the URL @var{scheme} is
985 gatewayed through @var{portnumber} on the specified @var{host}. An
986 exception is the pseudo scheme @code{"no_proxy"}, which is paired with
987 a regexp matching host names not to be proxied. This variable is
988 initialized from the environment as above.
991 (setq url-proxy-services
992 '(("http" . "proxy.aventail.com:80")
993 ("no_proxy" . "^.*\\(aventail\\|seanet\\)\\.com")))
997 @node Gateways in general
998 @section Gateways in General
1002 The library provides a general gateway layer through which all
1003 networking passes. It can both control access to the network and
1004 provide access through gateways in firewalls. This may make direct
1005 connections in some cases and pass through some sort of gateway in
1006 others.@footnote{Proxies (which only operate over HTTP) are
1007 implemented using this.} The library's basic function responsible for
1008 making connections is @code{url-open-stream}.
1010 @defun url-open-stream name buffer host service
1011 @cindex opening a stream
1012 @cindex stream, opening
1013 Open a stream to @var{host}, possibly via a gateway. The other
1014 arguments are as for @code{open-network-stream}. This will not make a
1015 connection if @code{url-gateway-unplugged} is non-@code{nil}.
1018 @defvar url-gateway-local-host-regexp
1019 This is a regular expression that matches local hosts that do not
1020 require the use of a gateway. If @code{nil}, all connections are made
1021 through the gateway.
1024 @defvar url-gateway-method
1025 This variable controls which gateway method is used. It may be useful
1026 to bind it temporarily in some applications. It has values taken from
1027 a list of symbols. Possible values are:
1031 @cindex @command{telnet}
1032 Use this method if you must first telnet and log into a gateway host,
1033 and then run telnet from that host to connect to outside machines.
1036 @cindex @command{rlogin}
1037 This method is identical to @code{telnet}, but uses @command{rlogin}
1038 to log into the remote machine without having to send the username and
1039 password over the wire every time.
1043 Use if the firewall has a @sc{socks} gateway running on it. The
1044 @sc{socks} v5 protocol is defined in RFC 1928.
1047 @c This probably shouldn't be documented
1048 @c Fixme: why not? -- fx
1051 This method uses Emacs's builtin networking directly. This is the
1052 default. It can be used only if there is no firewall blocking access.
1056 The following variables control the gateway methods.
1058 @defopt url-gateway-telnet-host
1059 The gateway host to telnet to. Once logged in there, you then telnet
1060 out to the hosts you want to connect to.
1062 @defopt url-gateway-telnet-parameters
1063 This should be a list of parameters to pass to the @command{telnet} program.
1065 @defopt url-gateway-telnet-password-prompt
1066 This is a regular expression that matches the password prompt when
1069 @defopt url-gateway-telnet-login-prompt
1070 This is a regular expression that matches the username prompt when
1073 @defopt url-gateway-telnet-user-name
1074 The username to log in with.
1076 @defopt url-gateway-telnet-password
1077 The password to send when logging in.
1079 @defopt url-gateway-prompt-pattern
1080 This is a regular expression that matches the shell prompt.
1083 @defopt url-gateway-rlogin-host
1084 Host to @samp{rlogin} to before telnetting out.
1086 @defopt url-gateway-rlogin-parameters
1087 Parameters to pass to @samp{rsh}.
1089 @defopt url-gateway-rlogin-user-name
1090 User name to use when logging in to the gateway.
1092 @defopt url-gateway-prompt-pattern
1093 This is a regular expression that matches the shell prompt.
1096 @defopt socks-server
1097 This specifies the default server, it takes the form
1098 @w{@code{("Default server" @var{server} @var{port} @var{version})}}
1099 where @var{version} can be either 4 or 5.
1101 @defvar socks-password
1102 If this is @code{nil} then you will be asked for the password,
1103 otherwise it will be used as the password for authenticating you to
1104 the @sc{socks} server.
1106 @defvar socks-username
1107 This is the username to use when authenticating yourself to the
1108 @sc{socks} server. By default this is your login name.
1110 @defvar socks-timeout
1111 This controls how long, in seconds, to wait for responses from the
1112 @sc{socks} server; it is 5 by default.
1114 @c fixme: these have been effectively commented-out in the code
1115 @c @defopt socks-server-aliases
1116 @c This a list of server aliases. It is a list of aliases of the form
1117 @c @var{(alias hostname port version)}.
1119 @c @defopt socks-network-aliases
1120 @c This a list of network aliases. Each entry in the list takes the form
1121 @c @var{(alias (network))} where @var{alias} is a string that names the
1122 @c @var{network}. The networks can contain a pair (not a dotted pair) of
1123 @c @sc{ip} addresses which specify a range of @sc{ip} addresses, an @sc{ip}
1124 @c address and a netmask, a domain name or a unique hostname or @sc{ip}
1127 @c @defopt socks-redirection-rules
1128 @c This a list of redirection rules. Each rule take the form
1129 @c @var{(Destination network Connection type)} where @var{Destination
1130 @c network} is a network alias from @code{socks-network-aliases} and
1131 @c @var{Connection type} can be @code{nil} in which case a direct
1132 @c connection is used, or it can be an alias from
1133 @c @code{socks-server-aliases} in which case that server is used as a
1136 @defopt socks-nslookup-program
1137 @cindex @command{nslookup}
1138 This the @samp{nslookup} program. It is @code{"nslookup"} by default.
1142 * Suppressing network connections::
1144 @c * Broken hostname resolution::
1146 @node Suppressing network connections
1147 @subsection Suppressing Network Connections
1149 @cindex network connections, suppressing
1150 @cindex suppressing network connections
1152 @cindex HTML ``bugs''
1153 In some circumstances it is desirable to suppress making network
1154 connections. A typical case is when rendering HTML in a mail user
1155 agent, when external URLs should not be activated, particularly to
1156 avoid ``bugs'' which ``call home'' by fetch single-pixel images and the
1157 like. To arrange this, bind the following variable for the duration
1160 @defvar url-gateway-unplugged
1161 If this variable is non-@code{nil} new network connections are never
1162 opened by the URL library.
1165 @c @node Broken hostname resolution
1166 @c @subsection Broken Hostname Resolution
1168 @c @cindex hostname resolver
1169 @c @cindex resolver, hostname
1170 @c Some C libraries do not include the hostname resolver routines in
1171 @c their static libraries. If Emacs was linked statically, and was not
1172 @c linked with the resolver libraries, it will not be able to get to any
1173 @c machines off the local network. This is characterized by being able
1174 @c to reach someplace with a raw ip number, but not its hostname
1175 @c (@url{http://129.79.254.191/} works, but
1176 @c @url{http://www.cs.indiana.edu/} doesn't). This used to happen on
1177 @c SunOS4 and Ultrix, but is now probably now rare. If Emacs can't be
1178 @c rebuilt linked against the resolver library, it can use the external
1179 @c @command{nslookup} program instead.
1181 @c @defopt url-gateway-broken-resolution
1182 @c @cindex @code{nslookup} program
1183 @c @cindex program, @code{nslookup}
1184 @c If non-@code{nil}, this variable says to use the program specified by
1185 @c @code{url-gateway-nslookup-program} program to do hostname resolution.
1188 @c @defopt url-gateway-nslookup-program
1189 @c The name of the program to do hostname lookup if Emacs can't do it
1190 @c directly. This program should expect a single argument on the command
1191 @c line---the hostname to resolve---and should produce output similar to
1192 @c the standard Unix @command{nslookup} program:
1194 @c Name: www.cs.indiana.edu
1195 @c Address: 129.79.254.191
1202 @findex url-do-setup
1203 The library can maintain a global history list tracking URLs accessed.
1204 URL completion can be done from it. The history mechanism is set up
1205 automatically via @code{url-do-setup} when it is configured to be on.
1206 Note that the size of the history list is currently not limited.
1208 @vindex url-history-hash-table
1209 The history ``list'' is actually a hash table,
1210 @code{url-history-hash-table}. It contains access times keyed by URL
1211 strings. The times are in the format returned by @code{current-time}.
1213 @defun url-history-update-url url time
1214 This function updates the history table with an entry for @var{url}
1215 accessed at the given @var{time}.
1218 @defopt url-history-track
1219 If non-@code{nil}, the library will keep track of all the URLs
1220 accessed. If it is @code{t}, the list is saved to disk at the end of
1221 each Emacs session. The default is @code{nil}.
1224 @defopt url-history-file
1225 The file storing the history list between sessions. It defaults to
1226 @file{history} in @code{url-configuration-directory}.
1229 @defopt url-history-save-interval
1230 @findex url-history-setup-save-timer
1231 The number of seconds between automatic saves of the history list.
1232 Default is one hour. Note that if you change this variable directly,
1233 rather than using Custom, after @code{url-do-setup} has been run, you
1234 need to run the function @code{url-history-setup-save-timer}.
1237 @defun url-history-parse-history &optional fname
1238 Parses the history file @var{fname} (default @code{url-history-file})
1239 and sets up the history list.
1242 @defun url-history-save-history &optional fname
1243 Saves the current history to file @var{fname} (default
1244 @code{url-history-file}).
1247 @defun url-completion-function string predicate function
1248 You can use this function to do completion of URLs from the history.
1252 @chapter Customization
1254 @cindex environment variables
1255 The following environment variables affect the @code{url} library's
1256 operation at startup.
1261 @vindex url-temporary-directory
1262 If this is defined, @var{url-temporary-directory} is initialized from
1266 The following user options affect the general operation of
1269 @defopt url-configuration-directory
1270 @cindex configuration files
1271 The value of this variable specifies the name of the directory where
1272 the @code{url} library stores its various configuration files, cache
1275 The default value specifies a subdirectory named @file{url/} in the
1276 standard Emacs user data directory specified by the variable
1277 @code{user-emacs-directory} (normally @file{~/.emacs.d}). However,
1278 the old default was @file{~/.url}, and this directory is used instead
1284 Specifies the types of debug messages which are logged to
1285 the @file{*URL-DEBUG*} buffer.
1286 @code{t} means log all messages.
1287 A number means log all messages and show them with @code{message}.
1288 It may also be a list of the types of messages to be logged.
1290 @defopt url-personal-mail-address
1292 @defopt url-privacy-level
1294 @defopt url-uncompressor-alist
1296 @defopt url-passwd-entry-func
1298 @defopt url-standalone-mode
1300 @defopt url-bad-port-list
1302 @defopt url-max-password-attempts
1304 @defopt url-temporary-directory
1306 @defopt url-show-status
1308 @defopt url-confirmation-func
1309 The function to use for asking yes or no functions. This is normally
1310 either @code{y-or-n-p} or @code{yes-or-no-p}, but could be another
1311 function taking a single argument (the prompt) and returning @code{t}
1312 only if an affirmative answer is given.
1314 @defopt url-gateway-method
1315 @c fixme: describe gatewaying
1316 A symbol specifying the type of gateway support to use for connections
1317 from the local machine. The supported methods are:
1321 Run telnet in a subprocess to connect;
1323 Rlogin to another machine to connect;
1325 Connect through a socks server;
1333 @defopt url-user-agent
1334 The User Agent string used for sending HTTP/HTTPS requests. The value
1335 should be a string or a function of no arguments that returns a
1336 string. The default value is @w{@samp{User-Agent: @var{package-name}
1337 URL/Emacs}}, where @var{package-name} is the value of
1338 @code{url-package-name} and its version, if they are non-@code{nil}.
1341 @node GNU Free Documentation License
1342 @appendix GNU Free Documentation License
1343 @include doclicense.texi
1345 @node Function Index
1346 @unnumbered Command and Function Index
1349 @node Variable Index
1350 @unnumbered Variable Index
1354 @unnumbered Concept Index