utf-8-codec, utf-16-codec, latin-1-codec - predefined codecs
(import (rnrs)) ;R6RS
(import (rnrs io ports)) ;R6RS
(utf-8-codec)
(utf-16-codec)
(latin-1-codec)
These are predefined codecs for the ISO 8859-1, UTF-8, and UTF-16 encoding
schemes.
- The latin-1 codec
- The latin-1 codec can only handle characters in the range [#\x00, #\xFF].
The transcoding is a one-to-one mapping between the byte values and
character objects in the sense of char->integer(3scm) and
integer->char(3scm).
- When transcoding in the input direction, each byte is read as one
character.
- When transcoding in the output direction, characters less than #\x100 are
written as one byte. Otherwise the behavior depends on the transcoder's
error handling mode. In raise mode an
&i/o-encoding(3scm) condition is raised; in ignore mode
the character is dropped; and in replace mode a \x3F byte (#\?) is
written.
- The UTF-8 codec
- The UTF-8 transcoder can handle all Unicode characters. For characters in
the range [#\x00, #\x7F] there is a one-to-one mapping between byte values
and character object in the sense of char->integer(3scm) and
integer->char(3scm).
- For characters above #\x7F, UTF-8 tags the initial byte with a length and
then follows that byte with a number of continuation bytes. There is a
specific pattern that must be followed and any deviation is an error. The
same applies to short sequences. Errors are handled according to the
transcoder's error handling mode. In raise mode an
&i/o-decoding(3scm) condition is raised; in ignore mode
the character is dropped; and in replace mode an #\xFFFD character
is read.
- Implementations vary in how many #\xFFFD characters are produced for an
invalid sequence.
- There are no encoding errors with the UTF-8 codec.
- The UTF-16 codec
- The UTF-16 codec works with pairs of bytes. The bytes are 16-bit
quantities encoded in either big endian or little endian. A byte order
mark (BOM) is either the byte sequence #xFE, #xFF for big endian or #xFF,
#xFE for little endian.
- The 16-bit quantities mostly map directly to character objects
integer->char(3scm).Thereisanexceptionfortherange
[#xD800, #xDFFF], which are called the surrogate pairs. A pair of these
16-bit quantities are used to encode characters that would otherwise not
fit in the 16-bit range.
- Short byte sequences and errors in surrogate pairs are errors. In
raise mode an &i/o-decoding(3scm) condition is raised;
in ignore mode the character is dropped; and in replace mode
an #\xFFFD character is read.
- There are no encoding errors with the UTF-16 codec.
A call to any of these procedures returns a value that is equal in the sense of
eqv?(3scm) to the result of any other call to the same procedure.
;; This is like utf8->string
(let* ((p (open-bytevector-input-port #vu8(194 169)))
(tc (make-transcoder (utf-8-codec)))
(tp (transcoded-port p tc)))
(get-string-all tp))
=> "©"
These procedures are used when creating transcoders.
These procedures can raise exceptions with the following condition types:
- &assertion (R6RS)
- The wrong number of arguments was passed.
These procedures first appeared in R6RS as part of the reworked I/O system.