Chris@4: <?xml version="1.0"?> <!-- -*- sgml -*- -->
Chris@4: <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
Chris@4:   "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd"[
Chris@4: 
Chris@4: <!-- various strings, dates etc. common to all docs -->
Chris@4: <!ENTITY % common-ents SYSTEM "entities.xml"> %common-ents;
Chris@4: ]>
Chris@4: 
Chris@4: <book lang="en" id="userman" xreflabel="bzip2 Manual">
Chris@4: 
Chris@4:  <bookinfo>
Chris@4:   <title>bzip2 and libbzip2, version 1.0.6</title>
Chris@4:   <subtitle>A program and library for data compression</subtitle>
Chris@4:   <copyright>
Chris@4:    <year>&bz-lifespan;</year>
Chris@4:    <holder>Julian Seward</holder>
Chris@4:   </copyright>
Chris@4:   <releaseinfo>Version &bz-version; of &bz-date;</releaseinfo>
Chris@4: 
Chris@4:   <authorgroup>
Chris@4:    <author>
Chris@4:     <firstname>Julian</firstname>
Chris@4:     <surname>Seward</surname>
Chris@4:     <affiliation>
Chris@4:      <orgname>&bz-url;</orgname>
Chris@4:     </affiliation>
Chris@4:    </author>
Chris@4:   </authorgroup>
Chris@4: 
Chris@4:   <legalnotice>
Chris@4: 
Chris@4:   <para>This program, <computeroutput>bzip2</computeroutput>, the
Chris@4:   associated library <computeroutput>libbzip2</computeroutput>, and
Chris@4:   all documentation, are copyright &copy; &bz-lifespan; Julian Seward.
Chris@4:   All rights reserved.</para>
Chris@4: 
Chris@4:   <para>Redistribution and use in source and binary forms, with
Chris@4:   or without modification, are permitted provided that the
Chris@4:   following conditions are met:</para>
Chris@4: 
Chris@4:   <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:    <listitem><para>Redistributions of source code must retain the
Chris@4:    above copyright notice, this list of conditions and the
Chris@4:    following disclaimer.</para></listitem>
Chris@4: 
Chris@4:    <listitem><para>The origin of this software must not be
Chris@4:    misrepresented; you must not claim that you wrote the original
Chris@4:    software.  If you use this software in a product, an
Chris@4:    acknowledgment in the product documentation would be
Chris@4:    appreciated but is not required.</para></listitem>
Chris@4: 
Chris@4:    <listitem><para>Altered source versions must be plainly marked
Chris@4:    as such, and must not be misrepresented as being the original
Chris@4:    software.</para></listitem>
Chris@4: 
Chris@4:    <listitem><para>The name of the author may not be used to
Chris@4:    endorse or promote products derived from this software without
Chris@4:    specific prior written permission.</para></listitem>
Chris@4: 
Chris@4:   </itemizedlist>
Chris@4: 
Chris@4:   <para>THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY
Chris@4:   EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
Chris@4:   THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
Chris@4:   PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE
Chris@4:   AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
Chris@4:   EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
Chris@4:   TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
Chris@4:   DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
Chris@4:   ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
Chris@4:   LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
Chris@4:   IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF
Chris@4:   THE POSSIBILITY OF SUCH DAMAGE.</para>
Chris@4: 
Chris@4:  <para>PATENTS: To the best of my knowledge,
Chris@4:  <computeroutput>bzip2</computeroutput> and
Chris@4:  <computeroutput>libbzip2</computeroutput> do not use any patented
Chris@4:  algorithms.  However, I do not have the resources to carry
Chris@4:  out a patent search.  Therefore I cannot give any guarantee of
Chris@4:  the above statement.
Chris@4:  </para>
Chris@4: 
Chris@4: </legalnotice>
Chris@4: 
Chris@4: </bookinfo>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <chapter id="intro" xreflabel="Introduction">
Chris@4: <title>Introduction</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> compresses files
Chris@4: using the Burrows-Wheeler block-sorting text compression
Chris@4: algorithm, and Huffman coding.  Compression is generally
Chris@4: considerably better than that achieved by more conventional
Chris@4: LZ77/LZ78-based compressors, and approaches the performance of
Chris@4: the PPM family of statistical compressors.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> is built on top of
Chris@4: <computeroutput>libbzip2</computeroutput>, a flexible library for
Chris@4: handling compressed data in the
Chris@4: <computeroutput>bzip2</computeroutput> format.  This manual
Chris@4: describes both how to use the program and how to work with the
Chris@4: library interface.  Most of the manual is devoted to this
Chris@4: library, not the program, which is good news if your interest is
Chris@4: only in the program.</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para><xref linkend="using"/> describes how to use
Chris@4:  <computeroutput>bzip2</computeroutput>; this is the only part
Chris@4:  you need to read if you just want to know how to operate the
Chris@4:  program.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><xref linkend="libprog"/> describes the
Chris@4:  programming interfaces in detail, and</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><xref linkend="misc"/> records some
Chris@4:  miscellaneous notes which I thought ought to be recorded
Chris@4:  somewhere.</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: </chapter>
Chris@4: 
Chris@4: 
Chris@4: <chapter id="using" xreflabel="How to use bzip2">
Chris@4: <title>How to use bzip2</title>
Chris@4: 
Chris@4: <para>This chapter contains a copy of the
Chris@4: <computeroutput>bzip2</computeroutput> man page, and nothing
Chris@4: else.</para>
Chris@4: 
Chris@4: <sect1 id="name" xreflabel="NAME">
Chris@4: <title>NAME</title>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzip2</computeroutput>,
Chris@4:   <computeroutput>bunzip2</computeroutput> - a block-sorting file
Chris@4:   compressor, v1.0.6</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzcat</computeroutput> -
Chris@4:    decompresses files to stdout</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzip2recover</computeroutput> -
Chris@4:    recovers data from damaged bzip2 files</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="synopsis" xreflabel="SYNOPSIS">
Chris@4: <title>SYNOPSIS</title>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzip2</computeroutput> [
Chris@4:   -cdfkqstvzVL123456789 ] [ filenames ...  ]</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bunzip2</computeroutput> [
Chris@4:   -fkvsVL ] [ filenames ...  ]</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzcat</computeroutput> [ -s ] [
Chris@4:   filenames ...  ]</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzip2recover</computeroutput>
Chris@4:   filename</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="description" xreflabel="DESCRIPTION">
Chris@4: <title>DESCRIPTION</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> compresses files
Chris@4: using the Burrows-Wheeler block sorting text compression
Chris@4: algorithm, and Huffman coding.  Compression is generally
Chris@4: considerably better than that achieved by more conventional
Chris@4: LZ77/LZ78-based compressors, and approaches the performance of
Chris@4: the PPM family of statistical compressors.</para>
Chris@4: 
Chris@4: <para>The command-line options are deliberately very similar to
Chris@4: those of GNU <computeroutput>gzip</computeroutput>, but they are
Chris@4: not identical.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> expects a list of
Chris@4: file names to accompany the command-line flags.  Each file is
Chris@4: replaced by a compressed version of itself, with the name
Chris@4: <computeroutput>original_name.bz2</computeroutput>.  Each
Chris@4: compressed file has the same modification date, permissions, and,
Chris@4: when possible, ownership as the corresponding original, so that
Chris@4: these properties can be correctly restored at decompression time.
Chris@4: File name handling is naive in the sense that there is no
Chris@4: mechanism for preserving original file names, permissions,
Chris@4: ownerships or dates in filesystems which lack these concepts, or
Chris@4: have serious file name length restrictions, such as
Chris@4: MS-DOS.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> and
Chris@4: <computeroutput>bunzip2</computeroutput> will by default not
Chris@4: overwrite existing files.  If you want this to happen, specify
Chris@4: the <computeroutput>-f</computeroutput> flag.</para>
Chris@4: 
Chris@4: <para>If no file names are specified,
Chris@4: <computeroutput>bzip2</computeroutput> compresses from standard
Chris@4: input to standard output.  In this case,
Chris@4: <computeroutput>bzip2</computeroutput> will decline to write
Chris@4: compressed output to a terminal, as this would be entirely
Chris@4: incomprehensible and therefore pointless.</para>
Chris@4: 
Chris@4: <para><computeroutput>bunzip2</computeroutput> (or
Chris@4: <computeroutput>bzip2 -d</computeroutput>) decompresses all
Chris@4: specified files.  Files which were not created by
Chris@4: <computeroutput>bzip2</computeroutput> will be detected and
Chris@4: ignored, and a warning issued.
Chris@4: <computeroutput>bzip2</computeroutput> attempts to guess the
Chris@4: filename for the decompressed file from that of the compressed
Chris@4: file as follows:</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>filename.bz2 </computeroutput>
Chris@4:   becomes
Chris@4:   <computeroutput>filename</computeroutput></para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>filename.bz </computeroutput>
Chris@4:   becomes
Chris@4:   <computeroutput>filename</computeroutput></para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>filename.tbz2</computeroutput>
Chris@4:   becomes
Chris@4:   <computeroutput>filename.tar</computeroutput></para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>filename.tbz </computeroutput>
Chris@4:   becomes
Chris@4:   <computeroutput>filename.tar</computeroutput></para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>anyothername </computeroutput>
Chris@4:   becomes
Chris@4:   <computeroutput>anyothername.out</computeroutput></para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: <para>If the file does not end in one of the recognised endings,
Chris@4: <computeroutput>.bz2</computeroutput>,
Chris@4: <computeroutput>.bz</computeroutput>,
Chris@4: <computeroutput>.tbz2</computeroutput> or
Chris@4: <computeroutput>.tbz</computeroutput>,
Chris@4: <computeroutput>bzip2</computeroutput> complains that it cannot
Chris@4: guess the name of the original file, and uses the original name
Chris@4: with <computeroutput>.out</computeroutput> appended.</para>
Chris@4: 
Chris@4: <para>As with compression, supplying no filenames causes
Chris@4: decompression from standard input to standard output.</para>
Chris@4: 
Chris@4: <para><computeroutput>bunzip2</computeroutput> will correctly
Chris@4: decompress a file which is the concatenation of two or more
Chris@4: compressed files.  The result is the concatenation of the
Chris@4: corresponding uncompressed files.  Integrity testing
Chris@4: (<computeroutput>-t</computeroutput>) of concatenated compressed
Chris@4: files is also supported.</para>
Chris@4: 
Chris@4: <para>You can also compress or decompress files to the standard
Chris@4: output by giving the <computeroutput>-c</computeroutput> flag.
Chris@4: Multiple files may be compressed and decompressed like this.  The
Chris@4: resulting outputs are fed sequentially to stdout.  Compression of
Chris@4: multiple files in this manner generates a stream containing
Chris@4: multiple compressed file representations.  Such a stream can be
Chris@4: decompressed correctly only by
Chris@4: <computeroutput>bzip2</computeroutput> version 0.9.0 or later.
Chris@4: Earlier versions of <computeroutput>bzip2</computeroutput> will
Chris@4: stop after decompressing the first file in the stream.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzcat</computeroutput> (or
Chris@4: <computeroutput>bzip2 -dc</computeroutput>) decompresses all
Chris@4: specified files to the standard output.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> will read arguments
Chris@4: from the environment variables
Chris@4: <computeroutput>BZIP2</computeroutput> and
Chris@4: <computeroutput>BZIP</computeroutput>, in that order, and will
Chris@4: process them before any arguments read from the command line.
Chris@4: This gives a convenient way to supply default arguments.</para>
Chris@4: 
Chris@4: <para>Compression is always performed, even if the compressed
Chris@4: file is slightly larger than the original.  Files of less than
Chris@4: about one hundred bytes tend to get larger, since the compression
Chris@4: mechanism has a constant overhead in the region of 50 bytes.
Chris@4: Random data (including the output of most file compressors) is
Chris@4: coded at about 8.05 bits per byte, giving an expansion of around
Chris@4: 0.5%.</para>
Chris@4: 
Chris@4: <para>As a self-check for your protection,
Chris@4: <computeroutput>bzip2</computeroutput> uses 32-bit CRCs to make
Chris@4: sure that the decompressed version of a file is identical to the
Chris@4: original.  This guards against corruption of the compressed data,
Chris@4: and against undetected bugs in
Chris@4: <computeroutput>bzip2</computeroutput> (hopefully very unlikely).
Chris@4: The chances of data corruption going undetected is microscopic,
Chris@4: about one chance in four billion for each file processed.  Be
Chris@4: aware, though, that the check occurs upon decompression, so it
Chris@4: can only tell you that something is wrong.  It can't help you
Chris@4: recover the original uncompressed data.  You can use
Chris@4: <computeroutput>bzip2recover</computeroutput> to try to recover
Chris@4: data from damaged files.</para>
Chris@4: 
Chris@4: <para>Return values: 0 for a normal exit, 1 for environmental
Chris@4: problems (file not found, invalid flags, I/O errors, etc.), 2
Chris@4: to indicate a corrupt compressed file, 3 for an internal
Chris@4: consistency error (eg, bug) which caused
Chris@4: <computeroutput>bzip2</computeroutput> to panic.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="options" xreflabel="OPTIONS">
Chris@4: <title>OPTIONS</title>
Chris@4: 
Chris@4: <variablelist>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-c --stdout</computeroutput></term>
Chris@4:  <listitem><para>Compress or decompress to standard
Chris@4:   output.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-d --decompress</computeroutput></term>
Chris@4:  <listitem><para>Force decompression.
Chris@4:   <computeroutput>bzip2</computeroutput>,
Chris@4:   <computeroutput>bunzip2</computeroutput> and
Chris@4:   <computeroutput>bzcat</computeroutput> are really the same
Chris@4:   program, and the decision about what actions to take is done on
Chris@4:   the basis of which name is used.  This flag overrides that
Chris@4:   mechanism, and forces bzip2 to decompress.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-z --compress</computeroutput></term>
Chris@4:  <listitem><para>The complement to
Chris@4:   <computeroutput>-d</computeroutput>: forces compression,
Chris@4:   regardless of the invokation name.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-t --test</computeroutput></term>
Chris@4:  <listitem><para>Check integrity of the specified file(s), but
Chris@4:   don't decompress them.  This really performs a trial
Chris@4:   decompression and throws away the result.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-f --force</computeroutput></term>
Chris@4:  <listitem><para>Force overwrite of output files.  Normally,
Chris@4:   <computeroutput>bzip2</computeroutput> will not overwrite
Chris@4:   existing output files.  Also forces
Chris@4:   <computeroutput>bzip2</computeroutput> to break hard links to
Chris@4:   files, which it otherwise wouldn't do.</para>
Chris@4:   <para><computeroutput>bzip2</computeroutput> normally declines
Chris@4:   to decompress files which don't have the correct magic header
Chris@4:   bytes. If forced (<computeroutput>-f</computeroutput>),
Chris@4:   however, it will pass such files through unmodified. This is
Chris@4:   how GNU <computeroutput>gzip</computeroutput> behaves.</para>
Chris@4:  </listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-k --keep</computeroutput></term>
Chris@4:  <listitem><para>Keep (don't delete) input files during
Chris@4:   compression or decompression.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-s --small</computeroutput></term>
Chris@4:  <listitem><para>Reduce memory usage, for compression,
Chris@4:   decompression and testing.  Files are decompressed and tested
Chris@4:   using a modified algorithm which only requires 2.5 bytes per
Chris@4:   block byte.  This means any file can be decompressed in 2300k
Chris@4:   of memory, albeit at about half the normal speed.</para>
Chris@4:   <para>During compression, <computeroutput>-s</computeroutput>
Chris@4:   selects a block size of 200k, which limits memory use to around
Chris@4:   the same figure, at the expense of your compression ratio.  In
Chris@4:   short, if your machine is low on memory (8 megabytes or less),
Chris@4:   use <computeroutput>-s</computeroutput> for everything.  See
Chris@4:   <xref linkend="memory-management"/> below.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-q --quiet</computeroutput></term>
Chris@4:  <listitem><para>Suppress non-essential warning messages.
Chris@4:   Messages pertaining to I/O errors and other critical events
Chris@4:   will not be suppressed.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-v --verbose</computeroutput></term>
Chris@4:  <listitem><para>Verbose mode -- show the compression ratio for
Chris@4:   each file processed.  Further
Chris@4:   <computeroutput>-v</computeroutput>'s increase the verbosity
Chris@4:   level, spewing out lots of information which is primarily of
Chris@4:   interest for diagnostic purposes.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-L --license -V --version</computeroutput></term>
Chris@4:  <listitem><para>Display the software version, license terms and
Chris@4:   conditions.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>-1</computeroutput> (or
Chris@4:  <computeroutput>--fast</computeroutput>) to
Chris@4:  <computeroutput>-9</computeroutput> (or
Chris@4:  <computeroutput>-best</computeroutput>)</term>
Chris@4:  <listitem><para>Set the block size to 100 k, 200 k ...  900 k
Chris@4:   when compressing.  Has no effect when decompressing.  See <xref
Chris@4:   linkend="memory-management" /> below.  The
Chris@4:   <computeroutput>--fast</computeroutput> and
Chris@4:   <computeroutput>--best</computeroutput> aliases are primarily
Chris@4:   for GNU <computeroutput>gzip</computeroutput> compatibility.
Chris@4:   In particular, <computeroutput>--fast</computeroutput> doesn't
Chris@4:   make things significantly faster.  And
Chris@4:   <computeroutput>--best</computeroutput> merely selects the
Chris@4:   default behaviour.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>--</computeroutput></term>
Chris@4:  <listitem><para>Treats all subsequent arguments as file names,
Chris@4:   even if they start with a dash.  This is so you can handle
Chris@4:   files with names beginning with a dash, for example:
Chris@4:   <computeroutput>bzip2 --
Chris@4:   -myfilename</computeroutput>.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:  <term><computeroutput>--repetitive-fast</computeroutput></term>
Chris@4:  <term><computeroutput>--repetitive-best</computeroutput></term>
Chris@4:  <listitem><para>These flags are redundant in versions 0.9.5 and
Chris@4:   above.  They provided some coarse control over the behaviour of
Chris@4:   the sorting algorithm in earlier versions, which was sometimes
Chris@4:   useful.  0.9.5 and above have an improved algorithm which
Chris@4:   renders these flags irrelevant.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4: </variablelist>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="memory-management" xreflabel="MEMORY MANAGEMENT">
Chris@4: <title>MEMORY MANAGEMENT</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> compresses large
Chris@4: files in blocks.  The block size affects both the compression
Chris@4: ratio achieved, and the amount of memory needed for compression
Chris@4: and decompression.  The flags <computeroutput>-1</computeroutput>
Chris@4: through <computeroutput>-9</computeroutput> specify the block
Chris@4: size to be 100,000 bytes through 900,000 bytes (the default)
Chris@4: respectively.  At decompression time, the block size used for
Chris@4: compression is read from the header of the compressed file, and
Chris@4: <computeroutput>bunzip2</computeroutput> then allocates itself
Chris@4: just enough memory to decompress the file.  Since block sizes are
Chris@4: stored in compressed files, it follows that the flags
Chris@4: <computeroutput>-1</computeroutput> to
Chris@4: <computeroutput>-9</computeroutput> are irrelevant to and so
Chris@4: ignored during decompression.</para>
Chris@4: 
Chris@4: <para>Compression and decompression requirements, in bytes, can be
Chris@4: estimated as:</para>
Chris@4: <programlisting>
Chris@4: Compression:   400k + ( 8 x block size )
Chris@4: 
Chris@4: Decompression: 100k + ( 4 x block size ), or
Chris@4:                100k + ( 2.5 x block size )
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Larger block sizes give rapidly diminishing marginal
Chris@4: returns.  Most of the compression comes from the first two or
Chris@4: three hundred k of block size, a fact worth bearing in mind when
Chris@4: using <computeroutput>bzip2</computeroutput> on small machines.
Chris@4: It is also important to appreciate that the decompression memory
Chris@4: requirement is set at compression time by the choice of block
Chris@4: size.</para>
Chris@4: 
Chris@4: <para>For files compressed with the default 900k block size,
Chris@4: <computeroutput>bunzip2</computeroutput> will require about 3700
Chris@4: kbytes to decompress.  To support decompression of any file on a
Chris@4: 4 megabyte machine, <computeroutput>bunzip2</computeroutput> has
Chris@4: an option to decompress using approximately half this amount of
Chris@4: memory, about 2300 kbytes.  Decompression speed is also halved,
Chris@4: so you should use this option only where necessary.  The relevant
Chris@4: flag is <computeroutput>-s</computeroutput>.</para>
Chris@4: 
Chris@4: <para>In general, try and use the largest block size memory
Chris@4: constraints allow, since that maximises the compression achieved.
Chris@4: Compression and decompression speed are virtually unaffected by
Chris@4: block size.</para>
Chris@4: 
Chris@4: <para>Another significant point applies to files which fit in a
Chris@4: single block -- that means most files you'd encounter using a
Chris@4: large block size.  The amount of real memory touched is
Chris@4: proportional to the size of the file, since the file is smaller
Chris@4: than a block.  For example, compressing a file 20,000 bytes long
Chris@4: with the flag <computeroutput>-9</computeroutput> will cause the
Chris@4: compressor to allocate around 7600k of memory, but only touch
Chris@4: 400k + 20000 * 8 = 560 kbytes of it.  Similarly, the decompressor
Chris@4: will allocate 3700k but only touch 100k + 20000 * 4 = 180
Chris@4: kbytes.</para>
Chris@4: 
Chris@4: <para>Here is a table which summarises the maximum memory usage
Chris@4: for different block sizes.  Also recorded is the total compressed
Chris@4: size for 14 files of the Calgary Text Compression Corpus
Chris@4: totalling 3,141,622 bytes.  This column gives some feel for how
Chris@4: compression varies with block size.  These figures tend to
Chris@4: understate the advantage of larger block sizes for larger files,
Chris@4: since the Corpus is dominated by smaller files.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4:         Compress   Decompress   Decompress   Corpus
Chris@4: Flag     usage      usage       -s usage     Size
Chris@4: 
Chris@4:  -1      1200k       500k         350k      914704
Chris@4:  -2      2000k       900k         600k      877703
Chris@4:  -3      2800k      1300k         850k      860338
Chris@4:  -4      3600k      1700k        1100k      846899
Chris@4:  -5      4400k      2100k        1350k      845160
Chris@4:  -6      5200k      2500k        1600k      838626
Chris@4:  -7      6100k      2900k        1850k      834096
Chris@4:  -8      6800k      3300k        2100k      828642
Chris@4:  -9      7600k      3700k        2350k      828642
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="recovering" xreflabel="RECOVERING DATA FROM DAMAGED FILES">
Chris@4: <title>RECOVERING DATA FROM DAMAGED FILES</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> compresses files in
Chris@4: blocks, usually 900kbytes long.  Each block is handled
Chris@4: independently.  If a media or transmission error causes a
Chris@4: multi-block <computeroutput>.bz2</computeroutput> file to become
Chris@4: damaged, it may be possible to recover data from the undamaged
Chris@4: blocks in the file.</para>
Chris@4: 
Chris@4: <para>The compressed representation of each block is delimited by
Chris@4: a 48-bit pattern, which makes it possible to find the block
Chris@4: boundaries with reasonable certainty.  Each block also carries
Chris@4: its own 32-bit CRC, so damaged blocks can be distinguished from
Chris@4: undamaged ones.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2recover</computeroutput> is a simple
Chris@4: program whose purpose is to search for blocks in
Chris@4: <computeroutput>.bz2</computeroutput> files, and write each block
Chris@4: out into its own <computeroutput>.bz2</computeroutput> file.  You
Chris@4: can then use <computeroutput>bzip2 -t</computeroutput> to test
Chris@4: the integrity of the resulting files, and decompress those which
Chris@4: are undamaged.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2recover</computeroutput> takes a
Chris@4: single argument, the name of the damaged file, and writes a
Chris@4: number of files <computeroutput>rec0001file.bz2</computeroutput>,
Chris@4: <computeroutput>rec0002file.bz2</computeroutput>, etc, containing
Chris@4: the extracted blocks.  The output filenames are designed so that
Chris@4: the use of wildcards in subsequent processing -- for example,
Chris@4: <computeroutput>bzip2 -dc rec*file.bz2 &#62;
Chris@4: recovered_data</computeroutput> -- lists the files in the correct
Chris@4: order.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2recover</computeroutput> should be of
Chris@4: most use dealing with large <computeroutput>.bz2</computeroutput>
Chris@4: files, as these will contain many blocks.  It is clearly futile
Chris@4: to use it on damaged single-block files, since a damaged block
Chris@4: cannot be recovered.  If you wish to minimise any potential data
Chris@4: loss through media or transmission errors, you might consider
Chris@4: compressing with a smaller block size.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="performance" xreflabel="PERFORMANCE NOTES">
Chris@4: <title>PERFORMANCE NOTES</title>
Chris@4: 
Chris@4: <para>The sorting phase of compression gathers together similar
Chris@4: strings in the file.  Because of this, files containing very long
Chris@4: runs of repeated symbols, like "aabaabaabaab ..."  (repeated
Chris@4: several hundred times) may compress more slowly than normal.
Chris@4: Versions 0.9.5 and above fare much better than previous versions
Chris@4: in this respect.  The ratio between worst-case and average-case
Chris@4: compression time is in the region of 10:1.  For previous
Chris@4: versions, this figure was more like 100:1.  You can use the
Chris@4: <computeroutput>-vvvv</computeroutput> option to monitor progress
Chris@4: in great detail, if you want.</para>
Chris@4: 
Chris@4: <para>Decompression speed is unaffected by these
Chris@4: phenomena.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> usually allocates
Chris@4: several megabytes of memory to operate in, and then charges all
Chris@4: over it in a fairly random fashion.  This means that performance,
Chris@4: both for compressing and decompressing, is largely determined by
Chris@4: the speed at which your machine can service cache misses.
Chris@4: Because of this, small changes to the code to reduce the miss
Chris@4: rate have been observed to give disproportionately large
Chris@4: performance improvements.  I imagine
Chris@4: <computeroutput>bzip2</computeroutput> will perform best on
Chris@4: machines with very large caches.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="caveats" xreflabel="CAVEATS">
Chris@4: <title>CAVEATS</title>
Chris@4: 
Chris@4: <para>I/O error messages are not as helpful as they could be.
Chris@4: <computeroutput>bzip2</computeroutput> tries hard to detect I/O
Chris@4: errors and exit cleanly, but the details of what the problem is
Chris@4: sometimes seem rather misleading.</para>
Chris@4: 
Chris@4: <para>This manual page pertains to version &bz-version; of
Chris@4: <computeroutput>bzip2</computeroutput>.  Compressed data created by
Chris@4: this version is entirely forwards and backwards compatible with the
Chris@4: previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
Chris@4: 1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
Chris@4: above can correctly decompress multiple concatenated compressed files.
Chris@4: 0.1pl2 cannot do this; it will stop after decompressing just the first
Chris@4: file in the stream.</para>
Chris@4: 
Chris@4: <para><computeroutput>bzip2recover</computeroutput> versions
Chris@4: prior to 1.0.2 used 32-bit integers to represent bit positions in
Chris@4: compressed files, so it could not handle compressed files more
Chris@4: than 512 megabytes long.  Versions 1.0.2 and above use 64-bit ints
Chris@4: on some platforms which support them (GNU supported targets, and
Chris@4: Windows). To establish whether or not
Chris@4: <computeroutput>bzip2recover</computeroutput> was built with such
Chris@4: a limitation, run it without arguments. In any event you can
Chris@4: build yourself an unlimited version if you can recompile it with
Chris@4: <computeroutput>MaybeUInt64</computeroutput> set to be an
Chris@4: unsigned 64-bit integer.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="author" xreflabel="AUTHOR">
Chris@4: <title>AUTHOR</title>
Chris@4: 
Chris@4: <para>Julian Seward,
Chris@4: <computeroutput>&bz-email;</computeroutput></para>
Chris@4: 
Chris@4: <para>The ideas embodied in
Chris@4: <computeroutput>bzip2</computeroutput> are due to (at least) the
Chris@4: following people: Michael Burrows and David Wheeler (for the
Chris@4: block sorting transformation), David Wheeler (again, for the
Chris@4: Huffman coder), Peter Fenwick (for the structured coding model in
Chris@4: the original <computeroutput>bzip</computeroutput>, and many
Chris@4: refinements), and Alistair Moffat, Radford Neal and Ian Witten
Chris@4: (for the arithmetic coder in the original
Chris@4: <computeroutput>bzip</computeroutput>).  I am much indebted for
Chris@4: their help, support and advice.  See the manual in the source
Chris@4: distribution for pointers to sources of documentation.  Christian
Chris@4: von Roques encouraged me to look for faster sorting algorithms,
Chris@4: so as to speed up compression.  Bela Lubkin encouraged me to
Chris@4: improve the worst-case compression performance.  
Chris@4: Donna Robinson XMLised the documentation.
Chris@4: Many people sent
Chris@4: patches, helped with portability problems, lent machines, gave
Chris@4: advice and were generally helpful.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: </chapter>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <chapter id="libprog" xreflabel="Programming with libbzip2">
Chris@4: <title>
Chris@4: Programming with <computeroutput>libbzip2</computeroutput>
Chris@4: </title>
Chris@4: 
Chris@4: <para>This chapter describes the programming interface to
Chris@4: <computeroutput>libbzip2</computeroutput>.</para>
Chris@4: 
Chris@4: <para>For general background information, particularly about
Chris@4: memory use and performance aspects, you'd be well advised to read
Chris@4: <xref linkend="using"/> as well.</para>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="top-level" xreflabel="Top-level structure">
Chris@4: <title>Top-level structure</title>
Chris@4: 
Chris@4: <para><computeroutput>libbzip2</computeroutput> is a flexible
Chris@4: library for compressing and decompressing data in the
Chris@4: <computeroutput>bzip2</computeroutput> data format.  Although
Chris@4: packaged as a single entity, it helps to regard the library as
Chris@4: three separate parts: the low level interface, and the high level
Chris@4: interface, and some utility functions.</para>
Chris@4: 
Chris@4: <para>The structure of
Chris@4: <computeroutput>libbzip2</computeroutput>'s interfaces is similar
Chris@4: to that of Jean-loup Gailly's and Mark Adler's excellent
Chris@4: <computeroutput>zlib</computeroutput> library.</para>
Chris@4: 
Chris@4: <para>All externally visible symbols have names beginning
Chris@4: <computeroutput>BZ2_</computeroutput>.  This is new in version
Chris@4: 1.0.  The intention is to minimise pollution of the namespaces of
Chris@4: library clients.</para>
Chris@4: 
Chris@4: <para>To use any part of the library, you need to
Chris@4: <computeroutput>#include &lt;bzlib.h&gt;</computeroutput>
Chris@4: into your sources.</para>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="ll-summary" xreflabel="Low-level summary">
Chris@4: <title>Low-level summary</title>
Chris@4: 
Chris@4: <para>This interface provides services for compressing and
Chris@4: decompressing data in memory.  There's no provision for dealing
Chris@4: with files, streams or any other I/O mechanisms, just straight
Chris@4: memory-to-memory work.  In fact, this part of the library can be
Chris@4: compiled without inclusion of
Chris@4: <computeroutput>stdio.h</computeroutput>, which may be helpful
Chris@4: for embedded applications.</para>
Chris@4: 
Chris@4: <para>The low-level part of the library has no global variables
Chris@4: and is therefore thread-safe.</para>
Chris@4: 
Chris@4: <para>Six routines make up the low level interface:
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>,
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>, and
Chris@4: <computeroutput>BZ2_bzCompressEnd</computeroutput> for
Chris@4: compression, and a corresponding trio
Chris@4: <computeroutput>BZ2_bzDecompressInit</computeroutput>,
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput> and
Chris@4: <computeroutput>BZ2_bzDecompressEnd</computeroutput> for
Chris@4: decompression.  The <computeroutput>*Init</computeroutput>
Chris@4: functions allocate memory for compression/decompression and do
Chris@4: other initialisations, whilst the
Chris@4: <computeroutput>*End</computeroutput> functions close down
Chris@4: operations and release memory.</para>
Chris@4: 
Chris@4: <para>The real work is done by
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> and
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput>.  These
Chris@4: compress and decompress data from a user-supplied input buffer to
Chris@4: a user-supplied output buffer.  These buffers can be any size;
Chris@4: arbitrary quantities of data are handled by making repeated calls
Chris@4: to these functions.  This is a flexible mechanism allowing a
Chris@4: consumer-pull style of activity, or producer-push, or a mixture
Chris@4: of both.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="hl-summary" xreflabel="High-level summary">
Chris@4: <title>High-level summary</title>
Chris@4: 
Chris@4: <para>This interface provides some handy wrappers around the
Chris@4: low-level interface to facilitate reading and writing
Chris@4: <computeroutput>bzip2</computeroutput> format files
Chris@4: (<computeroutput>.bz2</computeroutput> files).  The routines
Chris@4: provide hooks to facilitate reading files in which the
Chris@4: <computeroutput>bzip2</computeroutput> data stream is embedded
Chris@4: within some larger-scale file structure, or where there are
Chris@4: multiple <computeroutput>bzip2</computeroutput> data streams
Chris@4: concatenated end-to-end.</para>
Chris@4: 
Chris@4: <para>For reading files,
Chris@4: <computeroutput>BZ2_bzReadOpen</computeroutput>,
Chris@4: <computeroutput>BZ2_bzRead</computeroutput>,
Chris@4: <computeroutput>BZ2_bzReadClose</computeroutput> and 
Chris@4: <computeroutput>BZ2_bzReadGetUnused</computeroutput> are
Chris@4: supplied.  For writing files,
Chris@4: <computeroutput>BZ2_bzWriteOpen</computeroutput>,
Chris@4: <computeroutput>BZ2_bzWrite</computeroutput> and
Chris@4: <computeroutput>BZ2_bzWriteFinish</computeroutput> are
Chris@4: available.</para>
Chris@4: 
Chris@4: <para>As with the low-level library, no global variables are used
Chris@4: so the library is per se thread-safe.  However, if I/O errors
Chris@4: occur whilst reading or writing the underlying compressed files,
Chris@4: you may have to consult <computeroutput>errno</computeroutput> to
Chris@4: determine the cause of the error.  In that case, you'd need a C
Chris@4: library which correctly supports
Chris@4: <computeroutput>errno</computeroutput> in a multithreaded
Chris@4: environment.</para>
Chris@4: 
Chris@4: <para>To make the library a little simpler and more portable,
Chris@4: <computeroutput>BZ2_bzReadOpen</computeroutput> and
Chris@4: <computeroutput>BZ2_bzWriteOpen</computeroutput> require you to
Chris@4: pass them file handles (<computeroutput>FILE*</computeroutput>s)
Chris@4: which have previously been opened for reading or writing
Chris@4: respectively.  That avoids portability problems associated with
Chris@4: file operations and file attributes, whilst not being much of an
Chris@4: imposition on the programmer.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="util-fns-summary" xreflabel="Utility functions summary">
Chris@4: <title>Utility functions summary</title>
Chris@4: 
Chris@4: <para>For very simple needs,
Chris@4: <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
Chris@4: <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> are
Chris@4: provided.  These compress data in memory from one buffer to
Chris@4: another buffer in a single function call.  You should assess
Chris@4: whether these functions fulfill your memory-to-memory
Chris@4: compression/decompression requirements before investing effort in
Chris@4: understanding the more general but more complex low-level
Chris@4: interface.</para>
Chris@4: 
Chris@4: <para>Yoshioka Tsuneo
Chris@4: (<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>) has
Chris@4: contributed some functions to give better
Chris@4: <computeroutput>zlib</computeroutput> compatibility.  These
Chris@4: functions are <computeroutput>BZ2_bzopen</computeroutput>,
Chris@4: <computeroutput>BZ2_bzread</computeroutput>,
Chris@4: <computeroutput>BZ2_bzwrite</computeroutput>,
Chris@4: <computeroutput>BZ2_bzflush</computeroutput>,
Chris@4: <computeroutput>BZ2_bzclose</computeroutput>,
Chris@4: <computeroutput>BZ2_bzerror</computeroutput> and
Chris@4: <computeroutput>BZ2_bzlibVersion</computeroutput>.  You may find
Chris@4: these functions more convenient for simple file reading and
Chris@4: writing, than those in the high-level interface.  These functions
Chris@4: are not (yet) officially part of the library, and are minimally
Chris@4: documented here.  If they break, you get to keep all the pieces.
Chris@4: I hope to document them properly when time permits.</para>
Chris@4: 
Chris@4: <para>Yoshioka also contributed modifications to allow the
Chris@4: library to be built as a Windows DLL.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="err-handling" xreflabel="Error handling">
Chris@4: <title>Error handling</title>
Chris@4: 
Chris@4: <para>The library is designed to recover cleanly in all
Chris@4: situations, including the worst-case situation of decompressing
Chris@4: random data.  I'm not 100% sure that it can always do this, so
Chris@4: you might want to add a signal handler to catch segmentation
Chris@4: violations during decompression if you are feeling especially
Chris@4: paranoid.  I would be interested in hearing more about the
Chris@4: robustness of the library to corrupted compressed data.</para>
Chris@4: 
Chris@4: <para>Version 1.0.3 more robust in this respect than any
Chris@4: previous version.  Investigations with Valgrind (a tool for detecting
Chris@4: problems with memory management) indicate
Chris@4: that, at least for the few files I tested, all single-bit errors
Chris@4: in the decompressed data are caught properly, with no
Chris@4: segmentation faults, no uses of uninitialised data, no out of
Chris@4: range reads or writes, and no infinite looping in the decompressor.
Chris@4: So it's certainly pretty robust, although
Chris@4: I wouldn't claim it to be totally bombproof.</para>
Chris@4: 
Chris@4: <para>The file <computeroutput>bzlib.h</computeroutput> contains
Chris@4: all definitions needed to use the library.  In particular, you
Chris@4: should definitely not include
Chris@4: <computeroutput>bzlib_private.h</computeroutput>.</para>
Chris@4: 
Chris@4: <para>In <computeroutput>bzlib.h</computeroutput>, the various
Chris@4: return values are defined.  The following list is not intended as
Chris@4: an exhaustive description of the circumstances in which a given
Chris@4: value may be returned -- those descriptions are given later.
Chris@4: Rather, it is intended to convey the rough meaning of each return
Chris@4: value.  The first five actions are normal and not intended to
Chris@4: denote an error situation.</para>
Chris@4: 
Chris@4: <variablelist>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_OK</computeroutput></term>
Chris@4:   <listitem><para>The requested action was completed
Chris@4:    successfully.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_RUN_OK, BZ_FLUSH_OK,
Chris@4:     BZ_FINISH_OK</computeroutput></term>
Chris@4:   <listitem><para>In 
Chris@4:    <computeroutput>BZ2_bzCompress</computeroutput>, the requested
Chris@4:    flush/finish/nothing-special action was completed
Chris@4:    successfully.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_STREAM_END</computeroutput></term>
Chris@4:   <listitem><para>Compression of data was completed, or the
Chris@4:    logical stream end was detected during
Chris@4:    decompression.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4: </variablelist>
Chris@4: 
Chris@4: <para>The following return values indicate an error of some
Chris@4: kind.</para>
Chris@4: 
Chris@4: <variablelist>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_CONFIG_ERROR</computeroutput></term>
Chris@4:   <listitem><para>Indicates that the library has been improperly
Chris@4:    compiled on your platform -- a major configuration error.
Chris@4:    Specifically, it means that
Chris@4:    <computeroutput>sizeof(char)</computeroutput>,
Chris@4:    <computeroutput>sizeof(short)</computeroutput> and
Chris@4:    <computeroutput>sizeof(int)</computeroutput> are not 1, 2 and
Chris@4:    4 respectively, as they should be.  Note that the library
Chris@4:    should still work properly on 64-bit platforms which follow
Chris@4:    the LP64 programming model -- that is, where
Chris@4:    <computeroutput>sizeof(long)</computeroutput> and
Chris@4:    <computeroutput>sizeof(void*)</computeroutput> are 8.  Under
Chris@4:    LP64, <computeroutput>sizeof(int)</computeroutput> is still 4,
Chris@4:    so <computeroutput>libbzip2</computeroutput>, which doesn't
Chris@4:    use the <computeroutput>long</computeroutput> type, is
Chris@4:    OK.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_SEQUENCE_ERROR</computeroutput></term>
Chris@4:   <listitem><para>When using the library, it is important to call
Chris@4:    the functions in the correct sequence and with data structures
Chris@4:    (buffers etc) in the correct states.
Chris@4:    <computeroutput>libbzip2</computeroutput> checks as much as it
Chris@4:    can to ensure this is happening, and returns
Chris@4:    <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> if not.
Chris@4:    Code which complies precisely with the function semantics, as
Chris@4:    detailed below, should never receive this value; such an event
Chris@4:    denotes buggy code which you should
Chris@4:    investigate.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_PARAM_ERROR</computeroutput></term>
Chris@4:   <listitem><para>Returned when a parameter to a function call is
Chris@4:    out of range or otherwise manifestly incorrect.  As with
Chris@4:    <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, this
Chris@4:    denotes a bug in the client code.  The distinction between
Chris@4:    <computeroutput>BZ_PARAM_ERROR</computeroutput> and
Chris@4:    <computeroutput>BZ_SEQUENCE_ERROR</computeroutput> is a bit
Chris@4:    hazy, but still worth making.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_MEM_ERROR</computeroutput></term>
Chris@4:   <listitem><para>Returned when a request to allocate memory
Chris@4:    failed.  Note that the quantity of memory needed to decompress
Chris@4:    a stream cannot be determined until the stream's header has
Chris@4:    been read.  So
Chris@4:    <computeroutput>BZ2_bzDecompress</computeroutput> and
Chris@4:    <computeroutput>BZ2_bzRead</computeroutput> may return
Chris@4:    <computeroutput>BZ_MEM_ERROR</computeroutput> even though some
Chris@4:    of the compressed data has been read.  The same is not true
Chris@4:    for compression; once
Chris@4:    <computeroutput>BZ2_bzCompressInit</computeroutput> or
Chris@4:    <computeroutput>BZ2_bzWriteOpen</computeroutput> have
Chris@4:    successfully completed,
Chris@4:    <computeroutput>BZ_MEM_ERROR</computeroutput> cannot
Chris@4:    occur.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_DATA_ERROR</computeroutput></term>
Chris@4:   <listitem><para>Returned when a data integrity error is
Chris@4:    detected during decompression.  Most importantly, this means
Chris@4:    when stored and computed CRCs for the data do not match.  This
Chris@4:    value is also returned upon detection of any other anomaly in
Chris@4:    the compressed data.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_DATA_ERROR_MAGIC</computeroutput></term>
Chris@4:   <listitem><para>As a special case of
Chris@4:    <computeroutput>BZ_DATA_ERROR</computeroutput>, it is
Chris@4:    sometimes useful to know when the compressed stream does not
Chris@4:    start with the correct magic bytes (<computeroutput>'B' 'Z'
Chris@4:    'h'</computeroutput>).</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_IO_ERROR</computeroutput></term>
Chris@4:   <listitem><para>Returned by
Chris@4:    <computeroutput>BZ2_bzRead</computeroutput> and
Chris@4:    <computeroutput>BZ2_bzWrite</computeroutput> when there is an
Chris@4:    error reading or writing in the compressed file, and by
Chris@4:    <computeroutput>BZ2_bzReadOpen</computeroutput> and
Chris@4:    <computeroutput>BZ2_bzWriteOpen</computeroutput> for attempts
Chris@4:    to use a file for which the error indicator (viz,
Chris@4:    <computeroutput>ferror(f)</computeroutput>) is set.  On
Chris@4:    receipt of <computeroutput>BZ_IO_ERROR</computeroutput>, the
Chris@4:    caller should consult <computeroutput>errno</computeroutput>
Chris@4:    and/or <computeroutput>perror</computeroutput> to acquire
Chris@4:    operating-system specific information about the
Chris@4:    problem.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_UNEXPECTED_EOF</computeroutput></term>
Chris@4:   <listitem><para>Returned by
Chris@4:    <computeroutput>BZ2_bzRead</computeroutput> when the
Chris@4:    compressed file finishes before the logical end of stream is
Chris@4:    detected.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4:  <varlistentry>
Chris@4:   <term><computeroutput>BZ_OUTBUFF_FULL</computeroutput></term>
Chris@4:   <listitem><para>Returned by
Chris@4:    <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput> and
Chris@4:    <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> to
Chris@4:    indicate that the output data will not fit into the output
Chris@4:    buffer provided.</para></listitem>
Chris@4:  </varlistentry>
Chris@4: 
Chris@4: </variablelist>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="low-level" xreflabel=">Low-level interface">
Chris@4: <title>Low-level interface</title>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzcompress-init" xreflabel="BZ2_bzCompressInit">
Chris@4: <title>BZ2_bzCompressInit</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: typedef struct {
Chris@4:   char *next_in;
Chris@4:   unsigned int avail_in;
Chris@4:   unsigned int total_in_lo32;
Chris@4:   unsigned int total_in_hi32;
Chris@4: 
Chris@4:   char *next_out;
Chris@4:   unsigned int avail_out;
Chris@4:   unsigned int total_out_lo32;
Chris@4:   unsigned int total_out_hi32;
Chris@4: 
Chris@4:   void *state;
Chris@4: 
Chris@4:   void *(*bzalloc)(void *,int,int);
Chris@4:   void (*bzfree)(void *,void *);
Chris@4:   void *opaque;
Chris@4: } bz_stream;
Chris@4: 
Chris@4: int BZ2_bzCompressInit ( bz_stream *strm, 
Chris@4:                          int blockSize100k, 
Chris@4:                          int verbosity,
Chris@4:                          int workFactor );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Prepares for compression.  The
Chris@4: <computeroutput>bz_stream</computeroutput> structure holds all
Chris@4: data pertaining to the compression activity.  A
Chris@4: <computeroutput>bz_stream</computeroutput> structure should be
Chris@4: allocated and initialised prior to the call.  The fields of
Chris@4: <computeroutput>bz_stream</computeroutput> comprise the entirety
Chris@4: of the user-visible data.  <computeroutput>state</computeroutput>
Chris@4: is a pointer to the private data structures required for
Chris@4: compression.</para>
Chris@4: 
Chris@4: <para>Custom memory allocators are supported, via fields
Chris@4: <computeroutput>bzalloc</computeroutput>,
Chris@4: <computeroutput>bzfree</computeroutput>, and
Chris@4: <computeroutput>opaque</computeroutput>.  The value
Chris@4: <computeroutput>opaque</computeroutput> is passed to as the first
Chris@4: argument to all calls to <computeroutput>bzalloc</computeroutput>
Chris@4: and <computeroutput>bzfree</computeroutput>, but is otherwise
Chris@4: ignored by the library.  The call <computeroutput>bzalloc (
Chris@4: opaque, n, m )</computeroutput> is expected to return a pointer
Chris@4: <computeroutput>p</computeroutput> to <computeroutput>n *
Chris@4: m</computeroutput> bytes of memory, and <computeroutput>bzfree (
Chris@4: opaque, p )</computeroutput> should free that memory.</para>
Chris@4: 
Chris@4: <para>If you don't want to use a custom memory allocator, set
Chris@4: <computeroutput>bzalloc</computeroutput>,
Chris@4: <computeroutput>bzfree</computeroutput> and
Chris@4: <computeroutput>opaque</computeroutput> to
Chris@4: <computeroutput>NULL</computeroutput>, and the library will then
Chris@4: use the standard <computeroutput>malloc</computeroutput> /
Chris@4: <computeroutput>free</computeroutput> routines.</para>
Chris@4: 
Chris@4: <para>Before calling
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>, fields
Chris@4: <computeroutput>bzalloc</computeroutput>,
Chris@4: <computeroutput>bzfree</computeroutput> and
Chris@4: <computeroutput>opaque</computeroutput> should be filled
Chris@4: appropriately, as just described.  Upon return, the internal
Chris@4: state will have been allocated and initialised, and
Chris@4: <computeroutput>total_in_lo32</computeroutput>,
Chris@4: <computeroutput>total_in_hi32</computeroutput>,
Chris@4: <computeroutput>total_out_lo32</computeroutput> and
Chris@4: <computeroutput>total_out_hi32</computeroutput> will have been
Chris@4: set to zero.  These four fields are used by the library to inform
Chris@4: the caller of the total amount of data passed into and out of the
Chris@4: library, respectively.  You should not try to change them.  As of
Chris@4: version 1.0, 64-bit counts are maintained, even on 32-bit
Chris@4: platforms, using the <computeroutput>_hi32</computeroutput>
Chris@4: fields to store the upper 32 bits of the count.  So, for example,
Chris@4: the total amount of data in is <computeroutput>(total_in_hi32
Chris@4: &#60;&#60; 32) + total_in_lo32</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Parameter <computeroutput>blockSize100k</computeroutput>
Chris@4: specifies the block size to be used for compression.  It should
Chris@4: be a value between 1 and 9 inclusive, and the actual block size
Chris@4: used is 100000 x this figure.  9 gives the best compression but
Chris@4: takes most memory.</para>
Chris@4: 
Chris@4: <para>Parameter <computeroutput>verbosity</computeroutput> should
Chris@4: be set to a number between 0 and 4 inclusive.  0 is silent, and
Chris@4: greater numbers give increasingly verbose monitoring/debugging
Chris@4: output.  If the library has been compiled with
Chris@4: <computeroutput>-DBZ_NO_STDIO</computeroutput>, no such output
Chris@4: will appear for any verbosity setting.</para>
Chris@4: 
Chris@4: <para>Parameter <computeroutput>workFactor</computeroutput>
Chris@4: controls how the compression phase behaves when presented with
Chris@4: worst case, highly repetitive, input data.  If compression runs
Chris@4: into difficulties caused by repetitive data, the library switches
Chris@4: from the standard sorting algorithm to a fallback algorithm.  The
Chris@4: fallback is slower than the standard algorithm by perhaps a
Chris@4: factor of three, but always behaves reasonably, no matter how bad
Chris@4: the input.</para>
Chris@4: 
Chris@4: <para>Lower values of <computeroutput>workFactor</computeroutput>
Chris@4: reduce the amount of effort the standard algorithm will expend
Chris@4: before resorting to the fallback.  You should set this parameter
Chris@4: carefully; too low, and many inputs will be handled by the
Chris@4: fallback algorithm and so compress rather slowly, too high, and
Chris@4: your average-to-worst case compression times can become very
Chris@4: large.  The default value of 30 gives reasonable behaviour over a
Chris@4: wide range of circumstances.</para>
Chris@4: 
Chris@4: <para>Allowable values range from 0 to 250 inclusive.  0 is a
Chris@4: special case, equivalent to using the default value of 30.</para>
Chris@4: 
Chris@4: <para>Note that the compressed output generated is the same
Chris@4: regardless of whether or not the fallback algorithm is
Chris@4: used.</para>
Chris@4: 
Chris@4: <para>Be aware also that this parameter may disappear entirely in
Chris@4: future versions of the library.  In principle it should be
Chris@4: possible to devise a good way to automatically choose which
Chris@4: algorithm to use.  Such a mechanism would render the parameter
Chris@4: obsolete.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if strm is NULL 
Chris@4:   or blockSize < 1 or blockSize > 9
Chris@4:   or verbosity < 0 or verbosity > 4
Chris@4:   or workFactor < 0 or workFactor > 250
Chris@4: BZ_MEM_ERROR 
Chris@4:   if not enough memory is available
Chris@4: BZ_OK 
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzCompress
Chris@4:   if BZ_OK is returned
Chris@4:   no specific action needed in case of error
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzCompress" xreflabel="BZ2_bzCompress">
Chris@4: <title>BZ2_bzCompress</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzCompress ( bz_stream *strm, int action );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Provides more input and/or output buffer space for the
Chris@4: library.  The caller maintains input and output buffers, and
Chris@4: calls <computeroutput>BZ2_bzCompress</computeroutput> to transfer
Chris@4: data between them.</para>
Chris@4: 
Chris@4: <para>Before each call to
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>,
Chris@4: <computeroutput>next_in</computeroutput> should point at the data
Chris@4: to be compressed, and <computeroutput>avail_in</computeroutput>
Chris@4: should indicate how many bytes the library may read.
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> updates
Chris@4: <computeroutput>next_in</computeroutput>,
Chris@4: <computeroutput>avail_in</computeroutput> and
Chris@4: <computeroutput>total_in</computeroutput> to reflect the number
Chris@4: of bytes it has read.</para>
Chris@4: 
Chris@4: <para>Similarly, <computeroutput>next_out</computeroutput> should
Chris@4: point to a buffer in which the compressed data is to be placed,
Chris@4: with <computeroutput>avail_out</computeroutput> indicating how
Chris@4: much output space is available.
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> updates
Chris@4: <computeroutput>next_out</computeroutput>,
Chris@4: <computeroutput>avail_out</computeroutput> and
Chris@4: <computeroutput>total_out</computeroutput> to reflect the number
Chris@4: of bytes output.</para>
Chris@4: 
Chris@4: <para>You may provide and remove as little or as much data as you
Chris@4: like on each call of
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>.  In the limit,
Chris@4: it is acceptable to supply and remove data one byte at a time,
Chris@4: although this would be terribly inefficient.  You should always
Chris@4: ensure that at least one byte of output space is available at
Chris@4: each call.</para>
Chris@4: 
Chris@4: <para>A second purpose of
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> is to request a
Chris@4: change of mode of the compressed stream.</para>
Chris@4: 
Chris@4: <para>Conceptually, a compressed stream can be in one of four
Chris@4: states: IDLE, RUNNING, FLUSHING and FINISHING.  Before
Chris@4: initialisation
Chris@4: (<computeroutput>BZ2_bzCompressInit</computeroutput>) and after
Chris@4: termination (<computeroutput>BZ2_bzCompressEnd</computeroutput>),
Chris@4: a stream is regarded as IDLE.</para>
Chris@4: 
Chris@4: <para>Upon initialisation
Chris@4: (<computeroutput>BZ2_bzCompressInit</computeroutput>), the stream
Chris@4: is placed in the RUNNING state.  Subsequent calls to
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> should pass
Chris@4: <computeroutput>BZ_RUN</computeroutput> as the requested action;
Chris@4: other actions are illegal and will result in
Chris@4: <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.</para>
Chris@4: 
Chris@4: <para>At some point, the calling program will have provided all
Chris@4: the input data it wants to.  It will then want to finish up -- in
Chris@4: effect, asking the library to process any data it might have
Chris@4: buffered internally.  In this state,
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> will no longer
Chris@4: attempt to read data from
Chris@4: <computeroutput>next_in</computeroutput>, but it will want to
Chris@4: write data to <computeroutput>next_out</computeroutput>.  Because
Chris@4: the output buffer supplied by the user can be arbitrarily small,
Chris@4: the finishing-up operation cannot necessarily be done with a
Chris@4: single call of
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Instead, the calling program passes
Chris@4: <computeroutput>BZ_FINISH</computeroutput> as an action to
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>.  This changes
Chris@4: the stream's state to FINISHING.  Any remaining input (ie,
Chris@4: <computeroutput>next_in[0 .. avail_in-1]</computeroutput>) is
Chris@4: compressed and transferred to the output buffer.  To do this,
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> must be called
Chris@4: repeatedly until all the output has been consumed.  At that
Chris@4: point, <computeroutput>BZ2_bzCompress</computeroutput> returns
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput>, and the stream's
Chris@4: state is set back to IDLE.
Chris@4: <computeroutput>BZ2_bzCompressEnd</computeroutput> should then be
Chris@4: called.</para>
Chris@4: 
Chris@4: <para>Just to make sure the calling program does not cheat, the
Chris@4: library makes a note of <computeroutput>avail_in</computeroutput>
Chris@4: at the time of the first call to
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> which has
Chris@4: <computeroutput>BZ_FINISH</computeroutput> as an action (ie, at
Chris@4: the time the program has announced its intention to not supply
Chris@4: any more input).  By comparing this value with that of
Chris@4: <computeroutput>avail_in</computeroutput> over subsequent calls
Chris@4: to <computeroutput>BZ2_bzCompress</computeroutput>, the library
Chris@4: can detect any attempts to slip in more data to compress.  Any
Chris@4: calls for which this is detected will return
Chris@4: <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>.  This
Chris@4: indicates a programming mistake which should be corrected.</para>
Chris@4: 
Chris@4: <para>Instead of asking to finish, the calling program may ask
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> to take all the
Chris@4: remaining input, compress it and terminate the current
Chris@4: (Burrows-Wheeler) compression block.  This could be useful for
Chris@4: error control purposes.  The mechanism is analogous to that for
Chris@4: finishing: call <computeroutput>BZ2_bzCompress</computeroutput>
Chris@4: with an action of <computeroutput>BZ_FLUSH</computeroutput>,
Chris@4: remove output data, and persist with the
Chris@4: <computeroutput>BZ_FLUSH</computeroutput> action until the value
Chris@4: <computeroutput>BZ_RUN</computeroutput> is returned.  As with
Chris@4: finishing, <computeroutput>BZ2_bzCompress</computeroutput>
Chris@4: detects any attempt to provide more input data once the flush has
Chris@4: begun.</para>
Chris@4: 
Chris@4: <para>Once the flush is complete, the stream returns to the
Chris@4: normal RUNNING state.</para>
Chris@4: 
Chris@4: <para>This all sounds pretty complex, but isn't really.  Here's a
Chris@4: table which shows which actions are allowable in each state, what
Chris@4: action will be taken, what the next state is, and what the
Chris@4: non-error return values are.  Note that you can't explicitly ask
Chris@4: what state the stream is in, but nor do you need to -- it can be
Chris@4: inferred from the values returned by
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: IDLE/any
Chris@4:   Illegal.  IDLE state only exists after BZ2_bzCompressEnd or
Chris@4:   before BZ2_bzCompressInit.
Chris@4:   Return value = BZ_SEQUENCE_ERROR
Chris@4: 
Chris@4: RUNNING/BZ_RUN
Chris@4:   Compress from next_in to next_out as much as possible.
Chris@4:   Next state = RUNNING
Chris@4:   Return value = BZ_RUN_OK
Chris@4: 
Chris@4: RUNNING/BZ_FLUSH
Chris@4:   Remember current value of next_in. Compress from next_in
Chris@4:   to next_out as much as possible, but do not accept any more input.
Chris@4:   Next state = FLUSHING
Chris@4:   Return value = BZ_FLUSH_OK
Chris@4: 
Chris@4: RUNNING/BZ_FINISH
Chris@4:   Remember current value of next_in. Compress from next_in
Chris@4:   to next_out as much as possible, but do not accept any more input.
Chris@4:   Next state = FINISHING
Chris@4:   Return value = BZ_FINISH_OK
Chris@4: 
Chris@4: FLUSHING/BZ_FLUSH
Chris@4:   Compress from next_in to next_out as much as possible, 
Chris@4:   but do not accept any more input.
Chris@4:   If all the existing input has been used up and all compressed
Chris@4:   output has been removed
Chris@4:     Next state = RUNNING; Return value = BZ_RUN_OK
Chris@4:   else
Chris@4:     Next state = FLUSHING; Return value = BZ_FLUSH_OK
Chris@4: 
Chris@4: FLUSHING/other     
Chris@4:   Illegal.
Chris@4:   Return value = BZ_SEQUENCE_ERROR
Chris@4: 
Chris@4: FINISHING/BZ_FINISH
Chris@4:   Compress from next_in to next_out as much as possible,
Chris@4:   but to not accept any more input.  
Chris@4:   If all the existing input has been used up and all compressed
Chris@4:   output has been removed
Chris@4:     Next state = IDLE; Return value = BZ_STREAM_END
Chris@4:   else
Chris@4:     Next state = FINISHING; Return value = BZ_FINISH_OK
Chris@4: 
Chris@4: FINISHING/other
Chris@4:   Illegal.
Chris@4:   Return value = BZ_SEQUENCE_ERROR
Chris@4: </programlisting>
Chris@4: 
Chris@4: 
Chris@4: <para>That still looks complicated?  Well, fair enough.  The
Chris@4: usual sequence of calls for compressing a load of data is:</para>
Chris@4: 
Chris@4: <orderedlist>
Chris@4: 
Chris@4:  <listitem><para>Get started with
Chris@4:   <computeroutput>BZ2_bzCompressInit</computeroutput>.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>Shovel data in and shlurp out its compressed form
Chris@4:   using zero or more calls of
Chris@4:   <computeroutput>BZ2_bzCompress</computeroutput> with action =
Chris@4:   <computeroutput>BZ_RUN</computeroutput>.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>Finish up. Repeatedly call
Chris@4:   <computeroutput>BZ2_bzCompress</computeroutput> with action =
Chris@4:   <computeroutput>BZ_FINISH</computeroutput>, copying out the
Chris@4:   compressed output, until
Chris@4:   <computeroutput>BZ_STREAM_END</computeroutput> is
Chris@4:   returned.</para></listitem> <listitem><para>Close up and go home.  Call
Chris@4:   <computeroutput>BZ2_bzCompressEnd</computeroutput>.</para></listitem>
Chris@4: 
Chris@4: </orderedlist>
Chris@4: 
Chris@4: <para>If the data you want to compress fits into your input
Chris@4: buffer all at once, you can skip the calls of
Chris@4: <computeroutput>BZ2_bzCompress ( ..., BZ_RUN )</computeroutput>
Chris@4: and just do the <computeroutput>BZ2_bzCompress ( ..., BZ_FINISH
Chris@4: )</computeroutput> calls.</para>
Chris@4: 
Chris@4: <para>All required memory is allocated by
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>.  The
Chris@4: compression library can accept any data at all (obviously).  So
Chris@4: you shouldn't get any error return values from the
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> calls.  If you
Chris@4: do, they will be
Chris@4: <computeroutput>BZ_SEQUENCE_ERROR</computeroutput>, and indicate
Chris@4: a bug in your programming.</para>
Chris@4: 
Chris@4: <para>Trivial other possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if strm is NULL, or strm->s is NULL
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzCompress-end" xreflabel="BZ2_bzCompressEnd">
Chris@4: <title>BZ2_bzCompressEnd</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzCompressEnd ( bz_stream *strm );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Releases all memory associated with a compression
Chris@4: stream.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR  if strm is NULL or strm->s is NULL
Chris@4: BZ_OK           otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzDecompress-init" xreflabel="BZ2_bzDecompressInit">
Chris@4: <title>BZ2_bzDecompressInit</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Prepares for decompression.  As with
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>, a
Chris@4: <computeroutput>bz_stream</computeroutput> record should be
Chris@4: allocated and initialised before the call.  Fields
Chris@4: <computeroutput>bzalloc</computeroutput>,
Chris@4: <computeroutput>bzfree</computeroutput> and
Chris@4: <computeroutput>opaque</computeroutput> should be set if a custom
Chris@4: memory allocator is required, or made
Chris@4: <computeroutput>NULL</computeroutput> for the normal
Chris@4: <computeroutput>malloc</computeroutput> /
Chris@4: <computeroutput>free</computeroutput> routines.  Upon return, the
Chris@4: internal state will have been initialised, and
Chris@4: <computeroutput>total_in</computeroutput> and
Chris@4: <computeroutput>total_out</computeroutput> will be zero.</para>
Chris@4: 
Chris@4: <para>For the meaning of parameter
Chris@4: <computeroutput>verbosity</computeroutput>, see
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
Chris@4: 
Chris@4: <para>If <computeroutput>small</computeroutput> is nonzero, the
Chris@4: library will use an alternative decompression algorithm which
Chris@4: uses less memory but at the cost of decompressing more slowly
Chris@4: (roughly speaking, half the speed, but the maximum memory
Chris@4: requirement drops to around 2300k).  See <xref linkend="using"/>
Chris@4: for more information on memory management.</para>
Chris@4: 
Chris@4: <para>Note that the amount of memory needed to decompress a
Chris@4: stream cannot be determined until the stream's header has been
Chris@4: read, so even if
Chris@4: <computeroutput>BZ2_bzDecompressInit</computeroutput> succeeds, a
Chris@4: subsequent <computeroutput>BZ2_bzDecompress</computeroutput>
Chris@4: could fail with
Chris@4: <computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if ( small != 0 && small != 1 )
Chris@4:   or (verbosity <; 0 || verbosity > 4)
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory is available
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzDecompress
Chris@4:   if BZ_OK was returned
Chris@4:   no specific action required in case of error
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzDecompress" xreflabel="BZ2_bzDecompress">
Chris@4: <title>BZ2_bzDecompress</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzDecompress ( bz_stream *strm );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Provides more input and/out output buffer space for the
Chris@4: library.  The caller maintains input and output buffers, and uses
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput> to transfer
Chris@4: data between them.</para>
Chris@4: 
Chris@4: <para>Before each call to
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput>,
Chris@4: <computeroutput>next_in</computeroutput> should point at the
Chris@4: compressed data, and <computeroutput>avail_in</computeroutput>
Chris@4: should indicate how many bytes the library may read.
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput> updates
Chris@4: <computeroutput>next_in</computeroutput>,
Chris@4: <computeroutput>avail_in</computeroutput> and
Chris@4: <computeroutput>total_in</computeroutput> to reflect the number
Chris@4: of bytes it has read.</para>
Chris@4: 
Chris@4: <para>Similarly, <computeroutput>next_out</computeroutput> should
Chris@4: point to a buffer in which the uncompressed output is to be
Chris@4: placed, with <computeroutput>avail_out</computeroutput>
Chris@4: indicating how much output space is available.
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput> updates
Chris@4: <computeroutput>next_out</computeroutput>,
Chris@4: <computeroutput>avail_out</computeroutput> and
Chris@4: <computeroutput>total_out</computeroutput> to reflect the number
Chris@4: of bytes output.</para>
Chris@4: 
Chris@4: <para>You may provide and remove as little or as much data as you
Chris@4: like on each call of
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput>.  In the limit,
Chris@4: it is acceptable to supply and remove data one byte at a time,
Chris@4: although this would be terribly inefficient.  You should always
Chris@4: ensure that at least one byte of output space is available at
Chris@4: each call.</para>
Chris@4: 
Chris@4: <para>Use of <computeroutput>BZ2_bzDecompress</computeroutput> is
Chris@4: simpler than
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>.</para>
Chris@4: 
Chris@4: <para>You should provide input and remove output as described
Chris@4: above, and repeatedly call
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput> until
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput> is returned.
Chris@4: Appearance of <computeroutput>BZ_STREAM_END</computeroutput>
Chris@4: denotes that <computeroutput>BZ2_bzDecompress</computeroutput>
Chris@4: has detected the logical end of the compressed stream.
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput> will not
Chris@4: produce <computeroutput>BZ_STREAM_END</computeroutput> until all
Chris@4: output data has been placed into the output buffer, so once
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput> appears, you are
Chris@4: guaranteed to have available all the decompressed output, and
Chris@4: <computeroutput>BZ2_bzDecompressEnd</computeroutput> can safely
Chris@4: be called.</para>
Chris@4: 
Chris@4: <para>If case of an error return value, you should call
Chris@4: <computeroutput>BZ2_bzDecompressEnd</computeroutput> to clean up
Chris@4: and release memory.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if strm is NULL or strm->s is NULL
Chris@4:   or strm->avail_out < 1
Chris@4: BZ_DATA_ERROR
Chris@4:   if a data integrity error is detected in the compressed stream
Chris@4: BZ_DATA_ERROR_MAGIC
Chris@4:   if the compressed stream doesn't begin with the right magic bytes
Chris@4: BZ_MEM_ERROR
Chris@4:   if there wasn't enough memory available
Chris@4: BZ_STREAM_END
Chris@4:   if the logical end of the data stream was detected and all
Chris@4:   output in has been consumed, eg s-->avail_out > 0
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzDecompress
Chris@4:   if BZ_OK was returned
Chris@4: BZ2_bzDecompressEnd
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzDecompress-end" xreflabel="BZ2_bzDecompressEnd">
Chris@4: <title>BZ2_bzDecompressEnd</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzDecompressEnd ( bz_stream *strm );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Releases all memory associated with a decompression
Chris@4: stream.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if strm is NULL or strm->s is NULL
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4:   None.
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="hl-interface" xreflabel="High-level interface">
Chris@4: <title>High-level interface</title>
Chris@4: 
Chris@4: <para>This interface provides functions for reading and writing
Chris@4: <computeroutput>bzip2</computeroutput> format files.  First, some
Chris@4: general points.</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para>All of the functions take an
Chris@4:   <computeroutput>int*</computeroutput> first argument,
Chris@4:   <computeroutput>bzerror</computeroutput>.  After each call,
Chris@4:   <computeroutput>bzerror</computeroutput> should be consulted
Chris@4:   first to determine the outcome of the call.  If
Chris@4:   <computeroutput>bzerror</computeroutput> is
Chris@4:   <computeroutput>BZ_OK</computeroutput>, the call completed
Chris@4:   successfully, and only then should the return value of the
Chris@4:   function (if any) be consulted.  If
Chris@4:   <computeroutput>bzerror</computeroutput> is
Chris@4:   <computeroutput>BZ_IO_ERROR</computeroutput>, there was an
Chris@4:   error reading/writing the underlying compressed file, and you
Chris@4:   should then consult <computeroutput>errno</computeroutput> /
Chris@4:   <computeroutput>perror</computeroutput> to determine the cause
Chris@4:   of the difficulty.  <computeroutput>bzerror</computeroutput>
Chris@4:   may also be set to various other values; precise details are
Chris@4:   given on a per-function basis below.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>If <computeroutput>bzerror</computeroutput> indicates
Chris@4:   an error (ie, anything except
Chris@4:   <computeroutput>BZ_OK</computeroutput> and
Chris@4:   <computeroutput>BZ_STREAM_END</computeroutput>), you should
Chris@4:   immediately call
Chris@4:   <computeroutput>BZ2_bzReadClose</computeroutput> (or
Chris@4:   <computeroutput>BZ2_bzWriteClose</computeroutput>, depending on
Chris@4:   whether you are attempting to read or to write) to free up all
Chris@4:   resources associated with the stream.  Once an error has been
Chris@4:   indicated, behaviour of all calls except
Chris@4:   <computeroutput>BZ2_bzReadClose</computeroutput>
Chris@4:   (<computeroutput>BZ2_bzWriteClose</computeroutput>) is
Chris@4:   undefined.  The implication is that (1)
Chris@4:   <computeroutput>bzerror</computeroutput> should be checked
Chris@4:   after each call, and (2) if
Chris@4:   <computeroutput>bzerror</computeroutput> indicates an error,
Chris@4:   <computeroutput>BZ2_bzReadClose</computeroutput>
Chris@4:   (<computeroutput>BZ2_bzWriteClose</computeroutput>) should then
Chris@4:   be called to clean up.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>The <computeroutput>FILE*</computeroutput> arguments
Chris@4:   passed to <computeroutput>BZ2_bzReadOpen</computeroutput> /
Chris@4:   <computeroutput>BZ2_bzWriteOpen</computeroutput> should be set
Chris@4:   to binary mode.  Most Unix systems will do this by default, but
Chris@4:   other platforms, including Windows and Mac, will not.  If you
Chris@4:   omit this, you may encounter problems when moving code to new
Chris@4:   platforms.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>Memory allocation requests are handled by
Chris@4:   <computeroutput>malloc</computeroutput> /
Chris@4:   <computeroutput>free</computeroutput>.  At present there is no
Chris@4:   facility for user-defined memory allocators in the file I/O
Chris@4:   functions (could easily be added, though).</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzreadopen" xreflabel="BZ2_bzReadOpen">
Chris@4: <title>BZ2_bzReadOpen</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: typedef void BZFILE;
Chris@4: 
Chris@4: BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, 
Chris@4:                         int verbosity, int small,
Chris@4:                         void *unused, int nUnused );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Prepare to read compressed data from file handle
Chris@4: <computeroutput>f</computeroutput>.
Chris@4: <computeroutput>f</computeroutput> should refer to a file which
Chris@4: has been opened for reading, and for which the error indicator
Chris@4: (<computeroutput>ferror(f)</computeroutput>)is not set.  If
Chris@4: <computeroutput>small</computeroutput> is 1, the library will try
Chris@4: to decompress using less memory, at the expense of speed.</para>
Chris@4: 
Chris@4: <para>For reasons explained below,
Chris@4: <computeroutput>BZ2_bzRead</computeroutput> will decompress the
Chris@4: <computeroutput>nUnused</computeroutput> bytes starting at
Chris@4: <computeroutput>unused</computeroutput>, before starting to read
Chris@4: from the file <computeroutput>f</computeroutput>.  At most
Chris@4: <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes may be
Chris@4: supplied like this.  If this facility is not required, you should
Chris@4: pass <computeroutput>NULL</computeroutput> and
Chris@4: <computeroutput>0</computeroutput> for
Chris@4: <computeroutput>unused</computeroutput> and
Chris@4: n<computeroutput>Unused</computeroutput> respectively.</para>
Chris@4: 
Chris@4: <para>For the meaning of parameters
Chris@4: <computeroutput>small</computeroutput> and
Chris@4: <computeroutput>verbosity</computeroutput>, see
Chris@4: <computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
Chris@4: 
Chris@4: <para>The amount of memory needed to decompress a file cannot be
Chris@4: determined until the file's header has been read.  So it is
Chris@4: possible that <computeroutput>BZ2_bzReadOpen</computeroutput>
Chris@4: returns <computeroutput>BZ_OK</computeroutput> but a subsequent
Chris@4: call of <computeroutput>BZ2_bzRead</computeroutput> will return
Chris@4: <computeroutput>BZ_MEM_ERROR</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if f is NULL
Chris@4:   or small is neither 0 nor 1
Chris@4:   or ( unused == NULL && nUnused != 0 )
Chris@4:   or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) )
Chris@4: BZ_IO_ERROR
Chris@4:   if ferror(f) is nonzero
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory is available
Chris@4: BZ_OK
Chris@4:   otherwise.
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: Pointer to an abstract BZFILE
Chris@4:   if bzerror is BZ_OK
Chris@4: NULL
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzRead
Chris@4:   if bzerror is BZ_OK
Chris@4: BZ2_bzClose
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzread" xreflabel="BZ2_bzRead">
Chris@4: <title>BZ2_bzRead</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Reads up to <computeroutput>len</computeroutput>
Chris@4: (uncompressed) bytes from the compressed file
Chris@4: <computeroutput>b</computeroutput> into the buffer
Chris@4: <computeroutput>buf</computeroutput>.  If the read was
Chris@4: successful, <computeroutput>bzerror</computeroutput> is set to
Chris@4: <computeroutput>BZ_OK</computeroutput> and the number of bytes
Chris@4: read is returned.  If the logical end-of-stream was detected,
Chris@4: <computeroutput>bzerror</computeroutput> will be set to
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput>, and the number of
Chris@4: bytes read is returned.  All other
Chris@4: <computeroutput>bzerror</computeroutput> values denote an
Chris@4: error.</para>
Chris@4: 
Chris@4: <para><computeroutput>BZ2_bzRead</computeroutput> will supply
Chris@4: <computeroutput>len</computeroutput> bytes, unless the logical
Chris@4: stream end is detected or an error occurs.  Because of this, it
Chris@4: is possible to detect the stream end by observing when the number
Chris@4: of bytes returned is less than the number requested.
Chris@4: Nevertheless, this is regarded as inadvisable; you should instead
Chris@4: check <computeroutput>bzerror</computeroutput> after every call
Chris@4: and watch out for
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Internally, <computeroutput>BZ2_bzRead</computeroutput>
Chris@4: copies data from the compressed file in chunks of size
Chris@4: <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes before
Chris@4: decompressing it.  If the file contains more bytes than strictly
Chris@4: needed to reach the logical end-of-stream,
Chris@4: <computeroutput>BZ2_bzRead</computeroutput> will almost certainly
Chris@4: read some of the trailing data before signalling
Chris@4: <computeroutput>BZ_SEQUENCE_END</computeroutput>.  To collect the
Chris@4: read but unused data once
Chris@4: <computeroutput>BZ_SEQUENCE_END</computeroutput> has appeared,
Chris@4: call <computeroutput>BZ2_bzReadGetUnused</computeroutput>
Chris@4: immediately before
Chris@4: <computeroutput>BZ2_bzReadClose</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if b is NULL or buf is NULL or len < 0
Chris@4: BZ_SEQUENCE_ERROR
Chris@4:   if b was opened with BZ2_bzWriteOpen
Chris@4: BZ_IO_ERROR
Chris@4:   if there is an error reading from the compressed file
Chris@4: BZ_UNEXPECTED_EOF
Chris@4:   if the compressed file ended before 
Chris@4:   the logical end-of-stream was detected
Chris@4: BZ_DATA_ERROR
Chris@4:   if a data integrity error was detected in the compressed stream
Chris@4: BZ_DATA_ERROR_MAGIC
Chris@4:   if the stream does not begin with the requisite header bytes 
Chris@4:   (ie, is not a bzip2 data file).  This is really 
Chris@4:   a special case of BZ_DATA_ERROR.
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory was available
Chris@4: BZ_STREAM_END
Chris@4:   if the logical end of stream was detected.
Chris@4: BZ_OK
Chris@4:   otherwise.
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: number of bytes read
Chris@4:   if bzerror is BZ_OK or BZ_STREAM_END
Chris@4: undefined
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: collect data from buf, then BZ2_bzRead or BZ2_bzReadClose
Chris@4:   if bzerror is BZ_OK
Chris@4: collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused
Chris@4:   if bzerror is BZ_SEQUENCE_END
Chris@4: BZ2_bzReadClose
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzreadgetunused" xreflabel="BZ2_bzReadGetUnused">
Chris@4: <title>BZ2_bzReadGetUnused</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, 
Chris@4:                           void** unused, int* nUnused );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Returns data which was read from the compressed file but
Chris@4: was not needed to get to the logical end-of-stream.
Chris@4: <computeroutput>*unused</computeroutput> is set to the address of
Chris@4: the data, and <computeroutput>*nUnused</computeroutput> to the
Chris@4: number of bytes.  <computeroutput>*nUnused</computeroutput> will
Chris@4: be set to a value between <computeroutput>0</computeroutput> and
Chris@4: <computeroutput>BZ_MAX_UNUSED</computeroutput> inclusive.</para>
Chris@4: 
Chris@4: <para>This function may only be called once
Chris@4: <computeroutput>BZ2_bzRead</computeroutput> has signalled
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput> but before
Chris@4: <computeroutput>BZ2_bzReadClose</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if b is NULL
Chris@4:   or unused is NULL or nUnused is NULL
Chris@4: BZ_SEQUENCE_ERROR
Chris@4:   if BZ_STREAM_END has not been signalled
Chris@4:   or if b was opened with BZ2_bzWriteOpen
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzReadClose
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzreadclose" xreflabel="BZ2_bzReadClose">
Chris@4: <title>BZ2_bzReadClose</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: void BZ2_bzReadClose ( int *bzerror, BZFILE *b );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Releases all memory pertaining to the compressed file
Chris@4: <computeroutput>b</computeroutput>.
Chris@4: <computeroutput>BZ2_bzReadClose</computeroutput> does not call
Chris@4: <computeroutput>fclose</computeroutput> on the underlying file
Chris@4: handle, so you should do that yourself if appropriate.
Chris@4: <computeroutput>BZ2_bzReadClose</computeroutput> should be called
Chris@4: to clean up after all error situations.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_SEQUENCE_ERROR
Chris@4:   if b was opened with BZ2_bzOpenWrite
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: none
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzwriteopen" xreflabel="BZ2_bzWriteOpen">
Chris@4: <title>BZ2_bzWriteOpen</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, 
Chris@4:                          int blockSize100k, int verbosity,
Chris@4:                          int workFactor );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Prepare to write compressed data to file handle
Chris@4: <computeroutput>f</computeroutput>.
Chris@4: <computeroutput>f</computeroutput> should refer to a file which
Chris@4: has been opened for writing, and for which the error indicator
Chris@4: (<computeroutput>ferror(f)</computeroutput>)is not set.</para>
Chris@4: 
Chris@4: <para>For the meaning of parameters
Chris@4: <computeroutput>blockSize100k</computeroutput>,
Chris@4: <computeroutput>verbosity</computeroutput> and
Chris@4: <computeroutput>workFactor</computeroutput>, see
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
Chris@4: 
Chris@4: <para>All required memory is allocated at this stage, so if the
Chris@4: call completes successfully,
Chris@4: <computeroutput>BZ_MEM_ERROR</computeroutput> cannot be signalled
Chris@4: by a subsequent call to
Chris@4: <computeroutput>BZ2_bzWrite</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if f is NULL
Chris@4:   or blockSize100k < 1 or blockSize100k > 9
Chris@4: BZ_IO_ERROR
Chris@4:   if ferror(f) is nonzero
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory is available
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: Pointer to an abstract BZFILE
Chris@4:   if bzerror is BZ_OK
Chris@4: NULL
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Allowable next actions:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ2_bzWrite
Chris@4:   if bzerror is BZ_OK
Chris@4:   (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless)
Chris@4: BZ2_bzWriteClose
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzwrite" xreflabel="BZ2_bzWrite">
Chris@4: <title>BZ2_bzWrite</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Absorbs <computeroutput>len</computeroutput> bytes from the
Chris@4: buffer <computeroutput>buf</computeroutput>, eventually to be
Chris@4: compressed and written to the file.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_PARAM_ERROR
Chris@4:   if b is NULL or buf is NULL or len < 0
Chris@4: BZ_SEQUENCE_ERROR
Chris@4:   if b was opened with BZ2_bzReadOpen
Chris@4: BZ_IO_ERROR
Chris@4:   if there is an error writing the compressed file.
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzwriteclose" xreflabel="BZ2_bzWriteClose">
Chris@4: <title>BZ2_bzWriteClose</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: void BZ2_bzWriteClose( int *bzerror, BZFILE* f,
Chris@4:                        int abandon,
Chris@4:                        unsigned int* nbytes_in,
Chris@4:                        unsigned int* nbytes_out );
Chris@4: 
Chris@4: void BZ2_bzWriteClose64( int *bzerror, BZFILE* f,
Chris@4:                          int abandon,
Chris@4:                          unsigned int* nbytes_in_lo32,
Chris@4:                          unsigned int* nbytes_in_hi32,
Chris@4:                          unsigned int* nbytes_out_lo32,
Chris@4:                          unsigned int* nbytes_out_hi32 );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Compresses and flushes to the compressed file all data so
Chris@4: far supplied by <computeroutput>BZ2_bzWrite</computeroutput>.
Chris@4: The logical end-of-stream markers are also written, so subsequent
Chris@4: calls to <computeroutput>BZ2_bzWrite</computeroutput> are
Chris@4: illegal.  All memory associated with the compressed file
Chris@4: <computeroutput>b</computeroutput> is released.
Chris@4: <computeroutput>fflush</computeroutput> is called on the
Chris@4: compressed file, but it is not
Chris@4: <computeroutput>fclose</computeroutput>'d.</para>
Chris@4: 
Chris@4: <para>If <computeroutput>BZ2_bzWriteClose</computeroutput> is
Chris@4: called to clean up after an error, the only action is to release
Chris@4: the memory.  The library records the error codes issued by
Chris@4: previous calls, so this situation will be detected automatically.
Chris@4: There is no attempt to complete the compression operation, nor to
Chris@4: <computeroutput>fflush</computeroutput> the compressed file.  You
Chris@4: can force this behaviour to happen even in the case of no error,
Chris@4: by passing a nonzero value to
Chris@4: <computeroutput>abandon</computeroutput>.</para>
Chris@4: 
Chris@4: <para>If <computeroutput>nbytes_in</computeroutput> is non-null,
Chris@4: <computeroutput>*nbytes_in</computeroutput> will be set to be the
Chris@4: total volume of uncompressed data handled.  Similarly,
Chris@4: <computeroutput>nbytes_out</computeroutput> will be set to the
Chris@4: total volume of compressed data written.  For compatibility with
Chris@4: older versions of the library,
Chris@4: <computeroutput>BZ2_bzWriteClose</computeroutput> only yields the
Chris@4: lower 32 bits of these counts.  Use
Chris@4: <computeroutput>BZ2_bzWriteClose64</computeroutput> if you want
Chris@4: the full 64 bit counts.  These two functions are otherwise
Chris@4: absolutely identical.</para>
Chris@4: 
Chris@4: <para>Possible assignments to
Chris@4: <computeroutput>bzerror</computeroutput>:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_SEQUENCE_ERROR
Chris@4:   if b was opened with BZ2_bzReadOpen
Chris@4: BZ_IO_ERROR
Chris@4:   if there is an error writing the compressed file
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="embed" xreflabel="Handling embedded compressed data streams">
Chris@4: <title>Handling embedded compressed data streams</title>
Chris@4: 
Chris@4: <para>The high-level library facilitates use of
Chris@4: <computeroutput>bzip2</computeroutput> data streams which form
Chris@4: some part of a surrounding, larger data stream.</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para>For writing, the library takes an open file handle,
Chris@4:   writes compressed data to it,
Chris@4:   <computeroutput>fflush</computeroutput>es it but does not
Chris@4:   <computeroutput>fclose</computeroutput> it.  The calling
Chris@4:   application can write its own data before and after the
Chris@4:   compressed data stream, using that same file handle.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>Reading is more complex, and the facilities are not as
Chris@4:   general as they could be since generality is hard to reconcile
Chris@4:   with efficiency.  <computeroutput>BZ2_bzRead</computeroutput>
Chris@4:   reads from the compressed file in blocks of size
Chris@4:   <computeroutput>BZ_MAX_UNUSED</computeroutput> bytes, and in
Chris@4:   doing so probably will overshoot the logical end of compressed
Chris@4:   stream.  To recover this data once decompression has ended,
Chris@4:   call <computeroutput>BZ2_bzReadGetUnused</computeroutput> after
Chris@4:   the last call of <computeroutput>BZ2_bzRead</computeroutput>
Chris@4:   (the one returning
Chris@4:   <computeroutput>BZ_STREAM_END</computeroutput>) but before
Chris@4:   calling
Chris@4:   <computeroutput>BZ2_bzReadClose</computeroutput>.</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: <para>This mechanism makes it easy to decompress multiple
Chris@4: <computeroutput>bzip2</computeroutput> streams placed end-to-end.
Chris@4: As the end of one stream, when
Chris@4: <computeroutput>BZ2_bzRead</computeroutput> returns
Chris@4: <computeroutput>BZ_STREAM_END</computeroutput>, call
Chris@4: <computeroutput>BZ2_bzReadGetUnused</computeroutput> to collect
Chris@4: the unused data (copy it into your own buffer somewhere).  That
Chris@4: data forms the start of the next compressed stream.  To start
Chris@4: uncompressing that next stream, call
Chris@4: <computeroutput>BZ2_bzReadOpen</computeroutput> again, feeding in
Chris@4: the unused data via the <computeroutput>unused</computeroutput> /
Chris@4: <computeroutput>nUnused</computeroutput> parameters.  Keep doing
Chris@4: this until <computeroutput>BZ_STREAM_END</computeroutput> return
Chris@4: coincides with the physical end of file
Chris@4: (<computeroutput>feof(f)</computeroutput>).  In this situation
Chris@4: <computeroutput>BZ2_bzReadGetUnused</computeroutput> will of
Chris@4: course return no data.</para>
Chris@4: 
Chris@4: <para>This should give some feel for how the high-level interface
Chris@4: can be used.  If you require extra flexibility, you'll have to
Chris@4: bite the bullet and get to grips with the low-level
Chris@4: interface.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="std-rdwr" xreflabel="Standard file-reading/writing code">
Chris@4: <title>Standard file-reading/writing code</title>
Chris@4: 
Chris@4: <para>Here's how you'd write data to a compressed file:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: FILE*   f;
Chris@4: BZFILE* b;
Chris@4: int     nBuf;
Chris@4: char    buf[ /* whatever size you like */ ];
Chris@4: int     bzerror;
Chris@4: int     nWritten;
Chris@4: 
Chris@4: f = fopen ( "myfile.bz2", "w" );
Chris@4: if ( !f ) {
Chris@4:  /* handle error */
Chris@4: }
Chris@4: b = BZ2_bzWriteOpen( &bzerror, f, 9 );
Chris@4: if (bzerror != BZ_OK) {
Chris@4:  BZ2_bzWriteClose ( b );
Chris@4:  /* handle error */
Chris@4: }
Chris@4: 
Chris@4: while ( /* condition */ ) {
Chris@4:  /* get data to write into buf, and set nBuf appropriately */
Chris@4:  nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf );
Chris@4:  if (bzerror == BZ_IO_ERROR) { 
Chris@4:    BZ2_bzWriteClose ( &bzerror, b );
Chris@4:    /* handle error */
Chris@4:  }
Chris@4: }
Chris@4: 
Chris@4: BZ2_bzWriteClose( &bzerror, b );
Chris@4: if (bzerror == BZ_IO_ERROR) {
Chris@4:  /* handle error */
Chris@4: }
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>And to read from a compressed file:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: FILE*   f;
Chris@4: BZFILE* b;
Chris@4: int     nBuf;
Chris@4: char    buf[ /* whatever size you like */ ];
Chris@4: int     bzerror;
Chris@4: int     nWritten;
Chris@4: 
Chris@4: f = fopen ( "myfile.bz2", "r" );
Chris@4: if ( !f ) {
Chris@4:   /* handle error */
Chris@4: }
Chris@4: b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 );
Chris@4: if ( bzerror != BZ_OK ) {
Chris@4:   BZ2_bzReadClose ( &bzerror, b );
Chris@4:   /* handle error */
Chris@4: }
Chris@4: 
Chris@4: bzerror = BZ_OK;
Chris@4: while ( bzerror == BZ_OK && /* arbitrary other conditions */) {
Chris@4:   nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ );
Chris@4:   if ( bzerror == BZ_OK ) {
Chris@4:     /* do something with buf[0 .. nBuf-1] */
Chris@4:   }
Chris@4: }
Chris@4: if ( bzerror != BZ_STREAM_END ) {
Chris@4:    BZ2_bzReadClose ( &bzerror, b );
Chris@4:    /* handle error */
Chris@4: } else {
Chris@4:    BZ2_bzReadClose ( &bzerror, b );
Chris@4: }
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="util-fns" xreflabel="Utility functions">
Chris@4: <title>Utility functions</title>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzbufftobuffcompress" xreflabel="BZ2_bzBuffToBuffCompress">
Chris@4: <title>BZ2_bzBuffToBuffCompress</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzBuffToBuffCompress( char*         dest,
Chris@4:                               unsigned int* destLen,
Chris@4:                               char*         source,
Chris@4:                               unsigned int  sourceLen,
Chris@4:                               int           blockSize100k,
Chris@4:                               int           verbosity,
Chris@4:                               int           workFactor );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Attempts to compress the data in <computeroutput>source[0
Chris@4: .. sourceLen-1]</computeroutput> into the destination buffer,
Chris@4: <computeroutput>dest[0 .. *destLen-1]</computeroutput>.  If the
Chris@4: destination buffer is big enough,
Chris@4: <computeroutput>*destLen</computeroutput> is set to the size of
Chris@4: the compressed data, and <computeroutput>BZ_OK</computeroutput>
Chris@4: is returned.  If the compressed data won't fit,
Chris@4: <computeroutput>*destLen</computeroutput> is unchanged, and
Chris@4: <computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
Chris@4: returned.</para>
Chris@4: 
Chris@4: <para>Compression in this manner is a one-shot event, done with a
Chris@4: single call to this function.  The resulting compressed data is a
Chris@4: complete <computeroutput>bzip2</computeroutput> format data
Chris@4: stream.  There is no mechanism for making additional calls to
Chris@4: provide extra input data.  If you want that kind of mechanism,
Chris@4: use the low-level interface.</para>
Chris@4: 
Chris@4: <para>For the meaning of parameters
Chris@4: <computeroutput>blockSize100k</computeroutput>,
Chris@4: <computeroutput>verbosity</computeroutput> and
Chris@4: <computeroutput>workFactor</computeroutput>, see
Chris@4: <computeroutput>BZ2_bzCompressInit</computeroutput>.</para>
Chris@4: 
Chris@4: <para>To guarantee that the compressed data will fit in its
Chris@4: buffer, allocate an output buffer of size 1% larger than the
Chris@4: uncompressed data, plus six hundred extra bytes.</para>
Chris@4: 
Chris@4: <para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
Chris@4: will not write data at or beyond
Chris@4: <computeroutput>dest[*destLen]</computeroutput>, even in case of
Chris@4: buffer overflow.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if dest is NULL or destLen is NULL
Chris@4:   or blockSize100k < 1 or blockSize100k > 9
Chris@4:   or verbosity < 0 or verbosity > 4
Chris@4:   or workFactor < 0 or workFactor > 250
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory is available 
Chris@4: BZ_OUTBUFF_FULL
Chris@4:   if the size of the compressed data exceeds *destLen
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="bzbufftobuffdecompress" xreflabel="BZ2_bzBuffToBuffDecompress">
Chris@4: <title>BZ2_bzBuffToBuffDecompress</title>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzBuffToBuffDecompress( char*         dest,
Chris@4:                                 unsigned int* destLen,
Chris@4:                                 char*         source,
Chris@4:                                 unsigned int  sourceLen,
Chris@4:                                 int           small,
Chris@4:                                 int           verbosity );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Attempts to decompress the data in <computeroutput>source[0
Chris@4: .. sourceLen-1]</computeroutput> into the destination buffer,
Chris@4: <computeroutput>dest[0 .. *destLen-1]</computeroutput>.  If the
Chris@4: destination buffer is big enough,
Chris@4: <computeroutput>*destLen</computeroutput> is set to the size of
Chris@4: the uncompressed data, and <computeroutput>BZ_OK</computeroutput>
Chris@4: is returned.  If the compressed data won't fit,
Chris@4: <computeroutput>*destLen</computeroutput> is unchanged, and
Chris@4: <computeroutput>BZ_OUTBUFF_FULL</computeroutput> is
Chris@4: returned.</para>
Chris@4: 
Chris@4: <para><computeroutput>source</computeroutput> is assumed to hold
Chris@4: a complete <computeroutput>bzip2</computeroutput> format data
Chris@4: stream.
Chris@4: <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput> tries
Chris@4: to decompress the entirety of the stream into the output
Chris@4: buffer.</para>
Chris@4: 
Chris@4: <para>For the meaning of parameters
Chris@4: <computeroutput>small</computeroutput> and
Chris@4: <computeroutput>verbosity</computeroutput>, see
Chris@4: <computeroutput>BZ2_bzDecompressInit</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Because the compression ratio of the compressed data cannot
Chris@4: be known in advance, there is no easy way to guarantee that the
Chris@4: output buffer will be big enough.  You may of course make
Chris@4: arrangements in your code to record the size of the uncompressed
Chris@4: data, but such a mechanism is beyond the scope of this
Chris@4: library.</para>
Chris@4: 
Chris@4: <para><computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput>
Chris@4: will not write data at or beyond
Chris@4: <computeroutput>dest[*destLen]</computeroutput>, even in case of
Chris@4: buffer overflow.</para>
Chris@4: 
Chris@4: <para>Possible return values:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZ_CONFIG_ERROR
Chris@4:   if the library has been mis-compiled
Chris@4: BZ_PARAM_ERROR
Chris@4:   if dest is NULL or destLen is NULL
Chris@4:   or small != 0 && small != 1
Chris@4:   or verbosity < 0 or verbosity > 4
Chris@4: BZ_MEM_ERROR
Chris@4:   if insufficient memory is available 
Chris@4: BZ_OUTBUFF_FULL
Chris@4:   if the size of the compressed data exceeds *destLen
Chris@4: BZ_DATA_ERROR
Chris@4:   if a data integrity error was detected in the compressed data
Chris@4: BZ_DATA_ERROR_MAGIC
Chris@4:   if the compressed data doesn't begin with the right magic bytes
Chris@4: BZ_UNEXPECTED_EOF
Chris@4:   if the compressed data ends unexpectedly
Chris@4: BZ_OK
Chris@4:   otherwise
Chris@4: </programlisting>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="zlib-compat" xreflabel="zlib compatibility functions">
Chris@4: <title>zlib compatibility functions</title>
Chris@4: 
Chris@4: <para>Yoshioka Tsuneo has contributed some functions to give
Chris@4: better <computeroutput>zlib</computeroutput> compatibility.
Chris@4: These functions are <computeroutput>BZ2_bzopen</computeroutput>,
Chris@4: <computeroutput>BZ2_bzread</computeroutput>,
Chris@4: <computeroutput>BZ2_bzwrite</computeroutput>,
Chris@4: <computeroutput>BZ2_bzflush</computeroutput>,
Chris@4: <computeroutput>BZ2_bzclose</computeroutput>,
Chris@4: <computeroutput>BZ2_bzerror</computeroutput> and
Chris@4: <computeroutput>BZ2_bzlibVersion</computeroutput>.  These
Chris@4: functions are not (yet) officially part of the library.  If they
Chris@4: break, you get to keep all the pieces.  Nevertheless, I think
Chris@4: they work ok.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: typedef void BZFILE;
Chris@4: 
Chris@4: const char * BZ2_bzlibVersion ( void );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Returns a string indicating the library version.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: BZFILE * BZ2_bzopen  ( const char *path, const char *mode );
Chris@4: BZFILE * BZ2_bzdopen ( int        fd,    const char *mode );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Opens a <computeroutput>.bz2</computeroutput> file for
Chris@4: reading or writing, using either its name or a pre-existing file
Chris@4: descriptor.  Analogous to <computeroutput>fopen</computeroutput>
Chris@4: and <computeroutput>fdopen</computeroutput>.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int BZ2_bzread  ( BZFILE* b, void* buf, int len );
Chris@4: int BZ2_bzwrite ( BZFILE* b, void* buf, int len );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Reads/writes data from/to a previously opened
Chris@4: <computeroutput>BZFILE</computeroutput>.  Analogous to
Chris@4: <computeroutput>fread</computeroutput> and
Chris@4: <computeroutput>fwrite</computeroutput>.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: int  BZ2_bzflush ( BZFILE* b );
Chris@4: void BZ2_bzclose ( BZFILE* b );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Flushes/closes a <computeroutput>BZFILE</computeroutput>.
Chris@4: <computeroutput>BZ2_bzflush</computeroutput> doesn't actually do
Chris@4: anything.  Analogous to <computeroutput>fflush</computeroutput>
Chris@4: and <computeroutput>fclose</computeroutput>.</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: const char * BZ2_bzerror ( BZFILE *b, int *errnum )
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>Returns a string describing the more recent error status of
Chris@4: <computeroutput>b</computeroutput>, and also sets
Chris@4: <computeroutput>*errnum</computeroutput> to its numerical
Chris@4: value.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="stdio-free" 
Chris@4:        xreflabel="Using the library in a stdio-free environment">
Chris@4: <title>Using the library in a stdio-free environment</title>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="stdio-bye" xreflabel="Getting rid of stdio">
Chris@4: <title>Getting rid of stdio</title>
Chris@4: 
Chris@4: <para>In a deeply embedded application, you might want to use
Chris@4: just the memory-to-memory functions.  You can do this
Chris@4: conveniently by compiling the library with preprocessor symbol
Chris@4: <computeroutput>BZ_NO_STDIO</computeroutput> defined.  Doing this
Chris@4: gives you a library containing only the following eight
Chris@4: functions:</para>
Chris@4: 
Chris@4: <para><computeroutput>BZ2_bzCompressInit</computeroutput>,
Chris@4: <computeroutput>BZ2_bzCompress</computeroutput>,
Chris@4: <computeroutput>BZ2_bzCompressEnd</computeroutput>
Chris@4: <computeroutput>BZ2_bzDecompressInit</computeroutput>,
Chris@4: <computeroutput>BZ2_bzDecompress</computeroutput>,
Chris@4: <computeroutput>BZ2_bzDecompressEnd</computeroutput>
Chris@4: <computeroutput>BZ2_bzBuffToBuffCompress</computeroutput>,
Chris@4: <computeroutput>BZ2_bzBuffToBuffDecompress</computeroutput></para>
Chris@4: 
Chris@4: <para>When compiled like this, all functions will ignore
Chris@4: <computeroutput>verbosity</computeroutput> settings.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: 
Chris@4: <sect2 id="critical-error" xreflabel="Critical error handling">
Chris@4: <title>Critical error handling</title>
Chris@4: 
Chris@4: <para><computeroutput>libbzip2</computeroutput> contains a number
Chris@4: of internal assertion checks which should, needless to say, never
Chris@4: be activated.  Nevertheless, if an assertion should fail,
Chris@4: behaviour depends on whether or not the library was compiled with
Chris@4: <computeroutput>BZ_NO_STDIO</computeroutput> set.</para>
Chris@4: 
Chris@4: <para>For a normal compile, an assertion failure yields the
Chris@4: message:</para>
Chris@4: 
Chris@4: <blockquote>
Chris@4: <para>bzip2/libbzip2: internal error number N.</para>
Chris@4: <para>This is a bug in bzip2/libbzip2, &bz-version; of &bz-date;.
Chris@4: Please report it to me at: &bz-email;.  If this happened
Chris@4: when you were using some program which uses libbzip2 as a
Chris@4: component, you should also report this bug to the author(s)
Chris@4: of that program.  Please make an effort to report this bug;
Chris@4: timely and accurate bug reports eventually lead to higher
Chris@4: quality software.  Thanks.  Julian Seward, &bz-date;.
Chris@4: </para></blockquote>
Chris@4: 
Chris@4: <para>where <computeroutput>N</computeroutput> is some error code
Chris@4: number.  If <computeroutput>N == 1007</computeroutput>, it also
Chris@4: prints some extra text advising the reader that unreliable memory
Chris@4: is often associated with internal error 1007. (This is a
Chris@4: frequently-observed-phenomenon with versions 1.0.0/1.0.1).</para>
Chris@4: 
Chris@4: <para><computeroutput>exit(3)</computeroutput> is then
Chris@4: called.</para>
Chris@4: 
Chris@4: <para>For a <computeroutput>stdio</computeroutput>-free library,
Chris@4: assertion failures result in a call to a function declared
Chris@4: as:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: extern void bz_internal_error ( int errcode );
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>The relevant code is passed as a parameter.  You should
Chris@4: supply such a function.</para>
Chris@4: 
Chris@4: <para>In either case, once an assertion failure has occurred, any
Chris@4: <computeroutput>bz_stream</computeroutput> records involved can
Chris@4: be regarded as invalid.  You should not attempt to resume normal
Chris@4: operation with them.</para>
Chris@4: 
Chris@4: <para>You may, of course, change critical error handling to suit
Chris@4: your needs.  As I said above, critical errors indicate bugs in
Chris@4: the library and should not occur.  All "normal" error situations
Chris@4: are indicated via error return codes from functions, and can be
Chris@4: recovered from.</para>
Chris@4: 
Chris@4: </sect2>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="win-dll" xreflabel="Making a Windows DLL">
Chris@4: <title>Making a Windows DLL</title>
Chris@4: 
Chris@4: <para>Everything related to Windows has been contributed by
Chris@4: Yoshioka Tsuneo
Chris@4: (<computeroutput>tsuneo@rr.iij4u.or.jp</computeroutput>), so
Chris@4: you should send your queries to him (but perhaps Cc: me,
Chris@4: <computeroutput>&bz-email;</computeroutput>).</para>
Chris@4: 
Chris@4: <para>My vague understanding of what to do is: using Visual C++
Chris@4: 5.0, open the project file
Chris@4: <computeroutput>libbz2.dsp</computeroutput>, and build.  That's
Chris@4: all.</para>
Chris@4: 
Chris@4: <para>If you can't open the project file for some reason, make a
Chris@4: new one, naming these files:
Chris@4: <computeroutput>blocksort.c</computeroutput>,
Chris@4: <computeroutput>bzlib.c</computeroutput>,
Chris@4: <computeroutput>compress.c</computeroutput>,
Chris@4: <computeroutput>crctable.c</computeroutput>,
Chris@4: <computeroutput>decompress.c</computeroutput>,
Chris@4: <computeroutput>huffman.c</computeroutput>,
Chris@4: <computeroutput>randtable.c</computeroutput> and
Chris@4: <computeroutput>libbz2.def</computeroutput>.  You will also need
Chris@4: to name the header files <computeroutput>bzlib.h</computeroutput>
Chris@4: and <computeroutput>bzlib_private.h</computeroutput>.</para>
Chris@4: 
Chris@4: <para>If you don't use VC++, you may need to define the
Chris@4: proprocessor symbol
Chris@4: <computeroutput>_WIN32</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Finally, <computeroutput>dlltest.c</computeroutput> is a
Chris@4: sample program using the DLL.  It has a project file,
Chris@4: <computeroutput>dlltest.dsp</computeroutput>.</para>
Chris@4: 
Chris@4: <para>If you just want a makefile for Visual C, have a look at
Chris@4: <computeroutput>makefile.msc</computeroutput>.</para>
Chris@4: 
Chris@4: <para>Be aware that if you compile
Chris@4: <computeroutput>bzip2</computeroutput> itself on Win32, you must
Chris@4: set <computeroutput>BZ_UNIX</computeroutput> to 0 and
Chris@4: <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the file
Chris@4: <computeroutput>bzip2.c</computeroutput>, before compiling.
Chris@4: Otherwise the resulting binary won't work correctly.</para>
Chris@4: 
Chris@4: <para>I haven't tried any of this stuff myself, but it all looks
Chris@4: plausible.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: </chapter>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <chapter id="misc" xreflabel="Miscellanea">
Chris@4: <title>Miscellanea</title>
Chris@4: 
Chris@4: <para>These are just some random thoughts of mine.  Your mileage
Chris@4: may vary.</para>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="limits" xreflabel="Limitations of the compressed file format">
Chris@4: <title>Limitations of the compressed file format</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2-1.0.X</computeroutput>,
Chris@4: <computeroutput>0.9.5</computeroutput> and
Chris@4: <computeroutput>0.9.0</computeroutput> use exactly the same file
Chris@4: format as the original version,
Chris@4: <computeroutput>bzip2-0.1</computeroutput>.  This decision was
Chris@4: made in the interests of stability.  Creating yet another
Chris@4: incompatible compressed file format would create further
Chris@4: confusion and disruption for users.</para>
Chris@4: 
Chris@4: <para>Nevertheless, this is not a painless decision.  Development
Chris@4: work since the release of
Chris@4: <computeroutput>bzip2-0.1</computeroutput> in August 1997 has
Chris@4: shown complexities in the file format which slow down
Chris@4: decompression and, in retrospect, are unnecessary.  These
Chris@4: are:</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para>The run-length encoder, which is the first of the
Chris@4:    compression transformations, is entirely irrelevant.  The
Chris@4:    original purpose was to protect the sorting algorithm from the
Chris@4:    very worst case input: a string of repeated symbols.  But
Chris@4:    algorithm steps Q6a and Q6b in the original Burrows-Wheeler
Chris@4:    technical report (SRC-124) show how repeats can be handled
Chris@4:    without difficulty in block sorting.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>The randomisation mechanism doesn't really need to be
Chris@4:    there.  Udi Manber and Gene Myers published a suffix array
Chris@4:    construction algorithm a few years back, which can be employed
Chris@4:    to sort any block, no matter how repetitive, in O(N log N)
Chris@4:    time.  Subsequent work by Kunihiko Sadakane has produced a
Chris@4:    derivative O(N (log N)^2) algorithm which usually outperforms
Chris@4:    the Manber-Myers algorithm.</para>
Chris@4: 
Chris@4:    <para>I could have changed to Sadakane's algorithm, but I find
Chris@4:    it to be slower than <computeroutput>bzip2</computeroutput>'s
Chris@4:    existing algorithm for most inputs, and the randomisation
Chris@4:    mechanism protects adequately against bad cases.  I didn't
Chris@4:    think it was a good tradeoff to make.  Partly this is due to
Chris@4:    the fact that I was not flooded with email complaints about
Chris@4:    <computeroutput>bzip2-0.1</computeroutput>'s performance on
Chris@4:    repetitive data, so perhaps it isn't a problem for real
Chris@4:    inputs.</para>
Chris@4: 
Chris@4:    <para>Probably the best long-term solution, and the one I have
Chris@4:    incorporated into 0.9.5 and above, is to use the existing
Chris@4:    sorting algorithm initially, and fall back to a O(N (log N)^2)
Chris@4:    algorithm if the standard algorithm gets into
Chris@4:    difficulties.</para></listitem>
Chris@4: 
Chris@4:   <listitem><para>The compressed file format was never designed to be
Chris@4:    handled by a library, and I have had to jump though some hoops
Chris@4:    to produce an efficient implementation of decompression.  It's
Chris@4:    a bit hairy.  Try passing
Chris@4:    <computeroutput>decompress.c</computeroutput> through the C
Chris@4:    preprocessor and you'll see what I mean.  Much of this
Chris@4:    complexity could have been avoided if the compressed size of
Chris@4:    each block of data was recorded in the data stream.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>An Adler-32 checksum, rather than a CRC32 checksum,
Chris@4:    would be faster to compute.</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: <para>It would be fair to say that the
Chris@4: <computeroutput>bzip2</computeroutput> format was frozen before I
Chris@4: properly and fully understood the performance consequences of
Chris@4: doing so.</para>
Chris@4: 
Chris@4: <para>Improvements which I was able to incorporate into 0.9.0,
Chris@4: despite using the same file format, are:</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para>Single array implementation of the inverse BWT.  This
Chris@4:   significantly speeds up decompression, presumably because it
Chris@4:   reduces the number of cache misses.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>Faster inverse MTF transform for large MTF values.
Chris@4:   The new implementation is based on the notion of sliding blocks
Chris@4:   of values.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para><computeroutput>bzip2-0.9.0</computeroutput> now reads
Chris@4:   and writes files with <computeroutput>fread</computeroutput>
Chris@4:   and <computeroutput>fwrite</computeroutput>; version 0.1 used
Chris@4:   <computeroutput>putc</computeroutput> and
Chris@4:   <computeroutput>getc</computeroutput>.  Duh!  Well, you live
Chris@4:   and learn.</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: <para>Further ahead, it would be nice to be able to do random
Chris@4: access into files.  This will require some careful design of
Chris@4: compressed file formats.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="port-issues" xreflabel="Portability issues">
Chris@4: <title>Portability issues</title>
Chris@4: 
Chris@4: <para>After some consideration, I have decided not to use GNU
Chris@4: <computeroutput>autoconf</computeroutput> to configure 0.9.5 or
Chris@4: 1.0.</para>
Chris@4: 
Chris@4: <para><computeroutput>autoconf</computeroutput>, admirable and
Chris@4: wonderful though it is, mainly assists with portability problems
Chris@4: between Unix-like platforms.  But
Chris@4: <computeroutput>bzip2</computeroutput> doesn't have much in the
Chris@4: way of portability problems on Unix; most of the difficulties
Chris@4: appear when porting to the Mac, or to Microsoft's operating
Chris@4: systems.  <computeroutput>autoconf</computeroutput> doesn't help
Chris@4: in those cases, and brings in a whole load of new
Chris@4: complexity.</para>
Chris@4: 
Chris@4: <para>Most people should be able to compile the library and
Chris@4: program under Unix straight out-of-the-box, so to speak,
Chris@4: especially if you have a version of GNU C available.</para>
Chris@4: 
Chris@4: <para>There are a couple of
Chris@4: <computeroutput>__inline__</computeroutput> directives in the
Chris@4: code.  GNU C (<computeroutput>gcc</computeroutput>) should be
Chris@4: able to handle them.  If you're not using GNU C, your C compiler
Chris@4: shouldn't see them at all.  If your compiler does, for some
Chris@4: reason, see them and doesn't like them, just
Chris@4: <computeroutput>#define</computeroutput>
Chris@4: <computeroutput>__inline__</computeroutput> to be
Chris@4: <computeroutput>/* */</computeroutput>.  One easy way to do this
Chris@4: is to compile with the flag
Chris@4: <computeroutput>-D__inline__=</computeroutput>, which should be
Chris@4: understood by most Unix compilers.</para>
Chris@4: 
Chris@4: <para>If you still have difficulties, try compiling with the
Chris@4: macro <computeroutput>BZ_STRICT_ANSI</computeroutput> defined.
Chris@4: This should enable you to build the library in a strictly ANSI
Chris@4: compliant environment.  Building the program itself like this is
Chris@4: dangerous and not supported, since you remove
Chris@4: <computeroutput>bzip2</computeroutput>'s checks against
Chris@4: compressing directories, symbolic links, devices, and other
Chris@4: not-really-a-file entities.  This could cause filesystem
Chris@4: corruption!</para>
Chris@4: 
Chris@4: <para>One other thing: if you create a
Chris@4: <computeroutput>bzip2</computeroutput> binary for public distribution,
Chris@4: please consider linking it statically (<computeroutput>gcc
Chris@4: -static</computeroutput>).  This avoids all sorts of library-version
Chris@4: issues that others may encounter later on.</para>
Chris@4: 
Chris@4: <para>If you build <computeroutput>bzip2</computeroutput> on
Chris@4: Win32, you must set <computeroutput>BZ_UNIX</computeroutput> to 0
Chris@4: and <computeroutput>BZ_LCCWIN32</computeroutput> to 1, in the
Chris@4: file <computeroutput>bzip2.c</computeroutput>, before compiling.
Chris@4: Otherwise the resulting binary won't work correctly.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="bugs" xreflabel="Reporting bugs">
Chris@4: <title>Reporting bugs</title>
Chris@4: 
Chris@4: <para>I tried pretty hard to make sure
Chris@4: <computeroutput>bzip2</computeroutput> is bug free, both by
Chris@4: design and by testing.  Hopefully you'll never need to read this
Chris@4: section for real.</para>
Chris@4: 
Chris@4: <para>Nevertheless, if <computeroutput>bzip2</computeroutput> dies
Chris@4: with a segmentation fault, a bus error or an internal assertion
Chris@4: failure, it will ask you to email me a bug report.  Experience from
Chris@4: years of feedback of bzip2 users indicates that almost all these
Chris@4: problems can be traced to either compiler bugs or hardware
Chris@4: problems.</para>
Chris@4: 
Chris@4: <itemizedlist mark='bullet'>
Chris@4: 
Chris@4:  <listitem><para>Recompile the program with no optimisation, and
Chris@4:   see if it works.  And/or try a different compiler.  I heard all
Chris@4:   sorts of stories about various flavours of GNU C (and other
Chris@4:   compilers) generating bad code for
Chris@4:   <computeroutput>bzip2</computeroutput>, and I've run across two
Chris@4:   such examples myself.</para>
Chris@4: 
Chris@4:   <para>2.7.X versions of GNU C are known to generate bad code
Chris@4:   from time to time, at high optimisation levels.  If you get
Chris@4:   problems, try using the flags
Chris@4:   <computeroutput>-O2</computeroutput>
Chris@4:   <computeroutput>-fomit-frame-pointer</computeroutput>
Chris@4:   <computeroutput>-fno-strength-reduce</computeroutput>.  You
Chris@4:   should specifically <emphasis>not</emphasis> use
Chris@4:   <computeroutput>-funroll-loops</computeroutput>.</para>
Chris@4: 
Chris@4:   <para>You may notice that the Makefile runs six tests as part
Chris@4:   of the build process.  If the program passes all of these, it's
Chris@4:   a pretty good (but not 100%) indication that the compiler has
Chris@4:   done its job correctly.</para></listitem>
Chris@4: 
Chris@4:  <listitem><para>If <computeroutput>bzip2</computeroutput>
Chris@4:   crashes randomly, and the crashes are not repeatable, you may
Chris@4:   have a flaky memory subsystem.
Chris@4:   <computeroutput>bzip2</computeroutput> really hammers your
Chris@4:   memory hierarchy, and if it's a bit marginal, you may get these
Chris@4:   problems.  Ditto if your disk or I/O subsystem is slowly
Chris@4:   failing.  Yup, this really does happen.</para>
Chris@4: 
Chris@4:   <para>Try using a different machine of the same type, and see
Chris@4:   if you can repeat the problem.</para></listitem>
Chris@4: 
Chris@4:   <listitem><para>This isn't really a bug, but ... If
Chris@4:   <computeroutput>bzip2</computeroutput> tells you your file is
Chris@4:   corrupted on decompression, and you obtained the file via FTP,
Chris@4:   there is a possibility that you forgot to tell FTP to do a
Chris@4:   binary mode transfer.  That absolutely will cause the file to
Chris@4:   be non-decompressible.  You'll have to transfer it
Chris@4:   again.</para></listitem>
Chris@4: 
Chris@4: </itemizedlist>
Chris@4: 
Chris@4: <para>If you've incorporated
Chris@4: <computeroutput>libbzip2</computeroutput> into your own program
Chris@4: and are getting problems, please, please, please, check that the
Chris@4: parameters you are passing in calls to the library, are correct,
Chris@4: and in accordance with what the documentation says is allowable.
Chris@4: I have tried to make the library robust against such problems,
Chris@4: but I'm sure I haven't succeeded.</para>
Chris@4: 
Chris@4: <para>Finally, if the above comments don't help, you'll have to
Chris@4: send me a bug report.  Now, it's just amazing how many people
Chris@4: will send me a bug report saying something like:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: bzip2 crashed with segmentation fault on my machine
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>and absolutely nothing else.  Needless to say, a such a
Chris@4: report is <emphasis>totally, utterly, completely and
Chris@4: comprehensively 100% useless; a waste of your time, my time, and
Chris@4: net bandwidth</emphasis>.  With no details at all, there's no way
Chris@4: I can possibly begin to figure out what the problem is.</para>
Chris@4: 
Chris@4: <para>The rules of the game are: facts, facts, facts.  Don't omit
Chris@4: them because "oh, they won't be relevant".  At the bare
Chris@4: minimum:</para>
Chris@4: 
Chris@4: <programlisting>
Chris@4: Machine type.  Operating system version.  
Chris@4: Exact version of bzip2 (do bzip2 -V).  
Chris@4: Exact version of the compiler used.  
Chris@4: Flags passed to the compiler.
Chris@4: </programlisting>
Chris@4: 
Chris@4: <para>However, the most important single thing that will help me
Chris@4: is the file that you were trying to compress or decompress at the
Chris@4: time the problem happened.  Without that, my ability to do
Chris@4: anything more than speculate about the cause, is limited.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="package" xreflabel="Did you get the right package?">
Chris@4: <title>Did you get the right package?</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> is a resource hog.
Chris@4: It soaks up large amounts of CPU cycles and memory.  Also, it
Chris@4: gives very large latencies.  In the worst case, you can feed many
Chris@4: megabytes of uncompressed data into the library before getting
Chris@4: any compressed output, so this probably rules out applications
Chris@4: requiring interactive behaviour.</para>
Chris@4: 
Chris@4: <para>These aren't faults of my implementation, I hope, but more
Chris@4: an intrinsic property of the Burrows-Wheeler transform
Chris@4: (unfortunately).  Maybe this isn't what you want.</para>
Chris@4: 
Chris@4: <para>If you want a compressor and/or library which is faster,
Chris@4: uses less memory but gets pretty good compression, and has
Chris@4: minimal latency, consider Jean-loup Gailly's and Mark Adler's
Chris@4: work, <computeroutput>zlib-1.2.1</computeroutput> and
Chris@4: <computeroutput>gzip-1.2.4</computeroutput>.  Look for them at 
Chris@4: <ulink url="http://www.zlib.org">http://www.zlib.org</ulink> and 
Chris@4: <ulink url="http://www.gzip.org">http://www.gzip.org</ulink>
Chris@4: respectively.</para>
Chris@4: 
Chris@4: <para>For something faster and lighter still, you might try Markus F
Chris@4: X J Oberhumer's <computeroutput>LZO</computeroutput> real-time
Chris@4: compression/decompression library, at 
Chris@4: <ulink url="http://www.oberhumer.com/opensource">http://www.oberhumer.com/opensource</ulink>.</para>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: 
Chris@4: 
Chris@4: <sect1 id="reading" xreflabel="Further Reading">
Chris@4: <title>Further Reading</title>
Chris@4: 
Chris@4: <para><computeroutput>bzip2</computeroutput> is not research
Chris@4: work, in the sense that it doesn't present any new ideas.
Chris@4: Rather, it's an engineering exercise based on existing
Chris@4: ideas.</para>
Chris@4: 
Chris@4: <para>Four documents describe essentially all the ideas behind
Chris@4: <computeroutput>bzip2</computeroutput>:</para>
Chris@4: 
Chris@4: <literallayout>Michael Burrows and D. J. Wheeler:
Chris@4:   "A block-sorting lossless data compression algorithm"
Chris@4:    10th May 1994. 
Chris@4:    Digital SRC Research Report 124.
Chris@4:    ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz
Chris@4:    If you have trouble finding it, try searching at the
Chris@4:    New Zealand Digital Library, http://www.nzdl.org.
Chris@4: 
Chris@4: Daniel S. Hirschberg and Debra A. LeLewer
Chris@4:   "Efficient Decoding of Prefix Codes"
Chris@4:    Communications of the ACM, April 1990, Vol 33, Number 4.
Chris@4:    You might be able to get an electronic copy of this
Chris@4:    from the ACM Digital Library.
Chris@4: 
Chris@4: David J. Wheeler
Chris@4:    Program bred3.c and accompanying document bred3.ps.
Chris@4:    This contains the idea behind the multi-table Huffman coding scheme.
Chris@4:    ftp://ftp.cl.cam.ac.uk/users/djw3/
Chris@4: 
Chris@4: Jon L. Bentley and Robert Sedgewick
Chris@4:   "Fast Algorithms for Sorting and Searching Strings"
Chris@4:    Available from Sedgewick's web page,
Chris@4:    www.cs.princeton.edu/~rs
Chris@4: </literallayout>
Chris@4: 
Chris@4: <para>The following paper gives valuable additional insights into
Chris@4: the algorithm, but is not immediately the basis of any code used
Chris@4: in bzip2.</para>
Chris@4: 
Chris@4: <literallayout>Peter Fenwick:
Chris@4:    Block Sorting Text Compression
Chris@4:    Proceedings of the 19th Australasian Computer Science Conference,
Chris@4:      Melbourne, Australia.  Jan 31 - Feb 2, 1996.
Chris@4:    ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps</literallayout>
Chris@4: 
Chris@4: <para>Kunihiko Sadakane's sorting algorithm, mentioned above, is
Chris@4: available from:</para>
Chris@4: 
Chris@4: <literallayout>http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
Chris@4: </literallayout>
Chris@4: 
Chris@4: <para>The Manber-Myers suffix array construction algorithm is
Chris@4: described in a paper available from:</para>
Chris@4: 
Chris@4: <literallayout>http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
Chris@4: </literallayout>
Chris@4: 
Chris@4: <para>Finally, the following papers document some
Chris@4: investigations I made into the performance of sorting
Chris@4: and decompression algorithms:</para>
Chris@4: 
Chris@4: <literallayout>Julian Seward
Chris@4:    On the Performance of BWT Sorting Algorithms
Chris@4:    Proceedings of the IEEE Data Compression Conference 2000
Chris@4:      Snowbird, Utah.  28-30 March 2000.
Chris@4: 
Chris@4: Julian Seward
Chris@4:    Space-time Tradeoffs in the Inverse B-W Transform
Chris@4:    Proceedings of the IEEE Data Compression Conference 2001
Chris@4:      Snowbird, Utah.  27-29 March 2001.
Chris@4: </literallayout>
Chris@4: 
Chris@4: </sect1>
Chris@4: 
Chris@4: </chapter>
Chris@4: 
Chris@4: </book>