Version 1.0.6 of 6 September 2010
Copyright © 1996-2010 Julian Seward
This program, bzip2, the
Chris@4: associated library libbzip2, and
Chris@4: all documentation, are copyright © 1996-2010 Julian Seward.
Chris@4: All rights reserved.
Redistribution and use in source and binary forms, with Chris@4: or without modification, are permitted provided that the Chris@4: following conditions are met:
Chris@4:Redistributions of source code must retain the Chris@4: above copyright notice, this list of conditions and the Chris@4: following disclaimer.
The origin of this software must not be Chris@4: misrepresented; you must not claim that you wrote the original Chris@4: software. If you use this software in a product, an Chris@4: acknowledgment in the product documentation would be Chris@4: appreciated but is not required.
Altered source versions must be plainly marked Chris@4: as such, and must not be misrepresented as being the original Chris@4: software.
The name of the author may not be used to Chris@4: endorse or promote products derived from this software without Chris@4: specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY Chris@4: EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, Chris@4: THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A Chris@4: PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE Chris@4: AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, Chris@4: EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED Chris@4: TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, Chris@4: DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND Chris@4: ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT Chris@4: LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING Chris@4: IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF Chris@4: THE POSSIBILITY OF SUCH DAMAGE.
Chris@4:PATENTS: To the best of my knowledge,
Chris@4: bzip2 and
Chris@4: libbzip2 do not use any patented
Chris@4: algorithms. However, I do not have the resources to carry
Chris@4: out a patent search. Therefore I cannot give any guarantee of
Chris@4: the above statement.
Chris@4:
Table of Contents
Chris@4:libbzip2
Chris@4: bzip2 compresses files
Chris@4: using the Burrows-Wheeler block-sorting text compression
Chris@4: algorithm, and Huffman coding. Compression is generally
Chris@4: considerably better than that achieved by more conventional
Chris@4: LZ77/LZ78-based compressors, and approaches the performance of
Chris@4: the PPM family of statistical compressors.
bzip2 is built on top of
Chris@4: libbzip2, a flexible library for
Chris@4: handling compressed data in the
Chris@4: bzip2 format. This manual
Chris@4: describes both how to use the program and how to work with the
Chris@4: library interface. Most of the manual is devoted to this
Chris@4: library, not the program, which is good news if your interest is
Chris@4: only in the program.
How to use bzip2 describes how to use
Chris@4: bzip2; this is the only part
Chris@4: you need to read if you just want to know how to operate the
Chris@4: program.
Programming with libbzip2 describes the Chris@4: programming interfaces in detail, and
Miscellanea records some Chris@4: miscellaneous notes which I thought ought to be recorded Chris@4: somewhere.
Table of Contents
Chris@4:This chapter contains a copy of the
Chris@4: bzip2 man page, and nothing
Chris@4: else.
bzip2,
Chris@4: bunzip2 - a block-sorting file
Chris@4: compressor, v1.0.6
bzcat -
Chris@4: decompresses files to stdout
bzip2recover -
Chris@4: recovers data from damaged bzip2 files
bzip2 [
Chris@4: -cdfkqstvzVL123456789 ] [ filenames ... ]
bunzip2 [
Chris@4: -fkvsVL ] [ filenames ... ]
bzcat [ -s ] [
Chris@4: filenames ... ]
bzip2recover
Chris@4: filename
bzip2 compresses files
Chris@4: using the Burrows-Wheeler block sorting text compression
Chris@4: algorithm, and Huffman coding. Compression is generally
Chris@4: considerably better than that achieved by more conventional
Chris@4: LZ77/LZ78-based compressors, and approaches the performance of
Chris@4: the PPM family of statistical compressors.
The command-line options are deliberately very similar to
Chris@4: those of GNU gzip, but they are
Chris@4: not identical.
bzip2 expects a list of
Chris@4: file names to accompany the command-line flags. Each file is
Chris@4: replaced by a compressed version of itself, with the name
Chris@4: original_name.bz2. Each
Chris@4: compressed file has the same modification date, permissions, and,
Chris@4: when possible, ownership as the corresponding original, so that
Chris@4: these properties can be correctly restored at decompression time.
Chris@4: File name handling is naive in the sense that there is no
Chris@4: mechanism for preserving original file names, permissions,
Chris@4: ownerships or dates in filesystems which lack these concepts, or
Chris@4: have serious file name length restrictions, such as
Chris@4: MS-DOS.
bzip2 and
Chris@4: bunzip2 will by default not
Chris@4: overwrite existing files. If you want this to happen, specify
Chris@4: the -f flag.
If no file names are specified,
Chris@4: bzip2 compresses from standard
Chris@4: input to standard output. In this case,
Chris@4: bzip2 will decline to write
Chris@4: compressed output to a terminal, as this would be entirely
Chris@4: incomprehensible and therefore pointless.
bunzip2 (or
Chris@4: bzip2 -d) decompresses all
Chris@4: specified files. Files which were not created by
Chris@4: bzip2 will be detected and
Chris@4: ignored, and a warning issued.
Chris@4: bzip2 attempts to guess the
Chris@4: filename for the decompressed file from that of the compressed
Chris@4: file as follows:
filename.bz2
Chris@4: becomes
Chris@4: filename
filename.bz
Chris@4: becomes
Chris@4: filename
filename.tbz2
Chris@4: becomes
Chris@4: filename.tar
filename.tbz
Chris@4: becomes
Chris@4: filename.tar
anyothername
Chris@4: becomes
Chris@4: anyothername.out
If the file does not end in one of the recognised endings,
Chris@4: .bz2,
Chris@4: .bz,
Chris@4: .tbz2 or
Chris@4: .tbz,
Chris@4: bzip2 complains that it cannot
Chris@4: guess the name of the original file, and uses the original name
Chris@4: with .out appended.
As with compression, supplying no filenames causes Chris@4: decompression from standard input to standard output.
Chris@4:bunzip2 will correctly
Chris@4: decompress a file which is the concatenation of two or more
Chris@4: compressed files. The result is the concatenation of the
Chris@4: corresponding uncompressed files. Integrity testing
Chris@4: (-t) of concatenated compressed
Chris@4: files is also supported.
You can also compress or decompress files to the standard
Chris@4: output by giving the -c flag.
Chris@4: Multiple files may be compressed and decompressed like this. The
Chris@4: resulting outputs are fed sequentially to stdout. Compression of
Chris@4: multiple files in this manner generates a stream containing
Chris@4: multiple compressed file representations. Such a stream can be
Chris@4: decompressed correctly only by
Chris@4: bzip2 version 0.9.0 or later.
Chris@4: Earlier versions of bzip2 will
Chris@4: stop after decompressing the first file in the stream.
bzcat (or
Chris@4: bzip2 -dc) decompresses all
Chris@4: specified files to the standard output.
bzip2 will read arguments
Chris@4: from the environment variables
Chris@4: BZIP2 and
Chris@4: BZIP, in that order, and will
Chris@4: process them before any arguments read from the command line.
Chris@4: This gives a convenient way to supply default arguments.
Compression is always performed, even if the compressed Chris@4: file is slightly larger than the original. Files of less than Chris@4: about one hundred bytes tend to get larger, since the compression Chris@4: mechanism has a constant overhead in the region of 50 bytes. Chris@4: Random data (including the output of most file compressors) is Chris@4: coded at about 8.05 bits per byte, giving an expansion of around Chris@4: 0.5%.
Chris@4:As a self-check for your protection,
Chris@4: bzip2 uses 32-bit CRCs to make
Chris@4: sure that the decompressed version of a file is identical to the
Chris@4: original. This guards against corruption of the compressed data,
Chris@4: and against undetected bugs in
Chris@4: bzip2 (hopefully very unlikely).
Chris@4: The chances of data corruption going undetected is microscopic,
Chris@4: about one chance in four billion for each file processed. Be
Chris@4: aware, though, that the check occurs upon decompression, so it
Chris@4: can only tell you that something is wrong. It can't help you
Chris@4: recover the original uncompressed data. You can use
Chris@4: bzip2recover to try to recover
Chris@4: data from damaged files.
Return values: 0 for a normal exit, 1 for environmental
Chris@4: problems (file not found, invalid flags, I/O errors, etc.), 2
Chris@4: to indicate a corrupt compressed file, 3 for an internal
Chris@4: consistency error (eg, bug) which caused
Chris@4: bzip2 to panic.
-c --stdoutCompress or decompress to standard Chris@4: output.
-d --decompressForce decompression.
Chris@4: bzip2,
Chris@4: bunzip2 and
Chris@4: bzcat are really the same
Chris@4: program, and the decision about what actions to take is done on
Chris@4: the basis of which name is used. This flag overrides that
Chris@4: mechanism, and forces bzip2 to decompress.
-z --compressThe complement to
Chris@4: -d: forces compression,
Chris@4: regardless of the invokation name.
-t --testCheck integrity of the specified file(s), but Chris@4: don't decompress them. This really performs a trial Chris@4: decompression and throws away the result.
-f --forceForce overwrite of output files. Normally,
Chris@4: bzip2 will not overwrite
Chris@4: existing output files. Also forces
Chris@4: bzip2 to break hard links to
Chris@4: files, which it otherwise wouldn't do.
bzip2 normally declines
Chris@4: to decompress files which don't have the correct magic header
Chris@4: bytes. If forced (-f),
Chris@4: however, it will pass such files through unmodified. This is
Chris@4: how GNU gzip behaves.
-k --keepKeep (don't delete) input files during Chris@4: compression or decompression.
-s --smallReduce memory usage, for compression, Chris@4: decompression and testing. Files are decompressed and tested Chris@4: using a modified algorithm which only requires 2.5 bytes per Chris@4: block byte. This means any file can be decompressed in 2300k Chris@4: of memory, albeit at about half the normal speed.
Chris@4:During compression, -s
Chris@4: selects a block size of 200k, which limits memory use to around
Chris@4: the same figure, at the expense of your compression ratio. In
Chris@4: short, if your machine is low on memory (8 megabytes or less),
Chris@4: use -s for everything. See
Chris@4: MEMORY MANAGEMENT below.
-q --quietSuppress non-essential warning messages. Chris@4: Messages pertaining to I/O errors and other critical events Chris@4: will not be suppressed.
-v --verboseVerbose mode -- show the compression ratio for
Chris@4: each file processed. Further
Chris@4: -v's increase the verbosity
Chris@4: level, spewing out lots of information which is primarily of
Chris@4: interest for diagnostic purposes.
-L --license -V --versionDisplay the software version, license terms and Chris@4: conditions.
-1 (or
Chris@4: --fast) to
Chris@4: -9 (or
Chris@4: -best)Set the block size to 100 k, 200 k ... 900 k
Chris@4: when compressing. Has no effect when decompressing. See MEMORY MANAGEMENT below. The
Chris@4: --fast and
Chris@4: --best aliases are primarily
Chris@4: for GNU gzip compatibility.
Chris@4: In particular, --fast doesn't
Chris@4: make things significantly faster. And
Chris@4: --best merely selects the
Chris@4: default behaviour.
--Treats all subsequent arguments as file names,
Chris@4: even if they start with a dash. This is so you can handle
Chris@4: files with names beginning with a dash, for example:
Chris@4: bzip2 --
Chris@4: -myfilename.
--repetitive-fast, --repetitive-best
Chris@4: These flags are redundant in versions 0.9.5 and Chris@4: above. They provided some coarse control over the behaviour of Chris@4: the sorting algorithm in earlier versions, which was sometimes Chris@4: useful. 0.9.5 and above have an improved algorithm which Chris@4: renders these flags irrelevant.
bzip2 compresses large
Chris@4: files in blocks. The block size affects both the compression
Chris@4: ratio achieved, and the amount of memory needed for compression
Chris@4: and decompression. The flags -1
Chris@4: through -9 specify the block
Chris@4: size to be 100,000 bytes through 900,000 bytes (the default)
Chris@4: respectively. At decompression time, the block size used for
Chris@4: compression is read from the header of the compressed file, and
Chris@4: bunzip2 then allocates itself
Chris@4: just enough memory to decompress the file. Since block sizes are
Chris@4: stored in compressed files, it follows that the flags
Chris@4: -1 to
Chris@4: -9 are irrelevant to and so
Chris@4: ignored during decompression.
Compression and decompression requirements, in bytes, can be Chris@4: estimated as:
Chris@4:Compression: 400k + ( 8 x block size ) Chris@4: Chris@4: Decompression: 100k + ( 4 x block size ), or Chris@4: 100k + ( 2.5 x block size )Chris@4:
Larger block sizes give rapidly diminishing marginal
Chris@4: returns. Most of the compression comes from the first two or
Chris@4: three hundred k of block size, a fact worth bearing in mind when
Chris@4: using bzip2 on small machines.
Chris@4: It is also important to appreciate that the decompression memory
Chris@4: requirement is set at compression time by the choice of block
Chris@4: size.
For files compressed with the default 900k block size,
Chris@4: bunzip2 will require about 3700
Chris@4: kbytes to decompress. To support decompression of any file on a
Chris@4: 4 megabyte machine, bunzip2 has
Chris@4: an option to decompress using approximately half this amount of
Chris@4: memory, about 2300 kbytes. Decompression speed is also halved,
Chris@4: so you should use this option only where necessary. The relevant
Chris@4: flag is -s.
In general, try and use the largest block size memory Chris@4: constraints allow, since that maximises the compression achieved. Chris@4: Compression and decompression speed are virtually unaffected by Chris@4: block size.
Chris@4:Another significant point applies to files which fit in a
Chris@4: single block -- that means most files you'd encounter using a
Chris@4: large block size. The amount of real memory touched is
Chris@4: proportional to the size of the file, since the file is smaller
Chris@4: than a block. For example, compressing a file 20,000 bytes long
Chris@4: with the flag -9 will cause the
Chris@4: compressor to allocate around 7600k of memory, but only touch
Chris@4: 400k + 20000 * 8 = 560 kbytes of it. Similarly, the decompressor
Chris@4: will allocate 3700k but only touch 100k + 20000 * 4 = 180
Chris@4: kbytes.
Here is a table which summarises the maximum memory usage Chris@4: for different block sizes. Also recorded is the total compressed Chris@4: size for 14 files of the Calgary Text Compression Corpus Chris@4: totalling 3,141,622 bytes. This column gives some feel for how Chris@4: compression varies with block size. These figures tend to Chris@4: understate the advantage of larger block sizes for larger files, Chris@4: since the Corpus is dominated by smaller files.
Chris@4:Compress Decompress Decompress Corpus Chris@4: Flag usage usage -s usage Size Chris@4: Chris@4: -1 1200k 500k 350k 914704 Chris@4: -2 2000k 900k 600k 877703 Chris@4: -3 2800k 1300k 850k 860338 Chris@4: -4 3600k 1700k 1100k 846899 Chris@4: -5 4400k 2100k 1350k 845160 Chris@4: -6 5200k 2500k 1600k 838626 Chris@4: -7 6100k 2900k 1850k 834096 Chris@4: -8 6800k 3300k 2100k 828642 Chris@4: -9 7600k 3700k 2350k 828642Chris@4:
bzip2 compresses files in
Chris@4: blocks, usually 900kbytes long. Each block is handled
Chris@4: independently. If a media or transmission error causes a
Chris@4: multi-block .bz2 file to become
Chris@4: damaged, it may be possible to recover data from the undamaged
Chris@4: blocks in the file.
The compressed representation of each block is delimited by Chris@4: a 48-bit pattern, which makes it possible to find the block Chris@4: boundaries with reasonable certainty. Each block also carries Chris@4: its own 32-bit CRC, so damaged blocks can be distinguished from Chris@4: undamaged ones.
Chris@4:bzip2recover is a simple
Chris@4: program whose purpose is to search for blocks in
Chris@4: .bz2 files, and write each block
Chris@4: out into its own .bz2 file. You
Chris@4: can then use bzip2 -t to test
Chris@4: the integrity of the resulting files, and decompress those which
Chris@4: are undamaged.
bzip2recover takes a
Chris@4: single argument, the name of the damaged file, and writes a
Chris@4: number of files rec0001file.bz2,
Chris@4: rec0002file.bz2, etc, containing
Chris@4: the extracted blocks. The output filenames are designed so that
Chris@4: the use of wildcards in subsequent processing -- for example,
Chris@4: bzip2 -dc rec*file.bz2 >
Chris@4: recovered_data -- lists the files in the correct
Chris@4: order.
bzip2recover should be of
Chris@4: most use dealing with large .bz2
Chris@4: files, as these will contain many blocks. It is clearly futile
Chris@4: to use it on damaged single-block files, since a damaged block
Chris@4: cannot be recovered. If you wish to minimise any potential data
Chris@4: loss through media or transmission errors, you might consider
Chris@4: compressing with a smaller block size.
The sorting phase of compression gathers together similar
Chris@4: strings in the file. Because of this, files containing very long
Chris@4: runs of repeated symbols, like "aabaabaabaab ..." (repeated
Chris@4: several hundred times) may compress more slowly than normal.
Chris@4: Versions 0.9.5 and above fare much better than previous versions
Chris@4: in this respect. The ratio between worst-case and average-case
Chris@4: compression time is in the region of 10:1. For previous
Chris@4: versions, this figure was more like 100:1. You can use the
Chris@4: -vvvv option to monitor progress
Chris@4: in great detail, if you want.
Decompression speed is unaffected by these Chris@4: phenomena.
Chris@4:bzip2 usually allocates
Chris@4: several megabytes of memory to operate in, and then charges all
Chris@4: over it in a fairly random fashion. This means that performance,
Chris@4: both for compressing and decompressing, is largely determined by
Chris@4: the speed at which your machine can service cache misses.
Chris@4: Because of this, small changes to the code to reduce the miss
Chris@4: rate have been observed to give disproportionately large
Chris@4: performance improvements. I imagine
Chris@4: bzip2 will perform best on
Chris@4: machines with very large caches.
I/O error messages are not as helpful as they could be.
Chris@4: bzip2 tries hard to detect I/O
Chris@4: errors and exit cleanly, but the details of what the problem is
Chris@4: sometimes seem rather misleading.
This manual page pertains to version 1.0.6 of
Chris@4: bzip2. Compressed data created by
Chris@4: this version is entirely forwards and backwards compatible with the
Chris@4: previous public releases, versions 0.1pl2, 0.9.0 and 0.9.5, 1.0.0,
Chris@4: 1.0.1, 1.0.2 and 1.0.3, but with the following exception: 0.9.0 and
Chris@4: above can correctly decompress multiple concatenated compressed files.
Chris@4: 0.1pl2 cannot do this; it will stop after decompressing just the first
Chris@4: file in the stream.
bzip2recover versions
Chris@4: prior to 1.0.2 used 32-bit integers to represent bit positions in
Chris@4: compressed files, so it could not handle compressed files more
Chris@4: than 512 megabytes long. Versions 1.0.2 and above use 64-bit ints
Chris@4: on some platforms which support them (GNU supported targets, and
Chris@4: Windows). To establish whether or not
Chris@4: bzip2recover was built with such
Chris@4: a limitation, run it without arguments. In any event you can
Chris@4: build yourself an unlimited version if you can recompile it with
Chris@4: MaybeUInt64 set to be an
Chris@4: unsigned 64-bit integer.
Julian Seward,
Chris@4: jseward@bzip.org
The ideas embodied in
Chris@4: bzip2 are due to (at least) the
Chris@4: following people: Michael Burrows and David Wheeler (for the
Chris@4: block sorting transformation), David Wheeler (again, for the
Chris@4: Huffman coder), Peter Fenwick (for the structured coding model in
Chris@4: the original bzip, and many
Chris@4: refinements), and Alistair Moffat, Radford Neal and Ian Witten
Chris@4: (for the arithmetic coder in the original
Chris@4: bzip). I am much indebted for
Chris@4: their help, support and advice. See the manual in the source
Chris@4: distribution for pointers to sources of documentation. Christian
Chris@4: von Roques encouraged me to look for faster sorting algorithms,
Chris@4: so as to speed up compression. Bela Lubkin encouraged me to
Chris@4: improve the worst-case compression performance.
Chris@4: Donna Robinson XMLised the documentation.
Chris@4: Many people sent
Chris@4: patches, helped with portability problems, lent machines, gave
Chris@4: advice and were generally helpful.
Table of Contents
Chris@4:This chapter describes the programming interface to
Chris@4: libbzip2.
For general background information, particularly about Chris@4: memory use and performance aspects, you'd be well advised to read Chris@4: How to use bzip2 as well.
Chris@4:libbzip2 is a flexible
Chris@4: library for compressing and decompressing data in the
Chris@4: bzip2 data format. Although
Chris@4: packaged as a single entity, it helps to regard the library as
Chris@4: three separate parts: the low level interface, and the high level
Chris@4: interface, and some utility functions.
The structure of
Chris@4: libbzip2's interfaces is similar
Chris@4: to that of Jean-loup Gailly's and Mark Adler's excellent
Chris@4: zlib library.
All externally visible symbols have names beginning
Chris@4: BZ2_. This is new in version
Chris@4: 1.0. The intention is to minimise pollution of the namespaces of
Chris@4: library clients.
To use any part of the library, you need to
Chris@4: #include <bzlib.h>
Chris@4: into your sources.
This interface provides services for compressing and
Chris@4: decompressing data in memory. There's no provision for dealing
Chris@4: with files, streams or any other I/O mechanisms, just straight
Chris@4: memory-to-memory work. In fact, this part of the library can be
Chris@4: compiled without inclusion of
Chris@4: stdio.h, which may be helpful
Chris@4: for embedded applications.
The low-level part of the library has no global variables Chris@4: and is therefore thread-safe.
Chris@4:Six routines make up the low level interface:
Chris@4: BZ2_bzCompressInit,
Chris@4: BZ2_bzCompress, and
Chris@4: BZ2_bzCompressEnd for
Chris@4: compression, and a corresponding trio
Chris@4: BZ2_bzDecompressInit,
Chris@4: BZ2_bzDecompress and
Chris@4: BZ2_bzDecompressEnd for
Chris@4: decompression. The *Init
Chris@4: functions allocate memory for compression/decompression and do
Chris@4: other initialisations, whilst the
Chris@4: *End functions close down
Chris@4: operations and release memory.
The real work is done by
Chris@4: BZ2_bzCompress and
Chris@4: BZ2_bzDecompress. These
Chris@4: compress and decompress data from a user-supplied input buffer to
Chris@4: a user-supplied output buffer. These buffers can be any size;
Chris@4: arbitrary quantities of data are handled by making repeated calls
Chris@4: to these functions. This is a flexible mechanism allowing a
Chris@4: consumer-pull style of activity, or producer-push, or a mixture
Chris@4: of both.
This interface provides some handy wrappers around the
Chris@4: low-level interface to facilitate reading and writing
Chris@4: bzip2 format files
Chris@4: (.bz2 files). The routines
Chris@4: provide hooks to facilitate reading files in which the
Chris@4: bzip2 data stream is embedded
Chris@4: within some larger-scale file structure, or where there are
Chris@4: multiple bzip2 data streams
Chris@4: concatenated end-to-end.
For reading files,
Chris@4: BZ2_bzReadOpen,
Chris@4: BZ2_bzRead,
Chris@4: BZ2_bzReadClose and
Chris@4: BZ2_bzReadGetUnused are
Chris@4: supplied. For writing files,
Chris@4: BZ2_bzWriteOpen,
Chris@4: BZ2_bzWrite and
Chris@4: BZ2_bzWriteFinish are
Chris@4: available.
As with the low-level library, no global variables are used
Chris@4: so the library is per se thread-safe. However, if I/O errors
Chris@4: occur whilst reading or writing the underlying compressed files,
Chris@4: you may have to consult errno to
Chris@4: determine the cause of the error. In that case, you'd need a C
Chris@4: library which correctly supports
Chris@4: errno in a multithreaded
Chris@4: environment.
To make the library a little simpler and more portable,
Chris@4: BZ2_bzReadOpen and
Chris@4: BZ2_bzWriteOpen require you to
Chris@4: pass them file handles (FILE*s)
Chris@4: which have previously been opened for reading or writing
Chris@4: respectively. That avoids portability problems associated with
Chris@4: file operations and file attributes, whilst not being much of an
Chris@4: imposition on the programmer.
For very simple needs,
Chris@4: BZ2_bzBuffToBuffCompress and
Chris@4: BZ2_bzBuffToBuffDecompress are
Chris@4: provided. These compress data in memory from one buffer to
Chris@4: another buffer in a single function call. You should assess
Chris@4: whether these functions fulfill your memory-to-memory
Chris@4: compression/decompression requirements before investing effort in
Chris@4: understanding the more general but more complex low-level
Chris@4: interface.
Yoshioka Tsuneo
Chris@4: (tsuneo@rr.iij4u.or.jp) has
Chris@4: contributed some functions to give better
Chris@4: zlib compatibility. These
Chris@4: functions are BZ2_bzopen,
Chris@4: BZ2_bzread,
Chris@4: BZ2_bzwrite,
Chris@4: BZ2_bzflush,
Chris@4: BZ2_bzclose,
Chris@4: BZ2_bzerror and
Chris@4: BZ2_bzlibVersion. You may find
Chris@4: these functions more convenient for simple file reading and
Chris@4: writing, than those in the high-level interface. These functions
Chris@4: are not (yet) officially part of the library, and are minimally
Chris@4: documented here. If they break, you get to keep all the pieces.
Chris@4: I hope to document them properly when time permits.
Yoshioka also contributed modifications to allow the Chris@4: library to be built as a Windows DLL.
Chris@4:The library is designed to recover cleanly in all Chris@4: situations, including the worst-case situation of decompressing Chris@4: random data. I'm not 100% sure that it can always do this, so Chris@4: you might want to add a signal handler to catch segmentation Chris@4: violations during decompression if you are feeling especially Chris@4: paranoid. I would be interested in hearing more about the Chris@4: robustness of the library to corrupted compressed data.
Chris@4:Version 1.0.3 more robust in this respect than any Chris@4: previous version. Investigations with Valgrind (a tool for detecting Chris@4: problems with memory management) indicate Chris@4: that, at least for the few files I tested, all single-bit errors Chris@4: in the decompressed data are caught properly, with no Chris@4: segmentation faults, no uses of uninitialised data, no out of Chris@4: range reads or writes, and no infinite looping in the decompressor. Chris@4: So it's certainly pretty robust, although Chris@4: I wouldn't claim it to be totally bombproof.
Chris@4:The file bzlib.h contains
Chris@4: all definitions needed to use the library. In particular, you
Chris@4: should definitely not include
Chris@4: bzlib_private.h.
In bzlib.h, the various
Chris@4: return values are defined. The following list is not intended as
Chris@4: an exhaustive description of the circumstances in which a given
Chris@4: value may be returned -- those descriptions are given later.
Chris@4: Rather, it is intended to convey the rough meaning of each return
Chris@4: value. The first five actions are normal and not intended to
Chris@4: denote an error situation.
BZ_OKThe requested action was completed Chris@4: successfully.
BZ_RUN_OK, BZ_FLUSH_OK,
Chris@4: BZ_FINISH_OKIn
Chris@4: BZ2_bzCompress, the requested
Chris@4: flush/finish/nothing-special action was completed
Chris@4: successfully.
BZ_STREAM_ENDCompression of data was completed, or the Chris@4: logical stream end was detected during Chris@4: decompression.
The following return values indicate an error of some Chris@4: kind.
Chris@4:BZ_CONFIG_ERRORIndicates that the library has been improperly
Chris@4: compiled on your platform -- a major configuration error.
Chris@4: Specifically, it means that
Chris@4: sizeof(char),
Chris@4: sizeof(short) and
Chris@4: sizeof(int) are not 1, 2 and
Chris@4: 4 respectively, as they should be. Note that the library
Chris@4: should still work properly on 64-bit platforms which follow
Chris@4: the LP64 programming model -- that is, where
Chris@4: sizeof(long) and
Chris@4: sizeof(void*) are 8. Under
Chris@4: LP64, sizeof(int) is still 4,
Chris@4: so libbzip2, which doesn't
Chris@4: use the long type, is
Chris@4: OK.
BZ_SEQUENCE_ERRORWhen using the library, it is important to call
Chris@4: the functions in the correct sequence and with data structures
Chris@4: (buffers etc) in the correct states.
Chris@4: libbzip2 checks as much as it
Chris@4: can to ensure this is happening, and returns
Chris@4: BZ_SEQUENCE_ERROR if not.
Chris@4: Code which complies precisely with the function semantics, as
Chris@4: detailed below, should never receive this value; such an event
Chris@4: denotes buggy code which you should
Chris@4: investigate.
BZ_PARAM_ERRORReturned when a parameter to a function call is
Chris@4: out of range or otherwise manifestly incorrect. As with
Chris@4: BZ_SEQUENCE_ERROR, this
Chris@4: denotes a bug in the client code. The distinction between
Chris@4: BZ_PARAM_ERROR and
Chris@4: BZ_SEQUENCE_ERROR is a bit
Chris@4: hazy, but still worth making.
BZ_MEM_ERRORReturned when a request to allocate memory
Chris@4: failed. Note that the quantity of memory needed to decompress
Chris@4: a stream cannot be determined until the stream's header has
Chris@4: been read. So
Chris@4: BZ2_bzDecompress and
Chris@4: BZ2_bzRead may return
Chris@4: BZ_MEM_ERROR even though some
Chris@4: of the compressed data has been read. The same is not true
Chris@4: for compression; once
Chris@4: BZ2_bzCompressInit or
Chris@4: BZ2_bzWriteOpen have
Chris@4: successfully completed,
Chris@4: BZ_MEM_ERROR cannot
Chris@4: occur.
BZ_DATA_ERRORReturned when a data integrity error is Chris@4: detected during decompression. Most importantly, this means Chris@4: when stored and computed CRCs for the data do not match. This Chris@4: value is also returned upon detection of any other anomaly in Chris@4: the compressed data.
BZ_DATA_ERROR_MAGICAs a special case of
Chris@4: BZ_DATA_ERROR, it is
Chris@4: sometimes useful to know when the compressed stream does not
Chris@4: start with the correct magic bytes ('B' 'Z'
Chris@4: 'h').
BZ_IO_ERRORReturned by
Chris@4: BZ2_bzRead and
Chris@4: BZ2_bzWrite when there is an
Chris@4: error reading or writing in the compressed file, and by
Chris@4: BZ2_bzReadOpen and
Chris@4: BZ2_bzWriteOpen for attempts
Chris@4: to use a file for which the error indicator (viz,
Chris@4: ferror(f)) is set. On
Chris@4: receipt of BZ_IO_ERROR, the
Chris@4: caller should consult errno
Chris@4: and/or perror to acquire
Chris@4: operating-system specific information about the
Chris@4: problem.
BZ_UNEXPECTED_EOFReturned by
Chris@4: BZ2_bzRead when the
Chris@4: compressed file finishes before the logical end of stream is
Chris@4: detected.
BZ_OUTBUFF_FULLReturned by
Chris@4: BZ2_bzBuffToBuffCompress and
Chris@4: BZ2_bzBuffToBuffDecompress to
Chris@4: indicate that the output data will not fit into the output
Chris@4: buffer provided.
typedef struct {
Chris@4: char *next_in;
Chris@4: unsigned int avail_in;
Chris@4: unsigned int total_in_lo32;
Chris@4: unsigned int total_in_hi32;
Chris@4:
Chris@4: char *next_out;
Chris@4: unsigned int avail_out;
Chris@4: unsigned int total_out_lo32;
Chris@4: unsigned int total_out_hi32;
Chris@4:
Chris@4: void *state;
Chris@4:
Chris@4: void *(*bzalloc)(void *,int,int);
Chris@4: void (*bzfree)(void *,void *);
Chris@4: void *opaque;
Chris@4: } bz_stream;
Chris@4:
Chris@4: int BZ2_bzCompressInit ( bz_stream *strm,
Chris@4: int blockSize100k,
Chris@4: int verbosity,
Chris@4: int workFactor );
Chris@4: Prepares for compression. The
Chris@4: bz_stream structure holds all
Chris@4: data pertaining to the compression activity. A
Chris@4: bz_stream structure should be
Chris@4: allocated and initialised prior to the call. The fields of
Chris@4: bz_stream comprise the entirety
Chris@4: of the user-visible data. state
Chris@4: is a pointer to the private data structures required for
Chris@4: compression.
Custom memory allocators are supported, via fields
Chris@4: bzalloc,
Chris@4: bzfree, and
Chris@4: opaque. The value
Chris@4: opaque is passed to as the first
Chris@4: argument to all calls to bzalloc
Chris@4: and bzfree, but is otherwise
Chris@4: ignored by the library. The call bzalloc (
Chris@4: opaque, n, m ) is expected to return a pointer
Chris@4: p to n *
Chris@4: m bytes of memory, and bzfree (
Chris@4: opaque, p ) should free that memory.
If you don't want to use a custom memory allocator, set
Chris@4: bzalloc,
Chris@4: bzfree and
Chris@4: opaque to
Chris@4: NULL, and the library will then
Chris@4: use the standard malloc /
Chris@4: free routines.
Before calling
Chris@4: BZ2_bzCompressInit, fields
Chris@4: bzalloc,
Chris@4: bzfree and
Chris@4: opaque should be filled
Chris@4: appropriately, as just described. Upon return, the internal
Chris@4: state will have been allocated and initialised, and
Chris@4: total_in_lo32,
Chris@4: total_in_hi32,
Chris@4: total_out_lo32 and
Chris@4: total_out_hi32 will have been
Chris@4: set to zero. These four fields are used by the library to inform
Chris@4: the caller of the total amount of data passed into and out of the
Chris@4: library, respectively. You should not try to change them. As of
Chris@4: version 1.0, 64-bit counts are maintained, even on 32-bit
Chris@4: platforms, using the _hi32
Chris@4: fields to store the upper 32 bits of the count. So, for example,
Chris@4: the total amount of data in is (total_in_hi32
Chris@4: << 32) + total_in_lo32.
Parameter blockSize100k
Chris@4: specifies the block size to be used for compression. It should
Chris@4: be a value between 1 and 9 inclusive, and the actual block size
Chris@4: used is 100000 x this figure. 9 gives the best compression but
Chris@4: takes most memory.
Parameter verbosity should
Chris@4: be set to a number between 0 and 4 inclusive. 0 is silent, and
Chris@4: greater numbers give increasingly verbose monitoring/debugging
Chris@4: output. If the library has been compiled with
Chris@4: -DBZ_NO_STDIO, no such output
Chris@4: will appear for any verbosity setting.
Parameter workFactor
Chris@4: controls how the compression phase behaves when presented with
Chris@4: worst case, highly repetitive, input data. If compression runs
Chris@4: into difficulties caused by repetitive data, the library switches
Chris@4: from the standard sorting algorithm to a fallback algorithm. The
Chris@4: fallback is slower than the standard algorithm by perhaps a
Chris@4: factor of three, but always behaves reasonably, no matter how bad
Chris@4: the input.
Lower values of workFactor
Chris@4: reduce the amount of effort the standard algorithm will expend
Chris@4: before resorting to the fallback. You should set this parameter
Chris@4: carefully; too low, and many inputs will be handled by the
Chris@4: fallback algorithm and so compress rather slowly, too high, and
Chris@4: your average-to-worst case compression times can become very
Chris@4: large. The default value of 30 gives reasonable behaviour over a
Chris@4: wide range of circumstances.
Allowable values range from 0 to 250 inclusive. 0 is a Chris@4: special case, equivalent to using the default value of 30.
Chris@4:Note that the compressed output generated is the same Chris@4: regardless of whether or not the fallback algorithm is Chris@4: used.
Chris@4:Be aware also that this parameter may disappear entirely in Chris@4: future versions of the library. In principle it should be Chris@4: possible to devise a good way to automatically choose which Chris@4: algorithm to use. Such a mechanism would render the parameter Chris@4: obsolete.
Chris@4:Possible return values:
Chris@4:BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if strm is NULL Chris@4: or blockSize < 1 or blockSize > 9 Chris@4: or verbosity < 0 or verbosity > 4 Chris@4: or workFactor < 0 or workFactor > 250 Chris@4: BZ_MEM_ERROR Chris@4: if not enough memory is available Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:BZ2_bzCompress Chris@4: if BZ_OK is returned Chris@4: no specific action needed in case of errorChris@4:
int BZ2_bzCompress ( bz_stream *strm, int action );Chris@4:
Provides more input and/or output buffer space for the
Chris@4: library. The caller maintains input and output buffers, and
Chris@4: calls BZ2_bzCompress to transfer
Chris@4: data between them.
Before each call to
Chris@4: BZ2_bzCompress,
Chris@4: next_in should point at the data
Chris@4: to be compressed, and avail_in
Chris@4: should indicate how many bytes the library may read.
Chris@4: BZ2_bzCompress updates
Chris@4: next_in,
Chris@4: avail_in and
Chris@4: total_in to reflect the number
Chris@4: of bytes it has read.
Similarly, next_out should
Chris@4: point to a buffer in which the compressed data is to be placed,
Chris@4: with avail_out indicating how
Chris@4: much output space is available.
Chris@4: BZ2_bzCompress updates
Chris@4: next_out,
Chris@4: avail_out and
Chris@4: total_out to reflect the number
Chris@4: of bytes output.
You may provide and remove as little or as much data as you
Chris@4: like on each call of
Chris@4: BZ2_bzCompress. In the limit,
Chris@4: it is acceptable to supply and remove data one byte at a time,
Chris@4: although this would be terribly inefficient. You should always
Chris@4: ensure that at least one byte of output space is available at
Chris@4: each call.
A second purpose of
Chris@4: BZ2_bzCompress is to request a
Chris@4: change of mode of the compressed stream.
Conceptually, a compressed stream can be in one of four
Chris@4: states: IDLE, RUNNING, FLUSHING and FINISHING. Before
Chris@4: initialisation
Chris@4: (BZ2_bzCompressInit) and after
Chris@4: termination (BZ2_bzCompressEnd),
Chris@4: a stream is regarded as IDLE.
Upon initialisation
Chris@4: (BZ2_bzCompressInit), the stream
Chris@4: is placed in the RUNNING state. Subsequent calls to
Chris@4: BZ2_bzCompress should pass
Chris@4: BZ_RUN as the requested action;
Chris@4: other actions are illegal and will result in
Chris@4: BZ_SEQUENCE_ERROR.
At some point, the calling program will have provided all
Chris@4: the input data it wants to. It will then want to finish up -- in
Chris@4: effect, asking the library to process any data it might have
Chris@4: buffered internally. In this state,
Chris@4: BZ2_bzCompress will no longer
Chris@4: attempt to read data from
Chris@4: next_in, but it will want to
Chris@4: write data to next_out. Because
Chris@4: the output buffer supplied by the user can be arbitrarily small,
Chris@4: the finishing-up operation cannot necessarily be done with a
Chris@4: single call of
Chris@4: BZ2_bzCompress.
Instead, the calling program passes
Chris@4: BZ_FINISH as an action to
Chris@4: BZ2_bzCompress. This changes
Chris@4: the stream's state to FINISHING. Any remaining input (ie,
Chris@4: next_in[0 .. avail_in-1]) is
Chris@4: compressed and transferred to the output buffer. To do this,
Chris@4: BZ2_bzCompress must be called
Chris@4: repeatedly until all the output has been consumed. At that
Chris@4: point, BZ2_bzCompress returns
Chris@4: BZ_STREAM_END, and the stream's
Chris@4: state is set back to IDLE.
Chris@4: BZ2_bzCompressEnd should then be
Chris@4: called.
Just to make sure the calling program does not cheat, the
Chris@4: library makes a note of avail_in
Chris@4: at the time of the first call to
Chris@4: BZ2_bzCompress which has
Chris@4: BZ_FINISH as an action (ie, at
Chris@4: the time the program has announced its intention to not supply
Chris@4: any more input). By comparing this value with that of
Chris@4: avail_in over subsequent calls
Chris@4: to BZ2_bzCompress, the library
Chris@4: can detect any attempts to slip in more data to compress. Any
Chris@4: calls for which this is detected will return
Chris@4: BZ_SEQUENCE_ERROR. This
Chris@4: indicates a programming mistake which should be corrected.
Instead of asking to finish, the calling program may ask
Chris@4: BZ2_bzCompress to take all the
Chris@4: remaining input, compress it and terminate the current
Chris@4: (Burrows-Wheeler) compression block. This could be useful for
Chris@4: error control purposes. The mechanism is analogous to that for
Chris@4: finishing: call BZ2_bzCompress
Chris@4: with an action of BZ_FLUSH,
Chris@4: remove output data, and persist with the
Chris@4: BZ_FLUSH action until the value
Chris@4: BZ_RUN is returned. As with
Chris@4: finishing, BZ2_bzCompress
Chris@4: detects any attempt to provide more input data once the flush has
Chris@4: begun.
Once the flush is complete, the stream returns to the Chris@4: normal RUNNING state.
Chris@4:This all sounds pretty complex, but isn't really. Here's a
Chris@4: table which shows which actions are allowable in each state, what
Chris@4: action will be taken, what the next state is, and what the
Chris@4: non-error return values are. Note that you can't explicitly ask
Chris@4: what state the stream is in, but nor do you need to -- it can be
Chris@4: inferred from the values returned by
Chris@4: BZ2_bzCompress.
IDLE/any Chris@4: Illegal. IDLE state only exists after BZ2_bzCompressEnd or Chris@4: before BZ2_bzCompressInit. Chris@4: Return value = BZ_SEQUENCE_ERROR Chris@4: Chris@4: RUNNING/BZ_RUN Chris@4: Compress from next_in to next_out as much as possible. Chris@4: Next state = RUNNING Chris@4: Return value = BZ_RUN_OK Chris@4: Chris@4: RUNNING/BZ_FLUSH Chris@4: Remember current value of next_in. Compress from next_in Chris@4: to next_out as much as possible, but do not accept any more input. Chris@4: Next state = FLUSHING Chris@4: Return value = BZ_FLUSH_OK Chris@4: Chris@4: RUNNING/BZ_FINISH Chris@4: Remember current value of next_in. Compress from next_in Chris@4: to next_out as much as possible, but do not accept any more input. Chris@4: Next state = FINISHING Chris@4: Return value = BZ_FINISH_OK Chris@4: Chris@4: FLUSHING/BZ_FLUSH Chris@4: Compress from next_in to next_out as much as possible, Chris@4: but do not accept any more input. Chris@4: If all the existing input has been used up and all compressed Chris@4: output has been removed Chris@4: Next state = RUNNING; Return value = BZ_RUN_OK Chris@4: else Chris@4: Next state = FLUSHING; Return value = BZ_FLUSH_OK Chris@4: Chris@4: FLUSHING/other Chris@4: Illegal. Chris@4: Return value = BZ_SEQUENCE_ERROR Chris@4: Chris@4: FINISHING/BZ_FINISH Chris@4: Compress from next_in to next_out as much as possible, Chris@4: but to not accept any more input. Chris@4: If all the existing input has been used up and all compressed Chris@4: output has been removed Chris@4: Next state = IDLE; Return value = BZ_STREAM_END Chris@4: else Chris@4: Next state = FINISHING; Return value = BZ_FINISH_OK Chris@4: Chris@4: FINISHING/other Chris@4: Illegal. Chris@4: Return value = BZ_SEQUENCE_ERRORChris@4:
That still looks complicated? Well, fair enough. The Chris@4: usual sequence of calls for compressing a load of data is:
Chris@4:Get started with
Chris@4: BZ2_bzCompressInit.
Shovel data in and shlurp out its compressed form
Chris@4: using zero or more calls of
Chris@4: BZ2_bzCompress with action =
Chris@4: BZ_RUN.
Finish up. Repeatedly call
Chris@4: BZ2_bzCompress with action =
Chris@4: BZ_FINISH, copying out the
Chris@4: compressed output, until
Chris@4: BZ_STREAM_END is
Chris@4: returned.
Close up and go home. Call
Chris@4: BZ2_bzCompressEnd.
If the data you want to compress fits into your input
Chris@4: buffer all at once, you can skip the calls of
Chris@4: BZ2_bzCompress ( ..., BZ_RUN )
Chris@4: and just do the BZ2_bzCompress ( ..., BZ_FINISH
Chris@4: ) calls.
All required memory is allocated by
Chris@4: BZ2_bzCompressInit. The
Chris@4: compression library can accept any data at all (obviously). So
Chris@4: you shouldn't get any error return values from the
Chris@4: BZ2_bzCompress calls. If you
Chris@4: do, they will be
Chris@4: BZ_SEQUENCE_ERROR, and indicate
Chris@4: a bug in your programming.
Trivial other possible return values:
Chris@4:BZ_PARAM_ERROR Chris@4: if strm is NULL, or strm->s is NULLChris@4:
int BZ2_bzCompressEnd ( bz_stream *strm );Chris@4:
Releases all memory associated with a compression Chris@4: stream.
Chris@4:Possible return values:
Chris@4:BZ_PARAM_ERROR if strm is NULL or strm->s is NULL Chris@4: BZ_OK otherwiseChris@4:
int BZ2_bzDecompressInit ( bz_stream *strm, int verbosity, int small );Chris@4:
Prepares for decompression. As with
Chris@4: BZ2_bzCompressInit, a
Chris@4: bz_stream record should be
Chris@4: allocated and initialised before the call. Fields
Chris@4: bzalloc,
Chris@4: bzfree and
Chris@4: opaque should be set if a custom
Chris@4: memory allocator is required, or made
Chris@4: NULL for the normal
Chris@4: malloc /
Chris@4: free routines. Upon return, the
Chris@4: internal state will have been initialised, and
Chris@4: total_in and
Chris@4: total_out will be zero.
For the meaning of parameter
Chris@4: verbosity, see
Chris@4: BZ2_bzCompressInit.
If small is nonzero, the
Chris@4: library will use an alternative decompression algorithm which
Chris@4: uses less memory but at the cost of decompressing more slowly
Chris@4: (roughly speaking, half the speed, but the maximum memory
Chris@4: requirement drops to around 2300k). See How to use bzip2
Chris@4: for more information on memory management.
Note that the amount of memory needed to decompress a
Chris@4: stream cannot be determined until the stream's header has been
Chris@4: read, so even if
Chris@4: BZ2_bzDecompressInit succeeds, a
Chris@4: subsequent BZ2_bzDecompress
Chris@4: could fail with
Chris@4: BZ_MEM_ERROR.
Possible return values:
Chris@4:BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if ( small != 0 && small != 1 ) Chris@4: or (verbosity <; 0 || verbosity > 4) Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory is availableChris@4:
Allowable next actions:
Chris@4:BZ2_bzDecompress Chris@4: if BZ_OK was returned Chris@4: no specific action required in case of errorChris@4:
int BZ2_bzDecompress ( bz_stream *strm );Chris@4:
Provides more input and/out output buffer space for the
Chris@4: library. The caller maintains input and output buffers, and uses
Chris@4: BZ2_bzDecompress to transfer
Chris@4: data between them.
Before each call to
Chris@4: BZ2_bzDecompress,
Chris@4: next_in should point at the
Chris@4: compressed data, and avail_in
Chris@4: should indicate how many bytes the library may read.
Chris@4: BZ2_bzDecompress updates
Chris@4: next_in,
Chris@4: avail_in and
Chris@4: total_in to reflect the number
Chris@4: of bytes it has read.
Similarly, next_out should
Chris@4: point to a buffer in which the uncompressed output is to be
Chris@4: placed, with avail_out
Chris@4: indicating how much output space is available.
Chris@4: BZ2_bzCompress updates
Chris@4: next_out,
Chris@4: avail_out and
Chris@4: total_out to reflect the number
Chris@4: of bytes output.
You may provide and remove as little or as much data as you
Chris@4: like on each call of
Chris@4: BZ2_bzDecompress. In the limit,
Chris@4: it is acceptable to supply and remove data one byte at a time,
Chris@4: although this would be terribly inefficient. You should always
Chris@4: ensure that at least one byte of output space is available at
Chris@4: each call.
Use of BZ2_bzDecompress is
Chris@4: simpler than
Chris@4: BZ2_bzCompress.
You should provide input and remove output as described
Chris@4: above, and repeatedly call
Chris@4: BZ2_bzDecompress until
Chris@4: BZ_STREAM_END is returned.
Chris@4: Appearance of BZ_STREAM_END
Chris@4: denotes that BZ2_bzDecompress
Chris@4: has detected the logical end of the compressed stream.
Chris@4: BZ2_bzDecompress will not
Chris@4: produce BZ_STREAM_END until all
Chris@4: output data has been placed into the output buffer, so once
Chris@4: BZ_STREAM_END appears, you are
Chris@4: guaranteed to have available all the decompressed output, and
Chris@4: BZ2_bzDecompressEnd can safely
Chris@4: be called.
If case of an error return value, you should call
Chris@4: BZ2_bzDecompressEnd to clean up
Chris@4: and release memory.
Possible return values:
Chris@4:BZ_PARAM_ERROR Chris@4: if strm is NULL or strm->s is NULL Chris@4: or strm->avail_out < 1 Chris@4: BZ_DATA_ERROR Chris@4: if a data integrity error is detected in the compressed stream Chris@4: BZ_DATA_ERROR_MAGIC Chris@4: if the compressed stream doesn't begin with the right magic bytes Chris@4: BZ_MEM_ERROR Chris@4: if there wasn't enough memory available Chris@4: BZ_STREAM_END Chris@4: if the logical end of the data stream was detected and all Chris@4: output in has been consumed, eg s-->avail_out > 0 Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:BZ2_bzDecompress Chris@4: if BZ_OK was returned Chris@4: BZ2_bzDecompressEnd Chris@4: otherwiseChris@4:
int BZ2_bzDecompressEnd ( bz_stream *strm );Chris@4:
Releases all memory associated with a decompression Chris@4: stream.
Chris@4:Possible return values:
Chris@4:BZ_PARAM_ERROR Chris@4: if strm is NULL or strm->s is NULL Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:None.Chris@4:
This interface provides functions for reading and writing
Chris@4: bzip2 format files. First, some
Chris@4: general points.
All of the functions take an
Chris@4: int* first argument,
Chris@4: bzerror. After each call,
Chris@4: bzerror should be consulted
Chris@4: first to determine the outcome of the call. If
Chris@4: bzerror is
Chris@4: BZ_OK, the call completed
Chris@4: successfully, and only then should the return value of the
Chris@4: function (if any) be consulted. If
Chris@4: bzerror is
Chris@4: BZ_IO_ERROR, there was an
Chris@4: error reading/writing the underlying compressed file, and you
Chris@4: should then consult errno /
Chris@4: perror to determine the cause
Chris@4: of the difficulty. bzerror
Chris@4: may also be set to various other values; precise details are
Chris@4: given on a per-function basis below.
If bzerror indicates
Chris@4: an error (ie, anything except
Chris@4: BZ_OK and
Chris@4: BZ_STREAM_END), you should
Chris@4: immediately call
Chris@4: BZ2_bzReadClose (or
Chris@4: BZ2_bzWriteClose, depending on
Chris@4: whether you are attempting to read or to write) to free up all
Chris@4: resources associated with the stream. Once an error has been
Chris@4: indicated, behaviour of all calls except
Chris@4: BZ2_bzReadClose
Chris@4: (BZ2_bzWriteClose) is
Chris@4: undefined. The implication is that (1)
Chris@4: bzerror should be checked
Chris@4: after each call, and (2) if
Chris@4: bzerror indicates an error,
Chris@4: BZ2_bzReadClose
Chris@4: (BZ2_bzWriteClose) should then
Chris@4: be called to clean up.
The FILE* arguments
Chris@4: passed to BZ2_bzReadOpen /
Chris@4: BZ2_bzWriteOpen should be set
Chris@4: to binary mode. Most Unix systems will do this by default, but
Chris@4: other platforms, including Windows and Mac, will not. If you
Chris@4: omit this, you may encounter problems when moving code to new
Chris@4: platforms.
Memory allocation requests are handled by
Chris@4: malloc /
Chris@4: free. At present there is no
Chris@4: facility for user-defined memory allocators in the file I/O
Chris@4: functions (could easily be added, though).
typedef void BZFILE; Chris@4: Chris@4: BZFILE *BZ2_bzReadOpen( int *bzerror, FILE *f, Chris@4: int verbosity, int small, Chris@4: void *unused, int nUnused );Chris@4:
Prepare to read compressed data from file handle
Chris@4: f.
Chris@4: f should refer to a file which
Chris@4: has been opened for reading, and for which the error indicator
Chris@4: (ferror(f))is not set. If
Chris@4: small is 1, the library will try
Chris@4: to decompress using less memory, at the expense of speed.
For reasons explained below,
Chris@4: BZ2_bzRead will decompress the
Chris@4: nUnused bytes starting at
Chris@4: unused, before starting to read
Chris@4: from the file f. At most
Chris@4: BZ_MAX_UNUSED bytes may be
Chris@4: supplied like this. If this facility is not required, you should
Chris@4: pass NULL and
Chris@4: 0 for
Chris@4: unused and
Chris@4: nUnused respectively.
For the meaning of parameters
Chris@4: small and
Chris@4: verbosity, see
Chris@4: BZ2_bzDecompressInit.
The amount of memory needed to decompress a file cannot be
Chris@4: determined until the file's header has been read. So it is
Chris@4: possible that BZ2_bzReadOpen
Chris@4: returns BZ_OK but a subsequent
Chris@4: call of BZ2_bzRead will return
Chris@4: BZ_MEM_ERROR.
Possible assignments to
Chris@4: bzerror:
BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if f is NULL Chris@4: or small is neither 0 nor 1 Chris@4: or ( unused == NULL && nUnused != 0 ) Chris@4: or ( unused != NULL && !(0 <= nUnused <= BZ_MAX_UNUSED) ) Chris@4: BZ_IO_ERROR Chris@4: if ferror(f) is nonzero Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory is available Chris@4: BZ_OK Chris@4: otherwise.Chris@4:
Possible return values:
Chris@4:Pointer to an abstract BZFILE Chris@4: if bzerror is BZ_OK Chris@4: NULL Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:BZ2_bzRead Chris@4: if bzerror is BZ_OK Chris@4: BZ2_bzClose Chris@4: otherwiseChris@4:
int BZ2_bzRead ( int *bzerror, BZFILE *b, void *buf, int len );Chris@4:
Reads up to len
Chris@4: (uncompressed) bytes from the compressed file
Chris@4: b into the buffer
Chris@4: buf. If the read was
Chris@4: successful, bzerror is set to
Chris@4: BZ_OK and the number of bytes
Chris@4: read is returned. If the logical end-of-stream was detected,
Chris@4: bzerror will be set to
Chris@4: BZ_STREAM_END, and the number of
Chris@4: bytes read is returned. All other
Chris@4: bzerror values denote an
Chris@4: error.
BZ2_bzRead will supply
Chris@4: len bytes, unless the logical
Chris@4: stream end is detected or an error occurs. Because of this, it
Chris@4: is possible to detect the stream end by observing when the number
Chris@4: of bytes returned is less than the number requested.
Chris@4: Nevertheless, this is regarded as inadvisable; you should instead
Chris@4: check bzerror after every call
Chris@4: and watch out for
Chris@4: BZ_STREAM_END.
Internally, BZ2_bzRead
Chris@4: copies data from the compressed file in chunks of size
Chris@4: BZ_MAX_UNUSED bytes before
Chris@4: decompressing it. If the file contains more bytes than strictly
Chris@4: needed to reach the logical end-of-stream,
Chris@4: BZ2_bzRead will almost certainly
Chris@4: read some of the trailing data before signalling
Chris@4: BZ_SEQUENCE_END. To collect the
Chris@4: read but unused data once
Chris@4: BZ_SEQUENCE_END has appeared,
Chris@4: call BZ2_bzReadGetUnused
Chris@4: immediately before
Chris@4: BZ2_bzReadClose.
Possible assignments to
Chris@4: bzerror:
BZ_PARAM_ERROR Chris@4: if b is NULL or buf is NULL or len < 0 Chris@4: BZ_SEQUENCE_ERROR Chris@4: if b was opened with BZ2_bzWriteOpen Chris@4: BZ_IO_ERROR Chris@4: if there is an error reading from the compressed file Chris@4: BZ_UNEXPECTED_EOF Chris@4: if the compressed file ended before Chris@4: the logical end-of-stream was detected Chris@4: BZ_DATA_ERROR Chris@4: if a data integrity error was detected in the compressed stream Chris@4: BZ_DATA_ERROR_MAGIC Chris@4: if the stream does not begin with the requisite header bytes Chris@4: (ie, is not a bzip2 data file). This is really Chris@4: a special case of BZ_DATA_ERROR. Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory was available Chris@4: BZ_STREAM_END Chris@4: if the logical end of stream was detected. Chris@4: BZ_OK Chris@4: otherwise.Chris@4:
Possible return values:
Chris@4:number of bytes read Chris@4: if bzerror is BZ_OK or BZ_STREAM_END Chris@4: undefined Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:collect data from buf, then BZ2_bzRead or BZ2_bzReadClose Chris@4: if bzerror is BZ_OK Chris@4: collect data from buf, then BZ2_bzReadClose or BZ2_bzReadGetUnused Chris@4: if bzerror is BZ_SEQUENCE_END Chris@4: BZ2_bzReadClose Chris@4: otherwiseChris@4:
void BZ2_bzReadGetUnused( int* bzerror, BZFILE *b, Chris@4: void** unused, int* nUnused );Chris@4:
Returns data which was read from the compressed file but
Chris@4: was not needed to get to the logical end-of-stream.
Chris@4: *unused is set to the address of
Chris@4: the data, and *nUnused to the
Chris@4: number of bytes. *nUnused will
Chris@4: be set to a value between 0 and
Chris@4: BZ_MAX_UNUSED inclusive.
This function may only be called once
Chris@4: BZ2_bzRead has signalled
Chris@4: BZ_STREAM_END but before
Chris@4: BZ2_bzReadClose.
Possible assignments to
Chris@4: bzerror:
BZ_PARAM_ERROR Chris@4: if b is NULL Chris@4: or unused is NULL or nUnused is NULL Chris@4: BZ_SEQUENCE_ERROR Chris@4: if BZ_STREAM_END has not been signalled Chris@4: or if b was opened with BZ2_bzWriteOpen Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:BZ2_bzReadCloseChris@4:
void BZ2_bzReadClose ( int *bzerror, BZFILE *b );Chris@4:
Releases all memory pertaining to the compressed file
Chris@4: b.
Chris@4: BZ2_bzReadClose does not call
Chris@4: fclose on the underlying file
Chris@4: handle, so you should do that yourself if appropriate.
Chris@4: BZ2_bzReadClose should be called
Chris@4: to clean up after all error situations.
Possible assignments to
Chris@4: bzerror:
BZ_SEQUENCE_ERROR Chris@4: if b was opened with BZ2_bzOpenWrite Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:noneChris@4:
BZFILE *BZ2_bzWriteOpen( int *bzerror, FILE *f, Chris@4: int blockSize100k, int verbosity, Chris@4: int workFactor );Chris@4:
Prepare to write compressed data to file handle
Chris@4: f.
Chris@4: f should refer to a file which
Chris@4: has been opened for writing, and for which the error indicator
Chris@4: (ferror(f))is not set.
For the meaning of parameters
Chris@4: blockSize100k,
Chris@4: verbosity and
Chris@4: workFactor, see
Chris@4: BZ2_bzCompressInit.
All required memory is allocated at this stage, so if the
Chris@4: call completes successfully,
Chris@4: BZ_MEM_ERROR cannot be signalled
Chris@4: by a subsequent call to
Chris@4: BZ2_bzWrite.
Possible assignments to
Chris@4: bzerror:
BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if f is NULL Chris@4: or blockSize100k < 1 or blockSize100k > 9 Chris@4: BZ_IO_ERROR Chris@4: if ferror(f) is nonzero Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory is available Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Possible return values:
Chris@4:Pointer to an abstract BZFILE Chris@4: if bzerror is BZ_OK Chris@4: NULL Chris@4: otherwiseChris@4:
Allowable next actions:
Chris@4:BZ2_bzWrite Chris@4: if bzerror is BZ_OK Chris@4: (you could go directly to BZ2_bzWriteClose, but this would be pretty pointless) Chris@4: BZ2_bzWriteClose Chris@4: otherwiseChris@4:
void BZ2_bzWrite ( int *bzerror, BZFILE *b, void *buf, int len );Chris@4:
Absorbs len bytes from the
Chris@4: buffer buf, eventually to be
Chris@4: compressed and written to the file.
Possible assignments to
Chris@4: bzerror:
BZ_PARAM_ERROR Chris@4: if b is NULL or buf is NULL or len < 0 Chris@4: BZ_SEQUENCE_ERROR Chris@4: if b was opened with BZ2_bzReadOpen Chris@4: BZ_IO_ERROR Chris@4: if there is an error writing the compressed file. Chris@4: BZ_OK Chris@4: otherwiseChris@4:
void BZ2_bzWriteClose( int *bzerror, BZFILE* f, Chris@4: int abandon, Chris@4: unsigned int* nbytes_in, Chris@4: unsigned int* nbytes_out ); Chris@4: Chris@4: void BZ2_bzWriteClose64( int *bzerror, BZFILE* f, Chris@4: int abandon, Chris@4: unsigned int* nbytes_in_lo32, Chris@4: unsigned int* nbytes_in_hi32, Chris@4: unsigned int* nbytes_out_lo32, Chris@4: unsigned int* nbytes_out_hi32 );Chris@4:
Compresses and flushes to the compressed file all data so
Chris@4: far supplied by BZ2_bzWrite.
Chris@4: The logical end-of-stream markers are also written, so subsequent
Chris@4: calls to BZ2_bzWrite are
Chris@4: illegal. All memory associated with the compressed file
Chris@4: b is released.
Chris@4: fflush is called on the
Chris@4: compressed file, but it is not
Chris@4: fclose'd.
If BZ2_bzWriteClose is
Chris@4: called to clean up after an error, the only action is to release
Chris@4: the memory. The library records the error codes issued by
Chris@4: previous calls, so this situation will be detected automatically.
Chris@4: There is no attempt to complete the compression operation, nor to
Chris@4: fflush the compressed file. You
Chris@4: can force this behaviour to happen even in the case of no error,
Chris@4: by passing a nonzero value to
Chris@4: abandon.
If nbytes_in is non-null,
Chris@4: *nbytes_in will be set to be the
Chris@4: total volume of uncompressed data handled. Similarly,
Chris@4: nbytes_out will be set to the
Chris@4: total volume of compressed data written. For compatibility with
Chris@4: older versions of the library,
Chris@4: BZ2_bzWriteClose only yields the
Chris@4: lower 32 bits of these counts. Use
Chris@4: BZ2_bzWriteClose64 if you want
Chris@4: the full 64 bit counts. These two functions are otherwise
Chris@4: absolutely identical.
Possible assignments to
Chris@4: bzerror:
BZ_SEQUENCE_ERROR Chris@4: if b was opened with BZ2_bzReadOpen Chris@4: BZ_IO_ERROR Chris@4: if there is an error writing the compressed file Chris@4: BZ_OK Chris@4: otherwiseChris@4:
The high-level library facilitates use of
Chris@4: bzip2 data streams which form
Chris@4: some part of a surrounding, larger data stream.
For writing, the library takes an open file handle,
Chris@4: writes compressed data to it,
Chris@4: fflushes it but does not
Chris@4: fclose it. The calling
Chris@4: application can write its own data before and after the
Chris@4: compressed data stream, using that same file handle.
Reading is more complex, and the facilities are not as
Chris@4: general as they could be since generality is hard to reconcile
Chris@4: with efficiency. BZ2_bzRead
Chris@4: reads from the compressed file in blocks of size
Chris@4: BZ_MAX_UNUSED bytes, and in
Chris@4: doing so probably will overshoot the logical end of compressed
Chris@4: stream. To recover this data once decompression has ended,
Chris@4: call BZ2_bzReadGetUnused after
Chris@4: the last call of BZ2_bzRead
Chris@4: (the one returning
Chris@4: BZ_STREAM_END) but before
Chris@4: calling
Chris@4: BZ2_bzReadClose.
This mechanism makes it easy to decompress multiple
Chris@4: bzip2 streams placed end-to-end.
Chris@4: As the end of one stream, when
Chris@4: BZ2_bzRead returns
Chris@4: BZ_STREAM_END, call
Chris@4: BZ2_bzReadGetUnused to collect
Chris@4: the unused data (copy it into your own buffer somewhere). That
Chris@4: data forms the start of the next compressed stream. To start
Chris@4: uncompressing that next stream, call
Chris@4: BZ2_bzReadOpen again, feeding in
Chris@4: the unused data via the unused /
Chris@4: nUnused parameters. Keep doing
Chris@4: this until BZ_STREAM_END return
Chris@4: coincides with the physical end of file
Chris@4: (feof(f)). In this situation
Chris@4: BZ2_bzReadGetUnused will of
Chris@4: course return no data.
This should give some feel for how the high-level interface Chris@4: can be used. If you require extra flexibility, you'll have to Chris@4: bite the bullet and get to grips with the low-level Chris@4: interface.
Chris@4:Here's how you'd write data to a compressed file:
Chris@4:FILE* f;
Chris@4: BZFILE* b;
Chris@4: int nBuf;
Chris@4: char buf[ /* whatever size you like */ ];
Chris@4: int bzerror;
Chris@4: int nWritten;
Chris@4:
Chris@4: f = fopen ( "myfile.bz2", "w" );
Chris@4: if ( !f ) {
Chris@4: /* handle error */
Chris@4: }
Chris@4: b = BZ2_bzWriteOpen( &bzerror, f, 9 );
Chris@4: if (bzerror != BZ_OK) {
Chris@4: BZ2_bzWriteClose ( b );
Chris@4: /* handle error */
Chris@4: }
Chris@4:
Chris@4: while ( /* condition */ ) {
Chris@4: /* get data to write into buf, and set nBuf appropriately */
Chris@4: nWritten = BZ2_bzWrite ( &bzerror, b, buf, nBuf );
Chris@4: if (bzerror == BZ_IO_ERROR) {
Chris@4: BZ2_bzWriteClose ( &bzerror, b );
Chris@4: /* handle error */
Chris@4: }
Chris@4: }
Chris@4:
Chris@4: BZ2_bzWriteClose( &bzerror, b );
Chris@4: if (bzerror == BZ_IO_ERROR) {
Chris@4: /* handle error */
Chris@4: }
Chris@4: And to read from a compressed file:
Chris@4:FILE* f;
Chris@4: BZFILE* b;
Chris@4: int nBuf;
Chris@4: char buf[ /* whatever size you like */ ];
Chris@4: int bzerror;
Chris@4: int nWritten;
Chris@4:
Chris@4: f = fopen ( "myfile.bz2", "r" );
Chris@4: if ( !f ) {
Chris@4: /* handle error */
Chris@4: }
Chris@4: b = BZ2_bzReadOpen ( &bzerror, f, 0, NULL, 0 );
Chris@4: if ( bzerror != BZ_OK ) {
Chris@4: BZ2_bzReadClose ( &bzerror, b );
Chris@4: /* handle error */
Chris@4: }
Chris@4:
Chris@4: bzerror = BZ_OK;
Chris@4: while ( bzerror == BZ_OK && /* arbitrary other conditions */) {
Chris@4: nBuf = BZ2_bzRead ( &bzerror, b, buf, /* size of buf */ );
Chris@4: if ( bzerror == BZ_OK ) {
Chris@4: /* do something with buf[0 .. nBuf-1] */
Chris@4: }
Chris@4: }
Chris@4: if ( bzerror != BZ_STREAM_END ) {
Chris@4: BZ2_bzReadClose ( &bzerror, b );
Chris@4: /* handle error */
Chris@4: } else {
Chris@4: BZ2_bzReadClose ( &bzerror, b );
Chris@4: }
Chris@4: int BZ2_bzBuffToBuffCompress( char* dest, Chris@4: unsigned int* destLen, Chris@4: char* source, Chris@4: unsigned int sourceLen, Chris@4: int blockSize100k, Chris@4: int verbosity, Chris@4: int workFactor );Chris@4:
Attempts to compress the data in source[0
Chris@4: .. sourceLen-1] into the destination buffer,
Chris@4: dest[0 .. *destLen-1]. If the
Chris@4: destination buffer is big enough,
Chris@4: *destLen is set to the size of
Chris@4: the compressed data, and BZ_OK
Chris@4: is returned. If the compressed data won't fit,
Chris@4: *destLen is unchanged, and
Chris@4: BZ_OUTBUFF_FULL is
Chris@4: returned.
Compression in this manner is a one-shot event, done with a
Chris@4: single call to this function. The resulting compressed data is a
Chris@4: complete bzip2 format data
Chris@4: stream. There is no mechanism for making additional calls to
Chris@4: provide extra input data. If you want that kind of mechanism,
Chris@4: use the low-level interface.
For the meaning of parameters
Chris@4: blockSize100k,
Chris@4: verbosity and
Chris@4: workFactor, see
Chris@4: BZ2_bzCompressInit.
To guarantee that the compressed data will fit in its Chris@4: buffer, allocate an output buffer of size 1% larger than the Chris@4: uncompressed data, plus six hundred extra bytes.
Chris@4:BZ2_bzBuffToBuffDecompress
Chris@4: will not write data at or beyond
Chris@4: dest[*destLen], even in case of
Chris@4: buffer overflow.
Possible return values:
Chris@4:BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if dest is NULL or destLen is NULL Chris@4: or blockSize100k < 1 or blockSize100k > 9 Chris@4: or verbosity < 0 or verbosity > 4 Chris@4: or workFactor < 0 or workFactor > 250 Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory is available Chris@4: BZ_OUTBUFF_FULL Chris@4: if the size of the compressed data exceeds *destLen Chris@4: BZ_OK Chris@4: otherwiseChris@4:
int BZ2_bzBuffToBuffDecompress( char* dest, Chris@4: unsigned int* destLen, Chris@4: char* source, Chris@4: unsigned int sourceLen, Chris@4: int small, Chris@4: int verbosity );Chris@4:
Attempts to decompress the data in source[0
Chris@4: .. sourceLen-1] into the destination buffer,
Chris@4: dest[0 .. *destLen-1]. If the
Chris@4: destination buffer is big enough,
Chris@4: *destLen is set to the size of
Chris@4: the uncompressed data, and BZ_OK
Chris@4: is returned. If the compressed data won't fit,
Chris@4: *destLen is unchanged, and
Chris@4: BZ_OUTBUFF_FULL is
Chris@4: returned.
source is assumed to hold
Chris@4: a complete bzip2 format data
Chris@4: stream.
Chris@4: BZ2_bzBuffToBuffDecompress tries
Chris@4: to decompress the entirety of the stream into the output
Chris@4: buffer.
For the meaning of parameters
Chris@4: small and
Chris@4: verbosity, see
Chris@4: BZ2_bzDecompressInit.
Because the compression ratio of the compressed data cannot Chris@4: be known in advance, there is no easy way to guarantee that the Chris@4: output buffer will be big enough. You may of course make Chris@4: arrangements in your code to record the size of the uncompressed Chris@4: data, but such a mechanism is beyond the scope of this Chris@4: library.
Chris@4:BZ2_bzBuffToBuffDecompress
Chris@4: will not write data at or beyond
Chris@4: dest[*destLen], even in case of
Chris@4: buffer overflow.
Possible return values:
Chris@4:BZ_CONFIG_ERROR Chris@4: if the library has been mis-compiled Chris@4: BZ_PARAM_ERROR Chris@4: if dest is NULL or destLen is NULL Chris@4: or small != 0 && small != 1 Chris@4: or verbosity < 0 or verbosity > 4 Chris@4: BZ_MEM_ERROR Chris@4: if insufficient memory is available Chris@4: BZ_OUTBUFF_FULL Chris@4: if the size of the compressed data exceeds *destLen Chris@4: BZ_DATA_ERROR Chris@4: if a data integrity error was detected in the compressed data Chris@4: BZ_DATA_ERROR_MAGIC Chris@4: if the compressed data doesn't begin with the right magic bytes Chris@4: BZ_UNEXPECTED_EOF Chris@4: if the compressed data ends unexpectedly Chris@4: BZ_OK Chris@4: otherwiseChris@4:
Yoshioka Tsuneo has contributed some functions to give
Chris@4: better zlib compatibility.
Chris@4: These functions are BZ2_bzopen,
Chris@4: BZ2_bzread,
Chris@4: BZ2_bzwrite,
Chris@4: BZ2_bzflush,
Chris@4: BZ2_bzclose,
Chris@4: BZ2_bzerror and
Chris@4: BZ2_bzlibVersion. These
Chris@4: functions are not (yet) officially part of the library. If they
Chris@4: break, you get to keep all the pieces. Nevertheless, I think
Chris@4: they work ok.
typedef void BZFILE; Chris@4: Chris@4: const char * BZ2_bzlibVersion ( void );Chris@4:
Returns a string indicating the library version.
Chris@4:BZFILE * BZ2_bzopen ( const char *path, const char *mode ); Chris@4: BZFILE * BZ2_bzdopen ( int fd, const char *mode );Chris@4:
Opens a .bz2 file for
Chris@4: reading or writing, using either its name or a pre-existing file
Chris@4: descriptor. Analogous to fopen
Chris@4: and fdopen.
int BZ2_bzread ( BZFILE* b, void* buf, int len ); Chris@4: int BZ2_bzwrite ( BZFILE* b, void* buf, int len );Chris@4:
Reads/writes data from/to a previously opened
Chris@4: BZFILE. Analogous to
Chris@4: fread and
Chris@4: fwrite.
int BZ2_bzflush ( BZFILE* b ); Chris@4: void BZ2_bzclose ( BZFILE* b );Chris@4:
Flushes/closes a BZFILE.
Chris@4: BZ2_bzflush doesn't actually do
Chris@4: anything. Analogous to fflush
Chris@4: and fclose.
const char * BZ2_bzerror ( BZFILE *b, int *errnum )Chris@4:
Returns a string describing the more recent error status of
Chris@4: b, and also sets
Chris@4: *errnum to its numerical
Chris@4: value.
In a deeply embedded application, you might want to use
Chris@4: just the memory-to-memory functions. You can do this
Chris@4: conveniently by compiling the library with preprocessor symbol
Chris@4: BZ_NO_STDIO defined. Doing this
Chris@4: gives you a library containing only the following eight
Chris@4: functions:
BZ2_bzCompressInit,
Chris@4: BZ2_bzCompress,
Chris@4: BZ2_bzCompressEnd
Chris@4: BZ2_bzDecompressInit,
Chris@4: BZ2_bzDecompress,
Chris@4: BZ2_bzDecompressEnd
Chris@4: BZ2_bzBuffToBuffCompress,
Chris@4: BZ2_bzBuffToBuffDecompress
When compiled like this, all functions will ignore
Chris@4: verbosity settings.
libbzip2 contains a number
Chris@4: of internal assertion checks which should, needless to say, never
Chris@4: be activated. Nevertheless, if an assertion should fail,
Chris@4: behaviour depends on whether or not the library was compiled with
Chris@4: BZ_NO_STDIO set.
For a normal compile, an assertion failure yields the Chris@4: message:
Chris@4:Chris@4:bzip2/libbzip2: internal error number N.
Chris@4:This is a bug in bzip2/libbzip2, 1.0.6 of 6 September 2010. Chris@4: Please report it to me at: jseward@bzip.org. If this happened Chris@4: when you were using some program which uses libbzip2 as a Chris@4: component, you should also report this bug to the author(s) Chris@4: of that program. Please make an effort to report this bug; Chris@4: timely and accurate bug reports eventually lead to higher Chris@4: quality software. Thanks. Julian Seward, 6 September 2010. Chris@4:
Chris@4:
where N is some error code
Chris@4: number. If N == 1007, it also
Chris@4: prints some extra text advising the reader that unreliable memory
Chris@4: is often associated with internal error 1007. (This is a
Chris@4: frequently-observed-phenomenon with versions 1.0.0/1.0.1).
exit(3) is then
Chris@4: called.
For a stdio-free library,
Chris@4: assertion failures result in a call to a function declared
Chris@4: as:
extern void bz_internal_error ( int errcode );Chris@4:
The relevant code is passed as a parameter. You should Chris@4: supply such a function.
Chris@4:In either case, once an assertion failure has occurred, any
Chris@4: bz_stream records involved can
Chris@4: be regarded as invalid. You should not attempt to resume normal
Chris@4: operation with them.
You may, of course, change critical error handling to suit Chris@4: your needs. As I said above, critical errors indicate bugs in Chris@4: the library and should not occur. All "normal" error situations Chris@4: are indicated via error return codes from functions, and can be Chris@4: recovered from.
Chris@4:Everything related to Windows has been contributed by
Chris@4: Yoshioka Tsuneo
Chris@4: (tsuneo@rr.iij4u.or.jp), so
Chris@4: you should send your queries to him (but perhaps Cc: me,
Chris@4: jseward@bzip.org).
My vague understanding of what to do is: using Visual C++
Chris@4: 5.0, open the project file
Chris@4: libbz2.dsp, and build. That's
Chris@4: all.
If you can't open the project file for some reason, make a
Chris@4: new one, naming these files:
Chris@4: blocksort.c,
Chris@4: bzlib.c,
Chris@4: compress.c,
Chris@4: crctable.c,
Chris@4: decompress.c,
Chris@4: huffman.c,
Chris@4: randtable.c and
Chris@4: libbz2.def. You will also need
Chris@4: to name the header files bzlib.h
Chris@4: and bzlib_private.h.
If you don't use VC++, you may need to define the
Chris@4: proprocessor symbol
Chris@4: _WIN32.
Finally, dlltest.c is a
Chris@4: sample program using the DLL. It has a project file,
Chris@4: dlltest.dsp.
If you just want a makefile for Visual C, have a look at
Chris@4: makefile.msc.
Be aware that if you compile
Chris@4: bzip2 itself on Win32, you must
Chris@4: set BZ_UNIX to 0 and
Chris@4: BZ_LCCWIN32 to 1, in the file
Chris@4: bzip2.c, before compiling.
Chris@4: Otherwise the resulting binary won't work correctly.
I haven't tried any of this stuff myself, but it all looks Chris@4: plausible.
Chris@4:Table of Contents
Chris@4:These are just some random thoughts of mine. Your mileage Chris@4: may vary.
Chris@4:bzip2-1.0.X,
Chris@4: 0.9.5 and
Chris@4: 0.9.0 use exactly the same file
Chris@4: format as the original version,
Chris@4: bzip2-0.1. This decision was
Chris@4: made in the interests of stability. Creating yet another
Chris@4: incompatible compressed file format would create further
Chris@4: confusion and disruption for users.
Nevertheless, this is not a painless decision. Development
Chris@4: work since the release of
Chris@4: bzip2-0.1 in August 1997 has
Chris@4: shown complexities in the file format which slow down
Chris@4: decompression and, in retrospect, are unnecessary. These
Chris@4: are:
The run-length encoder, which is the first of the Chris@4: compression transformations, is entirely irrelevant. The Chris@4: original purpose was to protect the sorting algorithm from the Chris@4: very worst case input: a string of repeated symbols. But Chris@4: algorithm steps Q6a and Q6b in the original Burrows-Wheeler Chris@4: technical report (SRC-124) show how repeats can be handled Chris@4: without difficulty in block sorting.
The randomisation mechanism doesn't really need to be Chris@4: there. Udi Manber and Gene Myers published a suffix array Chris@4: construction algorithm a few years back, which can be employed Chris@4: to sort any block, no matter how repetitive, in O(N log N) Chris@4: time. Subsequent work by Kunihiko Sadakane has produced a Chris@4: derivative O(N (log N)^2) algorithm which usually outperforms Chris@4: the Manber-Myers algorithm.
Chris@4:I could have changed to Sadakane's algorithm, but I find
Chris@4: it to be slower than bzip2's
Chris@4: existing algorithm for most inputs, and the randomisation
Chris@4: mechanism protects adequately against bad cases. I didn't
Chris@4: think it was a good tradeoff to make. Partly this is due to
Chris@4: the fact that I was not flooded with email complaints about
Chris@4: bzip2-0.1's performance on
Chris@4: repetitive data, so perhaps it isn't a problem for real
Chris@4: inputs.
Probably the best long-term solution, and the one I have Chris@4: incorporated into 0.9.5 and above, is to use the existing Chris@4: sorting algorithm initially, and fall back to a O(N (log N)^2) Chris@4: algorithm if the standard algorithm gets into Chris@4: difficulties.
Chris@4:The compressed file format was never designed to be
Chris@4: handled by a library, and I have had to jump though some hoops
Chris@4: to produce an efficient implementation of decompression. It's
Chris@4: a bit hairy. Try passing
Chris@4: decompress.c through the C
Chris@4: preprocessor and you'll see what I mean. Much of this
Chris@4: complexity could have been avoided if the compressed size of
Chris@4: each block of data was recorded in the data stream.
An Adler-32 checksum, rather than a CRC32 checksum, Chris@4: would be faster to compute.
It would be fair to say that the
Chris@4: bzip2 format was frozen before I
Chris@4: properly and fully understood the performance consequences of
Chris@4: doing so.
Improvements which I was able to incorporate into 0.9.0, Chris@4: despite using the same file format, are:
Chris@4:Single array implementation of the inverse BWT. This Chris@4: significantly speeds up decompression, presumably because it Chris@4: reduces the number of cache misses.
Faster inverse MTF transform for large MTF values. Chris@4: The new implementation is based on the notion of sliding blocks Chris@4: of values.
bzip2-0.9.0 now reads
Chris@4: and writes files with fread
Chris@4: and fwrite; version 0.1 used
Chris@4: putc and
Chris@4: getc. Duh! Well, you live
Chris@4: and learn.
Further ahead, it would be nice to be able to do random Chris@4: access into files. This will require some careful design of Chris@4: compressed file formats.
Chris@4:After some consideration, I have decided not to use GNU
Chris@4: autoconf to configure 0.9.5 or
Chris@4: 1.0.
autoconf, admirable and
Chris@4: wonderful though it is, mainly assists with portability problems
Chris@4: between Unix-like platforms. But
Chris@4: bzip2 doesn't have much in the
Chris@4: way of portability problems on Unix; most of the difficulties
Chris@4: appear when porting to the Mac, or to Microsoft's operating
Chris@4: systems. autoconf doesn't help
Chris@4: in those cases, and brings in a whole load of new
Chris@4: complexity.
Most people should be able to compile the library and Chris@4: program under Unix straight out-of-the-box, so to speak, Chris@4: especially if you have a version of GNU C available.
Chris@4:There are a couple of
Chris@4: __inline__ directives in the
Chris@4: code. GNU C (gcc) should be
Chris@4: able to handle them. If you're not using GNU C, your C compiler
Chris@4: shouldn't see them at all. If your compiler does, for some
Chris@4: reason, see them and doesn't like them, just
Chris@4: #define
Chris@4: __inline__ to be
Chris@4: /* */. One easy way to do this
Chris@4: is to compile with the flag
Chris@4: -D__inline__=, which should be
Chris@4: understood by most Unix compilers.
If you still have difficulties, try compiling with the
Chris@4: macro BZ_STRICT_ANSI defined.
Chris@4: This should enable you to build the library in a strictly ANSI
Chris@4: compliant environment. Building the program itself like this is
Chris@4: dangerous and not supported, since you remove
Chris@4: bzip2's checks against
Chris@4: compressing directories, symbolic links, devices, and other
Chris@4: not-really-a-file entities. This could cause filesystem
Chris@4: corruption!
One other thing: if you create a
Chris@4: bzip2 binary for public distribution,
Chris@4: please consider linking it statically (gcc
Chris@4: -static). This avoids all sorts of library-version
Chris@4: issues that others may encounter later on.
If you build bzip2 on
Chris@4: Win32, you must set BZ_UNIX to 0
Chris@4: and BZ_LCCWIN32 to 1, in the
Chris@4: file bzip2.c, before compiling.
Chris@4: Otherwise the resulting binary won't work correctly.
I tried pretty hard to make sure
Chris@4: bzip2 is bug free, both by
Chris@4: design and by testing. Hopefully you'll never need to read this
Chris@4: section for real.
Nevertheless, if bzip2 dies
Chris@4: with a segmentation fault, a bus error or an internal assertion
Chris@4: failure, it will ask you to email me a bug report. Experience from
Chris@4: years of feedback of bzip2 users indicates that almost all these
Chris@4: problems can be traced to either compiler bugs or hardware
Chris@4: problems.
Recompile the program with no optimisation, and
Chris@4: see if it works. And/or try a different compiler. I heard all
Chris@4: sorts of stories about various flavours of GNU C (and other
Chris@4: compilers) generating bad code for
Chris@4: bzip2, and I've run across two
Chris@4: such examples myself.
2.7.X versions of GNU C are known to generate bad code
Chris@4: from time to time, at high optimisation levels. If you get
Chris@4: problems, try using the flags
Chris@4: -O2
Chris@4: -fomit-frame-pointer
Chris@4: -fno-strength-reduce. You
Chris@4: should specifically not use
Chris@4: -funroll-loops.
You may notice that the Makefile runs six tests as part Chris@4: of the build process. If the program passes all of these, it's Chris@4: a pretty good (but not 100%) indication that the compiler has Chris@4: done its job correctly.
Chris@4:If bzip2
Chris@4: crashes randomly, and the crashes are not repeatable, you may
Chris@4: have a flaky memory subsystem.
Chris@4: bzip2 really hammers your
Chris@4: memory hierarchy, and if it's a bit marginal, you may get these
Chris@4: problems. Ditto if your disk or I/O subsystem is slowly
Chris@4: failing. Yup, this really does happen.
Try using a different machine of the same type, and see Chris@4: if you can repeat the problem.
Chris@4:This isn't really a bug, but ... If
Chris@4: bzip2 tells you your file is
Chris@4: corrupted on decompression, and you obtained the file via FTP,
Chris@4: there is a possibility that you forgot to tell FTP to do a
Chris@4: binary mode transfer. That absolutely will cause the file to
Chris@4: be non-decompressible. You'll have to transfer it
Chris@4: again.
If you've incorporated
Chris@4: libbzip2 into your own program
Chris@4: and are getting problems, please, please, please, check that the
Chris@4: parameters you are passing in calls to the library, are correct,
Chris@4: and in accordance with what the documentation says is allowable.
Chris@4: I have tried to make the library robust against such problems,
Chris@4: but I'm sure I haven't succeeded.
Finally, if the above comments don't help, you'll have to Chris@4: send me a bug report. Now, it's just amazing how many people Chris@4: will send me a bug report saying something like:
Chris@4:bzip2 crashed with segmentation fault on my machineChris@4:
and absolutely nothing else. Needless to say, a such a Chris@4: report is totally, utterly, completely and Chris@4: comprehensively 100% useless; a waste of your time, my time, and Chris@4: net bandwidth. With no details at all, there's no way Chris@4: I can possibly begin to figure out what the problem is.
Chris@4:The rules of the game are: facts, facts, facts. Don't omit Chris@4: them because "oh, they won't be relevant". At the bare Chris@4: minimum:
Chris@4:Machine type. Operating system version. Chris@4: Exact version of bzip2 (do bzip2 -V). Chris@4: Exact version of the compiler used. Chris@4: Flags passed to the compiler.Chris@4:
However, the most important single thing that will help me Chris@4: is the file that you were trying to compress or decompress at the Chris@4: time the problem happened. Without that, my ability to do Chris@4: anything more than speculate about the cause, is limited.
Chris@4:bzip2 is a resource hog.
Chris@4: It soaks up large amounts of CPU cycles and memory. Also, it
Chris@4: gives very large latencies. In the worst case, you can feed many
Chris@4: megabytes of uncompressed data into the library before getting
Chris@4: any compressed output, so this probably rules out applications
Chris@4: requiring interactive behaviour.
These aren't faults of my implementation, I hope, but more Chris@4: an intrinsic property of the Burrows-Wheeler transform Chris@4: (unfortunately). Maybe this isn't what you want.
Chris@4:If you want a compressor and/or library which is faster,
Chris@4: uses less memory but gets pretty good compression, and has
Chris@4: minimal latency, consider Jean-loup Gailly's and Mark Adler's
Chris@4: work, zlib-1.2.1 and
Chris@4: gzip-1.2.4. Look for them at
Chris@4: http://www.zlib.org and
Chris@4: http://www.gzip.org
Chris@4: respectively.
For something faster and lighter still, you might try Markus F
Chris@4: X J Oberhumer's LZO real-time
Chris@4: compression/decompression library, at
Chris@4: http://www.oberhumer.com/opensource.
bzip2 is not research
Chris@4: work, in the sense that it doesn't present any new ideas.
Chris@4: Rather, it's an engineering exercise based on existing
Chris@4: ideas.
Four documents describe essentially all the ideas behind
Chris@4: bzip2:
Michael Burrows and D. J. Wheeler:
Chris@4: "A block-sorting lossless data compression algorithm"
Chris@4: 10th May 1994.
Chris@4: Digital SRC Research Report 124.
Chris@4: ftp://ftp.digital.com/pub/DEC/SRC/research-reports/SRC-124.ps.gz
Chris@4: If you have trouble finding it, try searching at the
Chris@4: New Zealand Digital Library, http://www.nzdl.org.
Chris@4:
Chris@4: Daniel S. Hirschberg and Debra A. LeLewer
Chris@4: "Efficient Decoding of Prefix Codes"
Chris@4: Communications of the ACM, April 1990, Vol 33, Number 4.
Chris@4: You might be able to get an electronic copy of this
Chris@4: from the ACM Digital Library.
Chris@4:
Chris@4: David J. Wheeler
Chris@4: Program bred3.c and accompanying document bred3.ps.
Chris@4: This contains the idea behind the multi-table Huffman coding scheme.
Chris@4: ftp://ftp.cl.cam.ac.uk/users/djw3/
Chris@4:
Chris@4: Jon L. Bentley and Robert Sedgewick
Chris@4: "Fast Algorithms for Sorting and Searching Strings"
Chris@4: Available from Sedgewick's web page,
Chris@4: www.cs.princeton.edu/~rs
Chris@4:
The following paper gives valuable additional insights into Chris@4: the algorithm, but is not immediately the basis of any code used Chris@4: in bzip2.
Chris@4:Peter Fenwick:
Chris@4: Block Sorting Text Compression
Chris@4: Proceedings of the 19th Australasian Computer Science Conference,
Chris@4: Melbourne, Australia. Jan 31 - Feb 2, 1996.
Chris@4: ftp://ftp.cs.auckland.ac.nz/pub/peter-f/ACSC96paper.ps
Kunihiko Sadakane's sorting algorithm, mentioned above, is Chris@4: available from:
Chris@4:http://naomi.is.s.u-tokyo.ac.jp/~sada/papers/Sada98b.ps.gz
Chris@4:
The Manber-Myers suffix array construction algorithm is Chris@4: described in a paper available from:
Chris@4:http://www.cs.arizona.edu/people/gene/PAPERS/suffix.ps
Chris@4:
Finally, the following papers document some Chris@4: investigations I made into the performance of sorting Chris@4: and decompression algorithms:
Chris@4:Julian Seward
Chris@4: On the Performance of BWT Sorting Algorithms
Chris@4: Proceedings of the IEEE Data Compression Conference 2000
Chris@4: Snowbird, Utah. 28-30 March 2000.
Chris@4:
Chris@4: Julian Seward
Chris@4: Space-time Tradeoffs in the Inverse B-W Transform
Chris@4: Proceedings of the IEEE Data Compression Conference 2001
Chris@4: Snowbird, Utah. 27-29 March 2001.
Chris@4: