annotate src/bzip2-1.0.6/bzip2.1.preformatted @ 89:8a15ff55d9af

Add bzip2, zlib, liblo, portaudio sources
author Chris Cannam <cannam@all-day-breakfast.com>
date Wed, 20 Mar 2013 13:59:52 +0000
parents
children
rev   line source
cannam@89 1 bzip2(1) bzip2(1)
cannam@89 2
cannam@89 3
cannam@89 4
cannam@89 5 NNAAMMEE
cannam@89 6 bzip2, bunzip2 − a block‐sorting file compressor, v1.0.6
cannam@89 7 bzcat − decompresses files to stdout
cannam@89 8 bzip2recover − recovers data from damaged bzip2 files
cannam@89 9
cannam@89 10
cannam@89 11 SSYYNNOOPPSSIISS
cannam@89 12 bbzziipp22 [ −−ccddffkkqqssttvvzzVVLL112233445566778899 ] [ _f_i_l_e_n_a_m_e_s _._._. ]
cannam@89 13 bbuunnzziipp22 [ −−ffkkvvssVVLL ] [ _f_i_l_e_n_a_m_e_s _._._. ]
cannam@89 14 bbzzccaatt [ −−ss ] [ _f_i_l_e_n_a_m_e_s _._._. ]
cannam@89 15 bbzziipp22rreeccoovveerr _f_i_l_e_n_a_m_e
cannam@89 16
cannam@89 17
cannam@89 18 DDEESSCCRRIIPPTTIIOONN
cannam@89 19 _b_z_i_p_2 compresses files using the Burrows‐Wheeler block
cannam@89 20 sorting text compression algorithm, and Huffman coding.
cannam@89 21 Compression is generally considerably better than that
cannam@89 22 achieved by more conventional LZ77/LZ78‐based compressors,
cannam@89 23 and approaches the performance of the PPM family of sta­
cannam@89 24 tistical compressors.
cannam@89 25
cannam@89 26 The command‐line options are deliberately very similar to
cannam@89 27 those of _G_N_U _g_z_i_p_, but they are not identical.
cannam@89 28
cannam@89 29 _b_z_i_p_2 expects a list of file names to accompany the com­
cannam@89 30 mand‐line flags. Each file is replaced by a compressed
cannam@89 31 version of itself, with the name "original_name.bz2".
cannam@89 32 Each compressed file has the same modification date, per­
cannam@89 33 missions, and, when possible, ownership as the correspond­
cannam@89 34 ing original, so that these properties can be correctly
cannam@89 35 restored at decompression time. File name handling is
cannam@89 36 naive in the sense that there is no mechanism for preserv­
cannam@89 37 ing original file names, permissions, ownerships or dates
cannam@89 38 in filesystems which lack these concepts, or have serious
cannam@89 39 file name length restrictions, such as MS‐DOS.
cannam@89 40
cannam@89 41 _b_z_i_p_2 and _b_u_n_z_i_p_2 will by default not overwrite existing
cannam@89 42 files. If you want this to happen, specify the −f flag.
cannam@89 43
cannam@89 44 If no file names are specified, _b_z_i_p_2 compresses from
cannam@89 45 standard input to standard output. In this case, _b_z_i_p_2
cannam@89 46 will decline to write compressed output to a terminal, as
cannam@89 47 this would be entirely incomprehensible and therefore
cannam@89 48 pointless.
cannam@89 49
cannam@89 50 _b_u_n_z_i_p_2 (or _b_z_i_p_2 _−_d_) decompresses all specified files.
cannam@89 51 Files which were not created by _b_z_i_p_2 will be detected and
cannam@89 52 ignored, and a warning issued. _b_z_i_p_2 attempts to guess
cannam@89 53 the filename for the decompressed file from that of the
cannam@89 54 compressed file as follows:
cannam@89 55
cannam@89 56 filename.bz2 becomes filename
cannam@89 57 filename.bz becomes filename
cannam@89 58 filename.tbz2 becomes filename.tar
cannam@89 59 filename.tbz becomes filename.tar
cannam@89 60 anyothername becomes anyothername.out
cannam@89 61
cannam@89 62 If the file does not end in one of the recognised endings,
cannam@89 63 _._b_z_2_, _._b_z_, _._t_b_z_2 or _._t_b_z_, _b_z_i_p_2 complains that it cannot
cannam@89 64 guess the name of the original file, and uses the original
cannam@89 65 name with _._o_u_t appended.
cannam@89 66
cannam@89 67 As with compression, supplying no filenames causes decom­
cannam@89 68 pression from standard input to standard output.
cannam@89 69
cannam@89 70 _b_u_n_z_i_p_2 will correctly decompress a file which is the con­
cannam@89 71 catenation of two or more compressed files. The result is
cannam@89 72 the concatenation of the corresponding uncompressed files.
cannam@89 73 Integrity testing (−t) of concatenated compressed files is
cannam@89 74 also supported.
cannam@89 75
cannam@89 76 You can also compress or decompress files to the standard
cannam@89 77 output by giving the −c flag. Multiple files may be com­
cannam@89 78 pressed and decompressed like this. The resulting outputs
cannam@89 79 are fed sequentially to stdout. Compression of multiple
cannam@89 80 files in this manner generates a stream containing multi­
cannam@89 81 ple compressed file representations. Such a stream can be
cannam@89 82 decompressed correctly only by _b_z_i_p_2 version 0.9.0 or
cannam@89 83 later. Earlier versions of _b_z_i_p_2 will stop after decom­
cannam@89 84 pressing the first file in the stream.
cannam@89 85
cannam@89 86 _b_z_c_a_t (or _b_z_i_p_2 _‐_d_c_) decompresses all specified files to
cannam@89 87 the standard output.
cannam@89 88
cannam@89 89 _b_z_i_p_2 will read arguments from the environment variables
cannam@89 90 _B_Z_I_P_2 and _B_Z_I_P_, in that order, and will process them
cannam@89 91 before any arguments read from the command line. This
cannam@89 92 gives a convenient way to supply default arguments.
cannam@89 93
cannam@89 94 Compression is always performed, even if the compressed
cannam@89 95 file is slightly larger than the original. Files of less
cannam@89 96 than about one hundred bytes tend to get larger, since the
cannam@89 97 compression mechanism has a constant overhead in the
cannam@89 98 region of 50 bytes. Random data (including the output of
cannam@89 99 most file compressors) is coded at about 8.05 bits per
cannam@89 100 byte, giving an expansion of around 0.5%.
cannam@89 101
cannam@89 102 As a self‐check for your protection, _b_z_i_p_2 uses 32‐bit
cannam@89 103 CRCs to make sure that the decompressed version of a file
cannam@89 104 is identical to the original. This guards against corrup­
cannam@89 105 tion of the compressed data, and against undetected bugs
cannam@89 106 in _b_z_i_p_2 (hopefully very unlikely). The chances of data
cannam@89 107 corruption going undetected is microscopic, about one
cannam@89 108 chance in four billion for each file processed. Be aware,
cannam@89 109 though, that the check occurs upon decompression, so it
cannam@89 110 can only tell you that something is wrong. It can’t help
cannam@89 111 you recover the original uncompressed data. You can use
cannam@89 112 _b_z_i_p_2_r_e_c_o_v_e_r to try to recover data from damaged files.
cannam@89 113
cannam@89 114 Return values: 0 for a normal exit, 1 for environmental
cannam@89 115 problems (file not found, invalid flags, I/O errors, &c),
cannam@89 116 2 to indicate a corrupt compressed file, 3 for an internal
cannam@89 117 consistency error (eg, bug) which caused _b_z_i_p_2 to panic.
cannam@89 118
cannam@89 119
cannam@89 120 OOPPTTIIOONNSS
cannam@89 121 −−cc ‐‐‐‐ssttddoouutt
cannam@89 122 Compress or decompress to standard output.
cannam@89 123
cannam@89 124 −−dd ‐‐‐‐ddeeccoommpprreessss
cannam@89 125 Force decompression. _b_z_i_p_2_, _b_u_n_z_i_p_2 and _b_z_c_a_t are
cannam@89 126 really the same program, and the decision about
cannam@89 127 what actions to take is done on the basis of which
cannam@89 128 name is used. This flag overrides that mechanism,
cannam@89 129 and forces _b_z_i_p_2 to decompress.
cannam@89 130
cannam@89 131 −−zz ‐‐‐‐ccoommpprreessss
cannam@89 132 The complement to −d: forces compression,
cannam@89 133 regardless of the invocation name.
cannam@89 134
cannam@89 135 −−tt ‐‐‐‐tteesstt
cannam@89 136 Check integrity of the specified file(s), but don’t
cannam@89 137 decompress them. This really performs a trial
cannam@89 138 decompression and throws away the result.
cannam@89 139
cannam@89 140 −−ff ‐‐‐‐ffoorrccee
cannam@89 141 Force overwrite of output files. Normally, _b_z_i_p_2
cannam@89 142 will not overwrite existing output files. Also
cannam@89 143 forces _b_z_i_p_2 to break hard links to files, which it
cannam@89 144 otherwise wouldn’t do.
cannam@89 145
cannam@89 146 bzip2 normally declines to decompress files which
cannam@89 147 don’t have the correct magic header bytes. If
cannam@89 148 forced (‐f), however, it will pass such files
cannam@89 149 through unmodified. This is how GNU gzip behaves.
cannam@89 150
cannam@89 151 −−kk ‐‐‐‐kkeeeepp
cannam@89 152 Keep (don’t delete) input files during compression
cannam@89 153 or decompression.
cannam@89 154
cannam@89 155 −−ss ‐‐‐‐ssmmaallll
cannam@89 156 Reduce memory usage, for compression, decompression
cannam@89 157 and testing. Files are decompressed and tested
cannam@89 158 using a modified algorithm which only requires 2.5
cannam@89 159 bytes per block byte. This means any file can be
cannam@89 160 decompressed in 2300k of memory, albeit at about
cannam@89 161 half the normal speed.
cannam@89 162
cannam@89 163 During compression, −s selects a block size of
cannam@89 164 200k, which limits memory use to around the same
cannam@89 165 figure, at the expense of your compression ratio.
cannam@89 166 In short, if your machine is low on memory (8
cannam@89 167 megabytes or less), use −s for everything. See
cannam@89 168 MEMORY MANAGEMENT below.
cannam@89 169
cannam@89 170 −−qq ‐‐‐‐qquuiieett
cannam@89 171 Suppress non‐essential warning messages. Messages
cannam@89 172 pertaining to I/O errors and other critical events
cannam@89 173 will not be suppressed.
cannam@89 174
cannam@89 175 −−vv ‐‐‐‐vveerrbboossee
cannam@89 176 Verbose mode ‐‐ show the compression ratio for each
cannam@89 177 file processed. Further −v’s increase the ver­
cannam@89 178 bosity level, spewing out lots of information which
cannam@89 179 is primarily of interest for diagnostic purposes.
cannam@89 180
cannam@89 181 −−LL ‐‐‐‐lliicceennssee ‐‐VV ‐‐‐‐vveerrssiioonn
cannam@89 182 Display the software version, license terms and
cannam@89 183 conditions.
cannam@89 184
cannam@89 185 −−11 ((oorr −−−−ffaasstt)) ttoo −−99 ((oorr −−−−bbeesstt))
cannam@89 186 Set the block size to 100 k, 200 k .. 900 k when
cannam@89 187 compressing. Has no effect when decompressing.
cannam@89 188 See MEMORY MANAGEMENT below. The −−fast and −−best
cannam@89 189 aliases are primarily for GNU gzip compatibility.
cannam@89 190 In particular, −−fast doesn’t make things signifi­
cannam@89 191 cantly faster. And −−best merely selects the
cannam@89 192 default behaviour.
cannam@89 193
cannam@89 194 −−‐‐ Treats all subsequent arguments as file names, even
cannam@89 195 if they start with a dash. This is so you can han­
cannam@89 196 dle files with names beginning with a dash, for
cannam@89 197 example: bzip2 −‐ −myfilename.
cannam@89 198
cannam@89 199 −−‐‐rreeppeettiittiivvee‐‐ffaasstt ‐‐‐‐rreeppeettiittiivvee‐‐bbeesstt
cannam@89 200 These flags are redundant in versions 0.9.5 and
cannam@89 201 above. They provided some coarse control over the
cannam@89 202 behaviour of the sorting algorithm in earlier ver­
cannam@89 203 sions, which was sometimes useful. 0.9.5 and above
cannam@89 204 have an improved algorithm which renders these
cannam@89 205 flags irrelevant.
cannam@89 206
cannam@89 207
cannam@89 208 MMEEMMOORRYY MMAANNAAGGEEMMEENNTT
cannam@89 209 _b_z_i_p_2 compresses large files in blocks. The block size
cannam@89 210 affects both the compression ratio achieved, and the
cannam@89 211 amount of memory needed for compression and decompression.
cannam@89 212 The flags −1 through −9 specify the block size to be
cannam@89 213 100,000 bytes through 900,000 bytes (the default) respec­
cannam@89 214 tively. At decompression time, the block size used for
cannam@89 215 compression is read from the header of the compressed
cannam@89 216 file, and _b_u_n_z_i_p_2 then allocates itself just enough memory
cannam@89 217 to decompress the file. Since block sizes are stored in
cannam@89 218 compressed files, it follows that the flags −1 to −9 are
cannam@89 219 irrelevant to and so ignored during decompression.
cannam@89 220
cannam@89 221 Compression and decompression requirements, in bytes, can
cannam@89 222 be estimated as:
cannam@89 223
cannam@89 224 Compression: 400k + ( 8 x block size )
cannam@89 225
cannam@89 226 Decompression: 100k + ( 4 x block size ), or
cannam@89 227 100k + ( 2.5 x block size )
cannam@89 228
cannam@89 229 Larger block sizes give rapidly diminishing marginal
cannam@89 230 returns. Most of the compression comes from the first two
cannam@89 231 or three hundred k of block size, a fact worth bearing in
cannam@89 232 mind when using _b_z_i_p_2 on small machines. It is also
cannam@89 233 important to appreciate that the decompression memory
cannam@89 234 requirement is set at compression time by the choice of
cannam@89 235 block size.
cannam@89 236
cannam@89 237 For files compressed with the default 900k block size,
cannam@89 238 _b_u_n_z_i_p_2 will require about 3700 kbytes to decompress. To
cannam@89 239 support decompression of any file on a 4 megabyte machine,
cannam@89 240 _b_u_n_z_i_p_2 has an option to decompress using approximately
cannam@89 241 half this amount of memory, about 2300 kbytes. Decompres­
cannam@89 242 sion speed is also halved, so you should use this option
cannam@89 243 only where necessary. The relevant flag is ‐s.
cannam@89 244
cannam@89 245 In general, try and use the largest block size memory con­
cannam@89 246 straints allow, since that maximises the compression
cannam@89 247 achieved. Compression and decompression speed are virtu­
cannam@89 248 ally unaffected by block size.
cannam@89 249
cannam@89 250 Another significant point applies to files which fit in a
cannam@89 251 single block ‐‐ that means most files you’d encounter
cannam@89 252 using a large block size. The amount of real memory
cannam@89 253 touched is proportional to the size of the file, since the
cannam@89 254 file is smaller than a block. For example, compressing a
cannam@89 255 file 20,000 bytes long with the flag ‐9 will cause the
cannam@89 256 compressor to allocate around 7600k of memory, but only
cannam@89 257 touch 400k + 20000 * 8 = 560 kbytes of it. Similarly, the
cannam@89 258 decompressor will allocate 3700k but only touch 100k +
cannam@89 259 20000 * 4 = 180 kbytes.
cannam@89 260
cannam@89 261 Here is a table which summarises the maximum memory usage
cannam@89 262 for different block sizes. Also recorded is the total
cannam@89 263 compressed size for 14 files of the Calgary Text Compres­
cannam@89 264 sion Corpus totalling 3,141,622 bytes. This column gives
cannam@89 265 some feel for how compression varies with block size.
cannam@89 266 These figures tend to understate the advantage of larger
cannam@89 267 block sizes for larger files, since the Corpus is domi­
cannam@89 268 nated by smaller files.
cannam@89 269
cannam@89 270 Compress Decompress Decompress Corpus
cannam@89 271 Flag usage usage ‐s usage Size
cannam@89 272
cannam@89 273 ‐1 1200k 500k 350k 914704
cannam@89 274 ‐2 2000k 900k 600k 877703
cannam@89 275 ‐3 2800k 1300k 850k 860338
cannam@89 276 ‐4 3600k 1700k 1100k 846899
cannam@89 277 ‐5 4400k 2100k 1350k 845160
cannam@89 278 ‐6 5200k 2500k 1600k 838626
cannam@89 279 ‐7 6100k 2900k 1850k 834096
cannam@89 280 ‐8 6800k 3300k 2100k 828642
cannam@89 281 ‐9 7600k 3700k 2350k 828642
cannam@89 282
cannam@89 283
cannam@89 284 RREECCOOVVEERRIINNGG DDAATTAA FFRROOMM DDAAMMAAGGEEDD FFIILLEESS
cannam@89 285 _b_z_i_p_2 compresses files in blocks, usually 900kbytes long.
cannam@89 286 Each block is handled independently. If a media or trans­
cannam@89 287 mission error causes a multi‐block .bz2 file to become
cannam@89 288 damaged, it may be possible to recover data from the
cannam@89 289 undamaged blocks in the file.
cannam@89 290
cannam@89 291 The compressed representation of each block is delimited
cannam@89 292 by a 48‐bit pattern, which makes it possible to find the
cannam@89 293 block boundaries with reasonable certainty. Each block
cannam@89 294 also carries its own 32‐bit CRC, so damaged blocks can be
cannam@89 295 distinguished from undamaged ones.
cannam@89 296
cannam@89 297 _b_z_i_p_2_r_e_c_o_v_e_r is a simple program whose purpose is to
cannam@89 298 search for blocks in .bz2 files, and write each block out
cannam@89 299 into its own .bz2 file. You can then use _b_z_i_p_2 −t to test
cannam@89 300 the integrity of the resulting files, and decompress those
cannam@89 301 which are undamaged.
cannam@89 302
cannam@89 303 _b_z_i_p_2_r_e_c_o_v_e_r takes a single argument, the name of the dam­
cannam@89 304 aged file, and writes a number of files
cannam@89 305 "rec00001file.bz2", "rec00002file.bz2", etc, containing
cannam@89 306 the extracted blocks. The output filenames are
cannam@89 307 designed so that the use of wildcards in subsequent pro­
cannam@89 308 cessing ‐‐ for example, "bzip2 ‐dc rec*file.bz2 > recov­
cannam@89 309 ered_data" ‐‐ processes the files in the correct order.
cannam@89 310
cannam@89 311 _b_z_i_p_2_r_e_c_o_v_e_r should be of most use dealing with large .bz2
cannam@89 312 files, as these will contain many blocks. It is clearly
cannam@89 313 futile to use it on damaged single‐block files, since a
cannam@89 314 damaged block cannot be recovered. If you wish to min­
cannam@89 315 imise any potential data loss through media or transmis­
cannam@89 316 sion errors, you might consider compressing with a smaller
cannam@89 317 block size.
cannam@89 318
cannam@89 319
cannam@89 320 PPEERRFFOORRMMAANNCCEE NNOOTTEESS
cannam@89 321 The sorting phase of compression gathers together similar
cannam@89 322 strings in the file. Because of this, files containing
cannam@89 323 very long runs of repeated symbols, like "aabaabaabaab
cannam@89 324 ..." (repeated several hundred times) may compress more
cannam@89 325 slowly than normal. Versions 0.9.5 and above fare much
cannam@89 326 better than previous versions in this respect. The ratio
cannam@89 327 between worst‐case and average‐case compression time is in
cannam@89 328 the region of 10:1. For previous versions, this figure
cannam@89 329 was more like 100:1. You can use the −vvvv option to mon­
cannam@89 330 itor progress in great detail, if you want.
cannam@89 331
cannam@89 332 Decompression speed is unaffected by these phenomena.
cannam@89 333
cannam@89 334 _b_z_i_p_2 usually allocates several megabytes of memory to
cannam@89 335 operate in, and then charges all over it in a fairly ran­
cannam@89 336 dom fashion. This means that performance, both for com­
cannam@89 337 pressing and decompressing, is largely determined by the
cannam@89 338 speed at which your machine can service cache misses.
cannam@89 339 Because of this, small changes to the code to reduce the
cannam@89 340 miss rate have been observed to give disproportionately
cannam@89 341 large performance improvements. I imagine _b_z_i_p_2 will per­
cannam@89 342 form best on machines with very large caches.
cannam@89 343
cannam@89 344
cannam@89 345 CCAAVVEEAATTSS
cannam@89 346 I/O error messages are not as helpful as they could be.
cannam@89 347 _b_z_i_p_2 tries hard to detect I/O errors and exit cleanly,
cannam@89 348 but the details of what the problem is sometimes seem
cannam@89 349 rather misleading.
cannam@89 350
cannam@89 351 This manual page pertains to version 1.0.6 of _b_z_i_p_2_. Com­
cannam@89 352 pressed data created by this version is entirely forwards
cannam@89 353 and backwards compatible with the previous public
cannam@89 354 releases, versions 0.1pl2, 0.9.0, 0.9.5, 1.0.0, 1.0.1,
cannam@89 355 1.0.2 and above, but with the following exception: 0.9.0
cannam@89 356 and above can correctly decompress multiple concatenated
cannam@89 357 compressed files. 0.1pl2 cannot do this; it will stop
cannam@89 358 after decompressing just the first file in the stream.
cannam@89 359
cannam@89 360 _b_z_i_p_2_r_e_c_o_v_e_r versions prior to 1.0.2 used 32‐bit integers
cannam@89 361 to represent bit positions in compressed files, so they
cannam@89 362 could not handle compressed files more than 512 megabytes
cannam@89 363 long. Versions 1.0.2 and above use 64‐bit ints on some
cannam@89 364 platforms which support them (GNU supported targets, and
cannam@89 365 Windows). To establish whether or not bzip2recover was
cannam@89 366 built with such a limitation, run it without arguments.
cannam@89 367 In any event you can build yourself an unlimited version
cannam@89 368 if you can recompile it with MaybeUInt64 set to be an
cannam@89 369 unsigned 64‐bit integer.
cannam@89 370
cannam@89 371
cannam@89 372
cannam@89 373
cannam@89 374 AAUUTTHHOORR
cannam@89 375 Julian Seward, jsewardbzip.org.
cannam@89 376
cannam@89 377 http://www.bzip.org
cannam@89 378
cannam@89 379 The ideas embodied in _b_z_i_p_2 are due to (at least) the fol­
cannam@89 380 lowing people: Michael Burrows and David Wheeler (for the
cannam@89 381 block sorting transformation), David Wheeler (again, for
cannam@89 382 the Huffman coder), Peter Fenwick (for the structured cod­
cannam@89 383 ing model in the original _b_z_i_p_, and many refinements), and
cannam@89 384 Alistair Moffat, Radford Neal and Ian Witten (for the
cannam@89 385 arithmetic coder in the original _b_z_i_p_)_. I am much
cannam@89 386 indebted for their help, support and advice. See the man­
cannam@89 387 ual in the source distribution for pointers to sources of
cannam@89 388 documentation. Christian von Roques encouraged me to look
cannam@89 389 for faster sorting algorithms, so as to speed up compres­
cannam@89 390 sion. Bela Lubkin encouraged me to improve the worst‐case
cannam@89 391 compression performance. Donna Robinson XMLised the docu­
cannam@89 392 mentation. The bz* scripts are derived from those of GNU
cannam@89 393 gzip. Many people sent patches, helped with portability
cannam@89 394 problems, lent machines, gave advice and were generally
cannam@89 395 helpful.
cannam@89 396
cannam@89 397
cannam@89 398
cannam@89 399 bzip2(1)