Daniel@0: # Delta compression
Daniel@0: 
Daniel@0: Scripts in dml-cliopatrial/cpack/dml/scripts/compression provide a common interface
Daniel@0: to several delta compression programs.  The interface is 
Daniel@0: 
Daniel@0: 	stdin --->  [ <script name>  (encode|decode) <name of reference file> ] ---> stdout
Daniel@0: 
Daniel@0: The following scripts work this way:
Daniel@0: 
Daniel@0: 	zbs - use bsdiff
Daniel@0: 	zxd - uses xdelta3
Daniel@0: 	zvcd - uses open-vcdiff
Daniel@0: 	zvcz - uses vczip
Daniel@0: 	zdiff - converts binary to text and uses diff to produce an ed script
Daniel@0: 
Daniel@0: # bufs
Daniel@0: 
Daniel@0: The bufs script allows an arbitrary command to be run such that if the command expects a
Daniel@0: filename as its nth argument, then 
Daniel@0: 
Daniel@0: 	$ bufs <n> <command> <arg1> ... <argn> ...
Daniel@0: 	
Daniel@0: can be run with <argn> as a bash process redirection, even if <command> reads that
Daniel@0: source several times. bufs works by buffering the stream on the nth argument to a temporary
Daniel@0: file.
Daniel@0: 
Daniel@0: 
Daniel@0: # findcat
Daniel@0: 
Daniel@0: findcat dumps the contents of every file under a given directory to stdout.
Daniel@0: 
Daniel@0: # Examples
Daniel@0: 
Daniel@0: For example, to estimate the conditional K.C. of all the humdrum files in ~/lib/kern/ireland
Daniel@0: given those in ~/lib/ker/lorraine, using xdelta3,  do
Daniel@0: 
Daniel@0: 	$ findcat ~/lib/kern/ireland | bufs 2 zxd encode <(findcat ~/lib/kern/lorraine) | length
Daniel@0: 
Daniel@0: Scripts encode/decode include bufs, so an alternative is
Daniel@0: 
Daniel@0: 	$ findcat ~/lib/kern/ireland | encode zxd <(findcat ~/lib/kern/lorraine) | length
Daniel@0: 
Daniel@0: A better estimate is
Daniel@0: 
Daniel@0: 	$ findcat ~/lib/kern/ireland | rid -G | encode zxd <(findcat ~/lib/kern/lorraine | rid -G) | length
Daniel@0: 
Daniel@0: Sometimes the output can be compressed still further:
Daniel@0: 
Daniel@0: 	$ findcat ~/lib/kern/ireland | rid -G | encode zxd <(findcat ~/lib/kern/lorraine | rid -G) | lzma | length
Daniel@0: 
Daniel@0: rid -G is a humdrum command that removes comments.
Daniel@0: 
Daniel@0: 
Daniel@0: # zzcd and zzd
Daniel@0: 
Daniel@0: The scripts zzd and zzcd implement more complex schemes, where the input and the reference
Daniel@0: are concatenated and/or compressed before delta compressed. For example,
Daniel@0: 
Daniel@0: 	$ findcat ~/lib/kern/ireland | rid -G | bufs 3 zzcd lzma zxd  <(findcat ~/lib/kern/lorraine | rid -G) | length
Daniel@0: 
Daniel@0: computes (using a more functional notation)
Daniel@0: 
Daniel@0: 	length( zxd( lzma(lorraine), lzma(lorraine+ireland)))
Daniel@0: 
Daniel@0: that is, the amount of information needed to transform one LZMA compressed corpus
Daniel@0: into the LZMA compressed concatenation of two corpuses.
Daniel@0: 
Daniel@0: # dlzma
Daniel@0: 
Daniel@0: This is a program written in C using liblzma (part of xz utils package) to estimate the conditional
Daniel@0: complexity of an object. It works by using the SYNC_FLUSH feature of liblzma. The compressed data is
Daniel@0: discarded and only the number of bits used is output on stdout.