wolffd@0:
wolffd@0: Bayes Net Toolbox for Matlab
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: Bayes Net Toolbox for Matlab
wolffd@0: Written by Kevin Murphy, 1997--2002.
wolffd@0: Last updated: 19 October 2007.
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: - BNT supports many types of
wolffd@0: conditional probability distributions (nodes),
wolffd@0: and it is easy to add more.
wolffd@0:
wolffd@0: - Tabular (multinomial)
wolffd@0:
- Gaussian
wolffd@0:
- Softmax (logistic/ sigmoid)
wolffd@0:
- Multi-layer perceptron (neural network)
wolffd@0:
- Noisy-or
wolffd@0:
- Deterministic
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
- BNT supports decision and utility nodes, as well as chance
wolffd@0: nodes,
wolffd@0: i.e., influence diagrams as well as Bayes nets.
wolffd@0:
wolffd@0:
wolffd@0:
- BNT supports static and dynamic BNs (useful for modelling dynamical systems
wolffd@0: and sequence data).
wolffd@0:
wolffd@0:
wolffd@0:
- BNT supports many different inference algorithms,
wolffd@0: and it is easy to add more.
wolffd@0:
wolffd@0:
wolffd@0: - Exact inference for static BNs:
wolffd@0:
wolffd@0: - junction tree
wolffd@0:
- variable elimination
wolffd@0:
- brute force enumeration (for discrete nets)
wolffd@0:
- linear algebra (for Gaussian nets)
wolffd@0:
- Pearl's algorithm (for polytrees)
wolffd@0:
- quickscore (for QMR)
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
- Approximate inference for static BNs:
wolffd@0:
wolffd@0: - likelihood weighting
wolffd@0:
- Gibbs sampling
wolffd@0:
- loopy belief propagation
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
- Exact inference for DBNs:
wolffd@0:
wolffd@0: - junction tree
wolffd@0:
- frontier algorithm
wolffd@0:
- forwards-backwards (for HMMs)
wolffd@0:
- Kalman-RTS (for LDSs)
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
- Approximate inference for DBNs:
wolffd@0:
wolffd@0: - Boyen-Koller
wolffd@0:
- factored-frontier/loopy belief propagation
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: BNT supports several methods for parameter learning,
wolffd@0: and it is easy to add more.
wolffd@0:
wolffd@0:
wolffd@0: - Batch MLE/MAP parameter learning using EM.
wolffd@0: (Each node type has its own M method, e.g. softmax nodes use IRLS,
wolffd@0: and each inference engine has its own E method, so the code is fully modular.)
wolffd@0:
wolffd@0: - Sequential/batch Bayesian parameter learning (for fully observed tabular nodes only).
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: BNT supports several methods for regularization,
wolffd@0: and it is easy to add more.
wolffd@0:
wolffd@0: - Any node can have its parameters clamped (made non-adjustable).
wolffd@0:
- Any set of compatible nodes can have their parameters tied (c.f.,
wolffd@0: weight sharing in a neural net).
wolffd@0:
- Some node types (e.g., tabular) supports priors for MAP estimation.
wolffd@0:
- Gaussian covariance matrices can be declared full or diagonal, and can
wolffd@0: be tied across states of their discrete parents (if any).
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: BNT supports several methods for structure learning,
wolffd@0: and it is easy to add more.
wolffd@0:
wolffd@0:
wolffd@0: - Bayesian structure learning,
wolffd@0: using MCMC or local search (for fully observed tabular nodes only).
wolffd@0:
wolffd@0:
- Constraint-based structure learning (IC/PC and IC*/FCI).
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
- The source code is extensively documented, object-oriented, and free, making it
wolffd@0: an excellent tool for teaching, research and rapid prototyping.
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: It is trivial to implement all of
wolffd@0: the following probabilistic models using the toolbox.
wolffd@0:
wolffd@0: - Static
wolffd@0:
wolffd@0: - Linear regression, logistic regression, hierarchical mixtures of experts
wolffd@0:
wolffd@0:
- Naive Bayes classifiers, mixtures of Gaussians,
wolffd@0: sigmoid belief nets
wolffd@0:
wolffd@0:
- Factor analysis, probabilistic
wolffd@0: PCA, probabilistic ICA, mixtures of these models
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: - Dynamic
wolffd@0:
wolffd@0:
wolffd@0: - HMMs, Factorial HMMs, coupled HMMs, input-output HMMs, DBNs
wolffd@0:
wolffd@0:
- Kalman filters, ARMAX models, switching Kalman filters,
wolffd@0: tree-structured Kalman filters, multiscale AR models
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: - Many other combinations, for which there are (as yet) no names!
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: -
wolffd@0: I was hoping for a Linux-style effect, whereby people would contribute
wolffd@0: their own Matlab code so that the package would grow. With a few
wolffd@0: exceptions, this has not happened,
wolffd@0: although several people have provided bug-fixes (see the acknowledgements).
wolffd@0: Perhaps the Open
wolffd@0: Bayes Project will be more
wolffd@0: succesful in this regard, although the evidence to date is not promising.
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: Knowing that someone else might read your code forces one to
wolffd@0: document it properly, a good practice in any case, as anyone knows who
wolffd@0: has revisited old code.
wolffd@0: In addition, by having many "eye balls", it is easier to spot bugs.
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: I believe in the concept of
wolffd@0:
wolffd@0: reproducible research.
wolffd@0: Good science requires that other people be able
wolffd@0: to replicate your experiments.
wolffd@0: Often a paper does not give enough details about how exactly an
wolffd@0: algorithm was implemented (e.g., how were the parameters chosen? what
wolffd@0: initial conditions were used?), and these can make a big difference in
wolffd@0: practice.
wolffd@0: Hence one should release the code that
wolffd@0: was actually used to generate the results in one's paper.
wolffd@0: This also prevents re-inventing the wheel.
wolffd@0:
wolffd@0:
wolffd@0:
-
wolffd@0: I was fed up with reading papers where all people do is figure out how
wolffd@0: to do exact inference and/or learning
wolffd@0: in a model which is just a trivial special case of a general Bayes net, e.g.,
wolffd@0: input-output HMMs, coupled-HMMs, auto-regressive HMMs.
wolffd@0: My hope is that, by releasing general purpose software, the field can
wolffd@0: move on to more interesting questions.
wolffd@0: As Alfred North Whitehead said in 1911,
wolffd@0: "Civilization advances by extending the number of important operations
wolffd@0: that we can do without thinking about them."
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: Matlab is an interactive, matrix-oriented programming language that
wolffd@0: enables one to express one's (mathematical) ideas very concisely and directly,
wolffd@0: without having to worry about annoying details like memory allocation
wolffd@0: or type checking. This considerably reduces development time and
wolffd@0: keeps code short, readable and fully portable.
wolffd@0: Matlab has excellent built-in support for many data analysis and
wolffd@0: visualization routines. In addition, there are many useful toolboxes, e.g., for
wolffd@0: neural networks, signal and image processing.
wolffd@0: The main disadvantages of Matlab are that it can be slow (which is why
wolffd@0: we are currently rewriting parts of BNT in C), and that the commercial
wolffd@0: license is expensive (although the student version is only $100 in the US).
wolffd@0:
wolffd@0: Many people ask me why I did not use
wolffd@0: Octave,
wolffd@0: an open-source Matlab clone.
wolffd@0: The reason is that
wolffd@0: Octave does not support multi-dimensional arrays,
wolffd@0: cell arrays, objects, etc.
wolffd@0:
wolffd@0: Click here for a more detailed
wolffd@0: comparison of matlab and other languages.
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: I would like to thank numerous people for bug fixes, including:
wolffd@0: Rainer Deventer, Michael Robert James, Philippe Leray, Pedrito Maynard-Reid II, Andrew Ng,
wolffd@0: Ron Parr, Ilya Shpitser, Xuejing Sun, Ursula Sondhauss.
wolffd@0:
wolffd@0: I would like to thank the following people for contributing code:
wolffd@0: Pierpaolo Brutti, Ali Taylan Cemgil, Tamar Kushnir,
wolffd@0: Tom Murray,
wolffd@0: Nicholas Saunier,
wolffd@0: Ken Shan,
wolffd@0: Yair Weiss,
wolffd@0: Bob Welch,
wolffd@0: Ron Zohar.
wolffd@0:
wolffd@0: The following Intel employees have also contributed code:
wolffd@0: Qian Diao, Shan Huang, Yimin Zhang and especially Wei Hu.
wolffd@0:
wolffd@0:
wolffd@0: I would like to thank Stuart Russell for funding me over the years as
wolffd@0: I developed BNT, and Gary Bradksi for hiring me as an intern at Intel,
wolffd@0: which has supported much of the recent developments of BNT.
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0:
wolffd@0: 