Daniel@0:
Daniel@0: Bayes Net Toolbox for Matlab
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: Bayes Net Toolbox for Matlab
Daniel@0: Written by Kevin Murphy, 1997--2002.
Daniel@0: Last updated: 19 October 2007.
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: - BNT supports many types of
Daniel@0: conditional probability distributions (nodes),
Daniel@0: and it is easy to add more.
Daniel@0:
Daniel@0: - Tabular (multinomial)
Daniel@0:
- Gaussian
Daniel@0:
- Softmax (logistic/ sigmoid)
Daniel@0:
- Multi-layer perceptron (neural network)
Daniel@0:
- Noisy-or
Daniel@0:
- Deterministic
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
- BNT supports decision and utility nodes, as well as chance
Daniel@0: nodes,
Daniel@0: i.e., influence diagrams as well as Bayes nets.
Daniel@0:
Daniel@0:
Daniel@0:
- BNT supports static and dynamic BNs (useful for modelling dynamical systems
Daniel@0: and sequence data).
Daniel@0:
Daniel@0:
Daniel@0:
- BNT supports many different inference algorithms,
Daniel@0: and it is easy to add more.
Daniel@0:
Daniel@0:
Daniel@0: - Exact inference for static BNs:
Daniel@0:
Daniel@0: - junction tree
Daniel@0:
- variable elimination
Daniel@0:
- brute force enumeration (for discrete nets)
Daniel@0:
- linear algebra (for Gaussian nets)
Daniel@0:
- Pearl's algorithm (for polytrees)
Daniel@0:
- quickscore (for QMR)
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
- Approximate inference for static BNs:
Daniel@0:
Daniel@0: - likelihood weighting
Daniel@0:
- Gibbs sampling
Daniel@0:
- loopy belief propagation
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
- Exact inference for DBNs:
Daniel@0:
Daniel@0: - junction tree
Daniel@0:
- frontier algorithm
Daniel@0:
- forwards-backwards (for HMMs)
Daniel@0:
- Kalman-RTS (for LDSs)
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
- Approximate inference for DBNs:
Daniel@0:
Daniel@0: - Boyen-Koller
Daniel@0:
- factored-frontier/loopy belief propagation
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: BNT supports several methods for parameter learning,
Daniel@0: and it is easy to add more.
Daniel@0:
Daniel@0:
Daniel@0: - Batch MLE/MAP parameter learning using EM.
Daniel@0: (Each node type has its own M method, e.g. softmax nodes use IRLS,
Daniel@0: and each inference engine has its own E method, so the code is fully modular.)
Daniel@0:
Daniel@0: - Sequential/batch Bayesian parameter learning (for fully observed tabular nodes only).
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: BNT supports several methods for regularization,
Daniel@0: and it is easy to add more.
Daniel@0:
Daniel@0: - Any node can have its parameters clamped (made non-adjustable).
Daniel@0:
- Any set of compatible nodes can have their parameters tied (c.f.,
Daniel@0: weight sharing in a neural net).
Daniel@0:
- Some node types (e.g., tabular) supports priors for MAP estimation.
Daniel@0:
- Gaussian covariance matrices can be declared full or diagonal, and can
Daniel@0: be tied across states of their discrete parents (if any).
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: BNT supports several methods for structure learning,
Daniel@0: and it is easy to add more.
Daniel@0:
Daniel@0:
Daniel@0: - Bayesian structure learning,
Daniel@0: using MCMC or local search (for fully observed tabular nodes only).
Daniel@0:
Daniel@0:
- Constraint-based structure learning (IC/PC and IC*/FCI).
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
- The source code is extensively documented, object-oriented, and free, making it
Daniel@0: an excellent tool for teaching, research and rapid prototyping.
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: It is trivial to implement all of
Daniel@0: the following probabilistic models using the toolbox.
Daniel@0:
Daniel@0: - Static
Daniel@0:
Daniel@0: - Linear regression, logistic regression, hierarchical mixtures of experts
Daniel@0:
Daniel@0:
- Naive Bayes classifiers, mixtures of Gaussians,
Daniel@0: sigmoid belief nets
Daniel@0:
Daniel@0:
- Factor analysis, probabilistic
Daniel@0: PCA, probabilistic ICA, mixtures of these models
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: - Dynamic
Daniel@0:
Daniel@0:
Daniel@0: - HMMs, Factorial HMMs, coupled HMMs, input-output HMMs, DBNs
Daniel@0:
Daniel@0:
- Kalman filters, ARMAX models, switching Kalman filters,
Daniel@0: tree-structured Kalman filters, multiscale AR models
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: - Many other combinations, for which there are (as yet) no names!
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: -
Daniel@0: I was hoping for a Linux-style effect, whereby people would contribute
Daniel@0: their own Matlab code so that the package would grow. With a few
Daniel@0: exceptions, this has not happened,
Daniel@0: although several people have provided bug-fixes (see the acknowledgements).
Daniel@0: Perhaps the Open
Daniel@0: Bayes Project will be more
Daniel@0: succesful in this regard, although the evidence to date is not promising.
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: Knowing that someone else might read your code forces one to
Daniel@0: document it properly, a good practice in any case, as anyone knows who
Daniel@0: has revisited old code.
Daniel@0: In addition, by having many "eye balls", it is easier to spot bugs.
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: I believe in the concept of
Daniel@0:
Daniel@0: reproducible research.
Daniel@0: Good science requires that other people be able
Daniel@0: to replicate your experiments.
Daniel@0: Often a paper does not give enough details about how exactly an
Daniel@0: algorithm was implemented (e.g., how were the parameters chosen? what
Daniel@0: initial conditions were used?), and these can make a big difference in
Daniel@0: practice.
Daniel@0: Hence one should release the code that
Daniel@0: was actually used to generate the results in one's paper.
Daniel@0: This also prevents re-inventing the wheel.
Daniel@0:
Daniel@0:
Daniel@0:
-
Daniel@0: I was fed up with reading papers where all people do is figure out how
Daniel@0: to do exact inference and/or learning
Daniel@0: in a model which is just a trivial special case of a general Bayes net, e.g.,
Daniel@0: input-output HMMs, coupled-HMMs, auto-regressive HMMs.
Daniel@0: My hope is that, by releasing general purpose software, the field can
Daniel@0: move on to more interesting questions.
Daniel@0: As Alfred North Whitehead said in 1911,
Daniel@0: "Civilization advances by extending the number of important operations
Daniel@0: that we can do without thinking about them."
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: Matlab is an interactive, matrix-oriented programming language that
Daniel@0: enables one to express one's (mathematical) ideas very concisely and directly,
Daniel@0: without having to worry about annoying details like memory allocation
Daniel@0: or type checking. This considerably reduces development time and
Daniel@0: keeps code short, readable and fully portable.
Daniel@0: Matlab has excellent built-in support for many data analysis and
Daniel@0: visualization routines. In addition, there are many useful toolboxes, e.g., for
Daniel@0: neural networks, signal and image processing.
Daniel@0: The main disadvantages of Matlab are that it can be slow (which is why
Daniel@0: we are currently rewriting parts of BNT in C), and that the commercial
Daniel@0: license is expensive (although the student version is only $100 in the US).
Daniel@0:
Daniel@0: Many people ask me why I did not use
Daniel@0: Octave,
Daniel@0: an open-source Matlab clone.
Daniel@0: The reason is that
Daniel@0: Octave does not support multi-dimensional arrays,
Daniel@0: cell arrays, objects, etc.
Daniel@0:
Daniel@0: Click here for a more detailed
Daniel@0: comparison of matlab and other languages.
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: I would like to thank numerous people for bug fixes, including:
Daniel@0: Rainer Deventer, Michael Robert James, Philippe Leray, Pedrito Maynard-Reid II, Andrew Ng,
Daniel@0: Ron Parr, Ilya Shpitser, Xuejing Sun, Ursula Sondhauss.
Daniel@0:
Daniel@0: I would like to thank the following people for contributing code:
Daniel@0: Pierpaolo Brutti, Ali Taylan Cemgil, Tamar Kushnir,
Daniel@0: Tom Murray,
Daniel@0: Nicholas Saunier,
Daniel@0: Ken Shan,
Daniel@0: Yair Weiss,
Daniel@0: Bob Welch,
Daniel@0: Ron Zohar.
Daniel@0:
Daniel@0: The following Intel employees have also contributed code:
Daniel@0: Qian Diao, Shan Huang, Yimin Zhang and especially Wei Hu.
Daniel@0:
Daniel@0:
Daniel@0: I would like to thank Stuart Russell for funding me over the years as
Daniel@0: I developed BNT, and Gary Bradksi for hiring me as an intern at Intel,
Daniel@0: which has supported much of the recent developments of BNT.
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0:
Daniel@0: 