Daniel@0: function net = mlp(nin, nhidden, nout, outfunc, prior, beta)
Daniel@0: %MLP	Create a 2-layer feedforward network.
Daniel@0: %
Daniel@0: %	Description
Daniel@0: %	NET = MLP(NIN, NHIDDEN, NOUT, FUNC) takes the number of inputs,
Daniel@0: %	hidden units and output units for a 2-layer feed-forward network,
Daniel@0: %	together with a string FUNC which specifies the output unit
Daniel@0: %	activation function, and returns a data structure NET. The weights
Daniel@0: %	are drawn from a zero mean, unit variance isotropic Gaussian, with
Daniel@0: %	varianced scaled by the fan-in of the hidden or output units as
Daniel@0: %	appropriate. This makes use of the Matlab function RANDN and so the
Daniel@0: %	seed for the random weight initialization can be  set using
Daniel@0: %	RANDN('STATE', S) where S is the seed value.  The hidden units use
Daniel@0: %	the TANH activation function.
Daniel@0: %
Daniel@0: %	The fields in NET are
Daniel@0: %	  type = 'mlp'
Daniel@0: %	  nin = number of inputs
Daniel@0: %	  nhidden = number of hidden units
Daniel@0: %	  nout = number of outputs
Daniel@0: %	  nwts = total number of weights and biases
Daniel@0: %	  actfn = string describing the output unit activation function:
Daniel@0: %	      'linear'
Daniel@0: %	      'logistic
Daniel@0: %	      'softmax'
Daniel@0: %	  w1 = first-layer weight matrix
Daniel@0: %	  b1 = first-layer bias vector
Daniel@0: %	  w2 = second-layer weight matrix
Daniel@0: %	  b2 = second-layer bias vector
Daniel@0: %	 Here W1 has dimensions NIN times NHIDDEN, B1 has dimensions 1 times
Daniel@0: %	NHIDDEN, W2 has dimensions NHIDDEN times NOUT, and B2 has dimensions
Daniel@0: %	1 times NOUT.
Daniel@0: %
Daniel@0: %	NET = MLP(NIN, NHIDDEN, NOUT, FUNC, PRIOR), in which PRIOR is a
Daniel@0: %	scalar, allows the field NET.ALPHA in the data structure NET to be
Daniel@0: %	set, corresponding to a zero-mean isotropic Gaussian prior with
Daniel@0: %	inverse variance with value PRIOR. Alternatively, PRIOR can consist
Daniel@0: %	of a data structure with fields ALPHA and INDEX, allowing individual
Daniel@0: %	Gaussian priors to be set over groups of weights in the network. Here
Daniel@0: %	ALPHA is a column vector in which each element corresponds to a
Daniel@0: %	separate group of weights, which need not be mutually exclusive.  The
Daniel@0: %	membership of the groups is defined by the matrix INDX in which the
Daniel@0: %	columns correspond to the elements of ALPHA. Each column has one
Daniel@0: %	element for each weight in the matrix, in the order defined by the
Daniel@0: %	function MLPPAK, and each element is 1 or 0 according to whether the
Daniel@0: %	weight is a member of the corresponding group or not. A utility
Daniel@0: %	function MLPPRIOR is provided to help in setting up the PRIOR data
Daniel@0: %	structure.
Daniel@0: %
Daniel@0: %	NET = MLP(NIN, NHIDDEN, NOUT, FUNC, PRIOR, BETA) also sets the
Daniel@0: %	additional field NET.BETA in the data structure NET, where beta
Daniel@0: %	corresponds to the inverse noise variance.
Daniel@0: %
Daniel@0: %	See also
Daniel@0: %	MLPPRIOR, MLPPAK, MLPUNPAK, MLPFWD, MLPERR, MLPBKP, MLPGRAD
Daniel@0: %
Daniel@0: 
Daniel@0: %	Copyright (c) Ian T Nabney (1996-2001)
Daniel@0: 
Daniel@0: net.type = 'mlp';
Daniel@0: net.nin = nin;
Daniel@0: net.nhidden = nhidden;
Daniel@0: net.nout = nout;
Daniel@0: net.nwts = (nin + 1)*nhidden + (nhidden + 1)*nout;
Daniel@0: 
Daniel@0: outfns = {'linear', 'logistic', 'softmax'};
Daniel@0: 
Daniel@0: if sum(strcmp(outfunc, outfns)) == 0
Daniel@0:   error('Undefined output function. Exiting.');
Daniel@0: else
Daniel@0:   net.outfn = outfunc;
Daniel@0: end
Daniel@0: 
Daniel@0: if nargin > 4
Daniel@0:   if isstruct(prior)
Daniel@0:     net.alpha = prior.alpha;
Daniel@0:     net.index = prior.index;
Daniel@0:   elseif size(prior) == [1 1]
Daniel@0:     net.alpha = prior;
Daniel@0:   else
Daniel@0:     error('prior must be a scalar or a structure');
Daniel@0:   end  
Daniel@0: end
Daniel@0: 
Daniel@0: net.w1 = randn(nin, nhidden)/sqrt(nin + 1);
Daniel@0: net.b1 = randn(1, nhidden)/sqrt(nin + 1);
Daniel@0: net.w2 = randn(nhidden, nout)/sqrt(nhidden + 1);
Daniel@0: net.b2 = randn(1, nout)/sqrt(nhidden + 1);
Daniel@0: 
Daniel@0: if nargin == 6
Daniel@0:   net.beta = beta;
Daniel@0: end