|
Daniel@0
|
1 <html>
|
|
Daniel@0
|
2 <head>
|
|
Daniel@0
|
3 <title>
|
|
Daniel@0
|
4 Netlab Reference Manual olgd
|
|
Daniel@0
|
5 </title>
|
|
Daniel@0
|
6 </head>
|
|
Daniel@0
|
7 <body>
|
|
Daniel@0
|
8 <H1> olgd
|
|
Daniel@0
|
9 </H1>
|
|
Daniel@0
|
10 <h2>
|
|
Daniel@0
|
11 Purpose
|
|
Daniel@0
|
12 </h2>
|
|
Daniel@0
|
13 On-line gradient descent optimization.
|
|
Daniel@0
|
14
|
|
Daniel@0
|
15 <p><h2>
|
|
Daniel@0
|
16 Description
|
|
Daniel@0
|
17 </h2>
|
|
Daniel@0
|
18 <CODE>[net, options, errlog, pointlog] = olgd(net, options, x, t)</CODE> uses
|
|
Daniel@0
|
19 on-line gradient descent to find a local minimum of the error function for the
|
|
Daniel@0
|
20 network
|
|
Daniel@0
|
21 <CODE>net</CODE> computed on the input data <CODE>x</CODE> and target values
|
|
Daniel@0
|
22 <CODE>t</CODE>. A log of the error values
|
|
Daniel@0
|
23 after each cycle is (optionally) returned in <CODE>errlog</CODE>, and a log
|
|
Daniel@0
|
24 of the points visited is (optionally) returned in <CODE>pointlog</CODE>.
|
|
Daniel@0
|
25 Because the gradient is computed on-line (i.e. after each pattern)
|
|
Daniel@0
|
26 this can be quite inefficient in Matlab.
|
|
Daniel@0
|
27
|
|
Daniel@0
|
28 <p>The error function value at final weight vector is returned
|
|
Daniel@0
|
29 in <CODE>options(8)</CODE>.
|
|
Daniel@0
|
30
|
|
Daniel@0
|
31 <p>The optional parameters have the following interpretations.
|
|
Daniel@0
|
32
|
|
Daniel@0
|
33 <p><CODE>options(1)</CODE> is set to 1 to display error values; also logs error
|
|
Daniel@0
|
34 values in the return argument <CODE>errlog</CODE>, and the points visited
|
|
Daniel@0
|
35 in the return argument <CODE>pointslog</CODE>. If <CODE>options(1)</CODE> is set to 0,
|
|
Daniel@0
|
36 then only warning messages are displayed. If <CODE>options(1)</CODE> is -1,
|
|
Daniel@0
|
37 then nothing is displayed.
|
|
Daniel@0
|
38
|
|
Daniel@0
|
39 <p><CODE>options(2)</CODE> is the precision required for the value
|
|
Daniel@0
|
40 of <CODE>x</CODE> at the solution. If the absolute difference between
|
|
Daniel@0
|
41 the values of <CODE>x</CODE> between two successive steps is less than
|
|
Daniel@0
|
42 <CODE>options(2)</CODE>, then this condition is satisfied.
|
|
Daniel@0
|
43
|
|
Daniel@0
|
44 <p><CODE>options(3)</CODE> is the precision required of the objective
|
|
Daniel@0
|
45 function at the solution. If the absolute difference between the
|
|
Daniel@0
|
46 error functions between two successive steps is less than
|
|
Daniel@0
|
47 <CODE>options(3)</CODE>, then this condition is satisfied.
|
|
Daniel@0
|
48 Both this and the previous condition must be
|
|
Daniel@0
|
49 satisfied for termination. Note that testing the function value at each
|
|
Daniel@0
|
50 iteration roughly halves the speed of the algorithm.
|
|
Daniel@0
|
51
|
|
Daniel@0
|
52 <p><CODE>options(5)</CODE> determines whether the patterns are sampled randomly
|
|
Daniel@0
|
53 with replacement. If it is 0 (the default), then patterns are sampled
|
|
Daniel@0
|
54 in order.
|
|
Daniel@0
|
55
|
|
Daniel@0
|
56 <p><CODE>options(6)</CODE> determines if the learning rate decays. If it is 1
|
|
Daniel@0
|
57 then the learning rate decays at a rate of <CODE>1/t</CODE>. If it is 0
|
|
Daniel@0
|
58 (the default) then the learning rate is constant.
|
|
Daniel@0
|
59
|
|
Daniel@0
|
60 <p><CODE>options(9)</CODE> should be set to 1 to check the user defined gradient
|
|
Daniel@0
|
61 function.
|
|
Daniel@0
|
62
|
|
Daniel@0
|
63 <p><CODE>options(10)</CODE> returns the total number of function evaluations (including
|
|
Daniel@0
|
64 those in any line searches).
|
|
Daniel@0
|
65
|
|
Daniel@0
|
66 <p><CODE>options(11)</CODE> returns the total number of gradient evaluations.
|
|
Daniel@0
|
67
|
|
Daniel@0
|
68 <p><CODE>options(14)</CODE> is the maximum number of iterations (passes through
|
|
Daniel@0
|
69 the complete pattern set); default 100.
|
|
Daniel@0
|
70
|
|
Daniel@0
|
71 <p><CODE>options(17)</CODE> is the momentum; default 0.5.
|
|
Daniel@0
|
72
|
|
Daniel@0
|
73 <p><CODE>options(18)</CODE> is the learning rate; default 0.01.
|
|
Daniel@0
|
74
|
|
Daniel@0
|
75 <p><h2>
|
|
Daniel@0
|
76 Examples
|
|
Daniel@0
|
77 </h2>
|
|
Daniel@0
|
78 The following example performs on-line gradient descent on an MLP with
|
|
Daniel@0
|
79 random sampling from the pattern set.
|
|
Daniel@0
|
80 <PRE>
|
|
Daniel@0
|
81
|
|
Daniel@0
|
82 net = mlp(5, 3, 1, 'linear');
|
|
Daniel@0
|
83 options = foptions;
|
|
Daniel@0
|
84 options(18) = 0.01;
|
|
Daniel@0
|
85 options(5) = 1;
|
|
Daniel@0
|
86 net = olgd(net, options, x, t);
|
|
Daniel@0
|
87 </PRE>
|
|
Daniel@0
|
88
|
|
Daniel@0
|
89
|
|
Daniel@0
|
90 <p><h2>
|
|
Daniel@0
|
91 See Also
|
|
Daniel@0
|
92 </h2>
|
|
Daniel@0
|
93 <CODE><a href="graddesc.htm">graddesc</a></CODE><hr>
|
|
Daniel@0
|
94 <b>Pages:</b>
|
|
Daniel@0
|
95 <a href="index.htm">Index</a>
|
|
Daniel@0
|
96 <hr>
|
|
Daniel@0
|
97 <p>Copyright (c) Ian T Nabney (1996-9)
|
|
Daniel@0
|
98
|
|
Daniel@0
|
99
|
|
Daniel@0
|
100 </body>
|
|
Daniel@0
|
101 </html> |
