scilab-ann-0.4.2.4/0000755000175000017500000000000011441407762014556 5ustar sylvestresylvestrescilab-ann-0.4.2.4/readme.txt0000644000175000017500000000202411441407762016552 0ustar sylvestresylvestreANN Toolbox ver. 0.4.2.4 for Scilab 5.3 ======================================= This represents a toolbox for artificial neural networks, based on my developments described in "Matrix ANN" book, under development, if interested send me an email at r.hristev@phys.canterbury.ac.nz Current feature:s - Only layered feedforward networks are supported *directly* at the moment (for others use the "hooks" provided) - Unlimited number of layers - Unlimited number of neurons per each layer separately - User defined activation function (defaults to logistic) - User defined error function (defaults to SSE) - Algorithms implemented so far: * standard (vanilla) with or without bias, on-line or batch * momentum with or without bias, on-line or batch * SuperSAB with or without bias, on-line or batch * Conjugate gradients * Jacobian computation * Computation of result of multiplication between "vector" and Hessian - Some helper functions provided For full descriptions start with the toplevel "ANN" man page. scilab-ann-0.4.2.4/.gitignore0000644000175000017500000000025511441407762016550 0ustar sylvestresylvestre# # Generated files (common) # jar/ *.bin lib names loader.sce cleaner.sce # # Generated files (Windows) # *.bak master_help.xml scilab_*_help # # Generated files (Linux) # scilab-ann-0.4.2.4/license.txt0000644000175000017500000000171011441407762016740 0ustar sylvestresylvestreÿþ--------------------------------------------------------------------- COPYRIGHT (C) The programs and associated files in this toolkit are released under GNU Public Licence version 2 Copyright 1998, 2001 (C) Ryurick M. Hristev updated by Allan CORNET for Scilab 5.x (2008) This toolbox is normal part of "Matrix ANN" book (covered by the same license) but separate distribution is explicitly allowed. --------------------------------------------------------------------- scilab-ann-0.4.2.4/changelog.txt0000644000175000017500000000527511441407762017257 0ustar sylvestresylvestreChangeLog: ANN Toolbox --- for Scilab ========================== ===================================================================== From 0.4.2.3 -> 0.4.2.4 : - compatibility with Scilab 5.3.0 (Allan CORNET , DIGITEO , 2010) ===================================================================== From 0.4.2.2 -> 0.4.2.3 : - compatibility with Scilab 5.2.0 (Allan CORNET , DIGITEO , 2009) ===================================================================== From 0.4.2.1 -> 0.4.2.2 : - remove some hardcoded paths (Allan CORNET , DIGITEO , 2008) ===================================================================== From 0.4.2 -> 0.4.2.1 : - Adjustments for Scilab 5.x macros and help updated uses standard architecture of toolboxes for scilab 5.0 (Allan CORNET , INRIA , 2008) ===================================================================== From 0.4.1 -> 0.4.2 Minor adjustments for Scilab 2.6: - install.sh script which will install/uninstall the toolbox (system-wide or in selected dir) - manual updates (macros change, contents unchanged) From 0.4 -> 0.4.1 Minor adjustments for the new Scilab 2.5: - new Makefile - README updated ===================================================================== From 0.3 -> 0.4 Function names have been rationalized to a more suitable nomenclature. Discrete time loops are now performed inside training engines. Some algorithms added. WHAT YOU HAVE TO DO TO CONVERT YOUR OLD SCRIPTS: - rename the functions according to the new nomenclature. - some training engines now perform the discrete time loops inside, so they require a new parameter T and external looping have to be removed. Also note that the format of optional ex parameter have also changed in order to acomodate this. ===================================================================== From 0.2x -> 0.3 This toolkit now uses hypermatrices available only on Scilab 2.4 and upward. This will make possible to easily add future algorithms. Also: the patterns are now represented by column vectors and weight matrices are as one would expect, i.e. as they are currently used in main ANN literature. For these reasons scripts written for previous version(s) of this toolkit will not work on this one. I apologies for inconveniences. The introduction of hypermatrices in Scilab 2.4 will (PROBABLY) make unnecessary any other major changes in future versions of this toolkit (from the user's scripts point of view). WHAT YOU HAVE TO DO TO CONVERT YOUR OLD SCRIPTS: - transpose the inputs and targets, e.g. x=x' and nothing else if you don't touch the weight matrix; - if you manipulated the weight matrix then convert that part to the new format of W, see the man pages for details. scilab-ann-0.4.2.4/macros/0000755000175000017500000000000011441407762016042 5ustar sylvestresylvestrescilab-ann-0.4.2.4/macros/ann_FF_grad_BP_nb.sci0000644000175000017500000000453611441407762021736 0ustar sylvestresylvestrefunction grad_E = ann_FF_grad_BP_nb(x, t, N, W, c, af, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Calculate the error gradient considering all patterns // trough a backpropagation procedure // this function is designed for networks without bias // see ANN_FF (help) [lsh,rsh] = argn(0); // define default parameters if necessary if rsh < 5, c = 0, end; if rsh < 6, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 7, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x,'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); // initialize grad_E, W is a hypermatrix, grad_E have same layout grad_E = hypermat(size(W)'); // calculate grad_E // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1),l-1) * z(1:N(l-1), l-1); // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than c execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), c)'); // "deriv_af" for output layer execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; // adding contribution of pattern p // using the transposed of z vector here grad_E(1:N(L), 1:N(L-1), L-1) = ... grad_E(1:N(L), 1:N(L-1), L-1) + ... err_dz_deriv_af * z(1:N(L-1), L-1)'; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 1:N(l), l))'; // new "deriv_af" execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))'); // same as for layer "L", "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E(1:N(l), 1:N(l-1), l-1) = ... grad_E(1:N(l), 1:N(l-1), l-1) + ... err_dz_deriv_af * z(1:N(l-1), l-1)'; end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_grad.sci0000644000175000017500000000331511441407762020670 0ustar sylvestresylvestrefunction grad_E = ann_FF_grad(x,t,N,W,dW,af,ef) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Calculates the error gradient following a finite difference procedure, // i.e. perturbing each weight in turn; // used for --- testing --- purposes only as is much slower than BP algorithm. // The gradient is calculated only for all patterns in "x" and "t" // see ANN_FF (help) [lsh,rsh] = argn(0); // define optional parameters if necessary if rsh < 6, af = 'ann_log_activ', end; if rsh < 7, ef = 'ann_sum_of_sqr', end; // create the return matrix grad_E = hypermat(size(W)'); // rl - run between layers, parameter for ann_FF_run function rl = [2,size(N,'c')]; // for each pattern for p = 1 : size(x,'c') // for each layer for l = 2 : size(N,'c') // for each neuron in layer for n = 1 : N(l) // for each connection to previous layer for i = 1 : N(l-1)+1 // hold the old value of W temp = W(n,i,l-1); // change W value W(n,i,l-1) = temp - dW; // run the net y = ann_FF_run(x(:,p),N,W,rl,af); // calculate new error, to the "left" execstr('err_n = ' + ef + '(y,t(:,p))'); // change W value W(n,i,l-1) = temp + dW; // run the net y = ann_FF_run(x(:,p),N,W,rl,af); // calculate new error, to the "right" execstr('err_p = ' + ef + '(y,t(:,p))'); // "2" because \Delta w = 2 * dW grad_E(n,i,l-1) = ... grad_E(n,i,l-1) + (err_p - err_n) / (2 * dW); // restore W W(n,i,l-1) = temp; end; end; end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Hess.sci0000644000175000017500000000647211441407762020664 0ustar sylvestresylvestrefunction H = ann_FF_Hess(x, t, N, W, dW, dW2, af, ef) // This file is part of: // ANN Toolbox for Scilab // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public Licence version 2 // Calculates the Hessian // using a finite difference procedure by perturbing two weights // used for --- testing --- purposes only as is very slow // The Hessian is calculated considering the whole set of patterns // see ANN_FF (help) [lsh,rsh] = argn(0); // define default parameters if necessary if rsh < 7, af = 'ann_log_activ', end; if rsh < 8, ef = 'ann_sum_of_sqr', end; // no. of layers L = size(N,'c'); // rl run between layers, parameter for ann_FF_run function rl = [2, size(N,'c')]; // create the return hypermatrix, // layout is of type W .*. W (NOT W .*. W') H = hypermat([size(W)', size(W)']); // first weights are W(k1,i1,l1-1), second ones are W(k2,i2,l2-1) // for each layer // WARNING: THIS WILL NOT CALCULATE CORECTLY THE "DIAGONAL" ELEMENTS for l1 = 2 : L, for l2 = 2 : L // for each neuron in layer for k1 = 1 : N(l1), for k2 = 1 : N(l2) // for each connection from previous layer for i1 = 1 : N(l1-1)+1, for i2 = 1 : N(l2-1)+1 // hold original weight values temp1 = W(k1,i1,l1-1); temp2 = W(k2,i2,l2-1); // first change: ++ W(k1,i1,l1-1) = temp1 + dW; W(k2,i2,l2-1) = temp2 + dW; y = ann_FF_run(x,N,W,rl,af); execstr('err1 = ' + ef + '(y,t)'); // second change -+ W(k1,i1,l1-1) = temp1 - dW; W(k2,i2,l2-1) = temp2 + dW; y = ann_FF_run(x,N,W,rl,af); execstr('err2 = ' + ef + '(y,t)'); // third change +- W(k1,i1,l1-1) = temp1 + dW; W(k2,i2,l2-1) = temp2 - dW; y = ann_FF_run(x,N,W,rl,af); execstr('err3 = ' + ef + '(y,t)'); // fourth change -- W(k1,i1,l1-1) = temp1 - dW; W(k2,i2,l2-1) = temp2 - dW; y = ann_FF_run(x,N,W,rl,af); execstr('err4 = ' + ef + '(y,t)'); // restore weights W(k1,i1,l1-1) = temp1; W(k2,i2,l2-1) = temp2; // calculate hessian term // "4" factor because (\Delta W)^2 = (2 dW)^2 H(k1,i1,l1-1,k2,i2,l2-1) = ... (err1 - err2 - err3 + err4) / (4 * dW^2); end, end; end, end; end, end; // NOW THE DIAGONAL ELEMENTS // (avoid "if"-s above, it's too slow) for l = 2 : L // for each neuron in layer for k = 1 : N(l) // for each connection from previous layer for i = 1 : N(l-1)+1 // hold original weight values temp = W(k,i,l-1); // first change: + W(k,i,l-1) = temp + (1 + dW2) * dW; y = ann_FF_run(x,N,W,rl,af); execstr('err1 = ' + ef + '(y,t)'); // second change + W(k,i,l-1) = temp + (1 - dW2) * dW; y = ann_FF_run(x,N,W,rl,af); execstr('err2 = ' + ef + '(y,t)'); err_p = (err1 - err2) / (2 * dW2 * dW); // first change: - W(k,i,l-1) = temp - (1 - dW2) * dW; y = ann_FF_run(x,N,W,rl,af); execstr('err1 = ' + ef + '(y,t)'); // second change - W(k,i,l-1) = temp - (1 + dW2) * dW; y = ann_FF_run(x,N,W,rl,af); execstr('err2 = ' + ef + '(y,t)'); err_n = (err1 - err2) / (2 * dW2 * dW); // restore weight W(k,i,l-1) = temp; // calculate hessian term H(k,i,l-1,k,i,l-1) = ... (err_p - err_n) / (2 * dW); end; end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Std_online.sci0000644000175000017500000000176011441407762022053 0ustar sylvestresylvestrefunction W = ann_FF_Std_online(x, t, N, W, lp, T, af, ex, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on backpropagation algorithm. // see ANN_FF (help) // "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "af", "ex" and "err_deriv_y" if necessary if rsh < 7, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 8, ex = [" "," "], end; if rsh < 9, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of patterns P = size(x,'c'); // repeat T times for time = 1 : T // go trough all patterns, one at a time for p = 1 : P // find gradient grad_E = ann_FF_grad_BP(x(:,p), t(:,p), N, W, lp(2), af, err_deriv_y); // update weights W = W - lp(1) * grad_E; // go trough "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_run_nb.sci0000644000175000017500000000230211441407762021231 0ustar sylvestresylvestrefunction y = ann_FF_run_nb(x, N, W, l, af) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // runs the network, // with input pattern(s) "x" injected al layer "l(1)" // returning the activation at layer "l(2)" // (defaults to whole network) // this function is designed for networks without bias // see ANN_FF (help) // "l" and "af" are optional [lsh, rsh] = argn(0); // "l" defaults to whole network if rsh < 4, l = [2, size(N,'c')], end; // "af" defaults to logistic activation function if rsh < 5, af = 'ann_log_activ', end; // initialize "y" y = zeros(N(l(2)), size(x,'c')); // go trough all patterns for p = 1 : size(x,'c') // first "input" layer uses "x(:,p)" and calculate total input ... z = W(1:N(l(1)), 1:N(l(1)-1), l(1)-1) * x(:,p); // ... then activation execstr("z = " + af + "(z)"); // propagate, same as above but use "z" for ll = l(1)+1 : l(2) // use old "z" to find total input ... z = W(1:N(ll), 1:N(ll-1), ll-1) * z; // ... then activation execstr("z = " + af + "(z)"); end; // collect data y(:,p) = z; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Jacobian_BP.sci0000644000175000017500000000256211441407762022045 0ustar sylvestresylvestrefunction J = ann_FF_Jacobian_BP(x,N,W,af) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculate the Jacobian following a backpropagation procedure [lsh,rsh] = argn(0); // optional parameters if rsh < 4, af = ["ann_log_activ", "ann_d_log_activ"], end; // no. of layers L = size(N,'c'); // ... and patterns P = size(x,'c'); // create the hypermatrix to hold (grad_{a(\ell)}} z^\T)^\T grad_a_z = hypermat([N(L), max(N(2:L)), L-1]); // the matrix containing the activities d_f = zeros(max(N(2:L)), L-1); // initialize J J = hypermat([N(L),N(1),P]); // for all patterns for p = 1 : P // forward propagation // initial activation z = x(:,p); for l = 1 : L-1 // find next activation, use extended z, i.e. bias execstr('z = ' + af(1) + '(W(1:N(l+1), 1:N(l)+1, l) * [1;z]);'); // and store its derivative execstr('d_f(1:N(l+1),l) = ' + af(2) + '(z)'); end; // backpropagation // initial values grad_a_z(:, 1:N(L), L-1) = diag(d_f(1:N(L),L-1)); for l = L-2 : -1 : 1 grad_a_z(:, 1:N(l+1), l) = ... (grad_a_z(:, 1:N(l+2), l+1) * ... W(1:N(l+2), 2:N(l+1)+1, l+1)) .* ... (ones(N(L),1) * d_f(1:N(l+1),l)') end; J(:,:,p) = grad_a_z(:, 1:N(2),1) * W(1:N(2), 2:N(1)+1, 1); end; endfunction scilab-ann-0.4.2.4/macros/buildmacros.sce0000644000175000017500000000042611441407762021044 0ustar sylvestresylvestre// ==================================================================== // Allan CORNET // DIGITEO 2010 // INRIA 2008 // ANN Toolbox // ==================================================================== tbx_build_macros(TOOLBOX_NAME,get_absolute_file_path("buildmacros.sce"));scilab-ann-0.4.2.4/macros/ann_FF_SSAB_batch.sci0000644000175000017500000000322111441407762021640 0ustar sylvestresylvestrefunction [W,Delta_W_old,Delta_W_oldold,mu]=ann_FF_SSAB_batch(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on backpropagation with SuperSAB algorithm (batch version). // see ANN_FF (help) // "mu", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // size of W hypermatrix, required in several places size_W = size(W)'; // define "mu", "af", "ex" and "err_deriv_y" if necessary if rsh < 9, mu = lp(1) * hypermat(size_W,ones(prod(size_W),1)), end; if rsh < 10, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 11, ex = " ", end; if rsh < 12, err_deriv_y = 'ann_d_sum_of_sqr', end; // repeat T times for time = 1 : T // error gradient grad_E = ann_FF_grad_BP(x,t,N,W,lp(2),af,err_deriv_y); // sign hypermatrix M = sign(sign(Delta_W_old .* Delta_W_oldold) ... + hypermat(size_W, ones(prod(size_W),1))); // mu hypermatrix update (former lp(1)) mu = ( (lp(4) - lp(5)) * M ... + lp(5) * hypermat(size_W, ones(prod(size_W),1)) ) .* mu; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) // same for Delta_W_oldold Delta_W_oldold = Delta_W_old; Delta_W_old = ... - mu .* grad_E ... - (lp(3) * Delta_W_old) ... .* (hypermat(size_W, ones(prod(size_W),1)) - M); W = W + Delta_W_old; // execute "ex" execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Std_online_nb.sci0000644000175000017500000000203211441407762022523 0ustar sylvestresylvestrefunction W = ann_FF_Std_online_nb(x, t, N, W, lp, T, af, ex, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on backpropagation algorithm. // this function is designed for networks without bias // see ANN_FF (help) // "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "af", "ex" and "err_deriv_y" if necessary if rsh < 7, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 8, ex = [" "," "], end; if rsh < 9, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of patterns P = size(x,'c'); // repeat T times for time = 1 : T // go trough all patterns, one at a time for p = 1 : P // find gradient grad_E = ann_FF_grad_BP_nb(x(:,p), t(:,p), N, W, lp(2), af, err_deriv_y); // update weights W = W - lp(1) * grad_E; // execute "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/ann_d_log_activ.sci0000644000175000017500000000056611441407762021657 0ustar sylvestresylvestrefunction z = ann_d_log_activ(y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates the derivative of logistic activation function, // given the actual value of the function // see ANN_GEN (help) z = y .* (1 - y); endfunctionscilab-ann-0.4.2.4/macros/ann_FF_SSAB_online.sci0000644000175000017500000000345011441407762022047 0ustar sylvestresylvestrefunction [W,Delta_W_old,Delta_W_oldold,mu]=ann_FF_SSAB_online(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on backpropagation with SuperSAB algorithm. // see ANN_FF (help) // "mu", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // size of W hypermatrix, required in several places size_W = size(W)'; // define "mu", "af", "ex" and "err_deriv_y" if necessary if rsh < 9, mu = lp(1) * hypermat(size_W,ones(prod(size_W),1)), end; if rsh < 10, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 11, ex = [" "," "], end; if rsh < 12, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of patterns P = size(x,'c'); // repeat T times for time = 1 : T // go trough all patterns for p = 1 : P // error gradient grad_E = ann_FF_grad_BP(x(:,p),t(:,p),N,W,lp(2),af,err_deriv_y); // sign hypermatrix M = sign(sign(Delta_W_old .* Delta_W_oldold) ... + hypermat(size_W, ones(prod(size_W),1))); // mu hypermatrix update (former lp(1)) mu = ( (lp(4) - lp(5)) * M ... + lp(5) * hypermat(size_W, ones(prod(size_W),1)) ) .* mu; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) // same for Delta_W_oldold Delta_W_oldold = Delta_W_old; Delta_W_old = ... - mu .* grad_E ... - (lp(3) * Delta_W_old) ... .* (hypermat(size_W, ones(prod(size_W),1)) - M); W = W + Delta_W_old; // execute "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Mom_online.sci0000644000175000017500000000611211441407762022045 0ustar sylvestresylvestrefunction [W,Delta_W_old]=ann_FF_Mom_online(x,t,N,W,lp,T,Delta_W_old,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on backpropagation algorithm with momentum. // see ANN_FF (help) // "Delta_W_old", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "Delta_W_old", "af", "ex" and "err_deriv_y" if necessary if rsh < 7, Delta_W_old = hypermat(size(W)'), end; if rsh < 8, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 9, ex = [" "," "], end; if rsh < 10, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x, 'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); // grad_E_mod is a hypermatrix with the same layout as W // because of the flat spot elimination grad_E is calculated here grad_E_mod = hypermat(size(W)'); // repeat T times for time = 1 : T // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // adding "1" to "z(1:N(l-1),l-1)" to represent bias // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1)+1, l-1) ... * [1; z(1:N(l-1),l-1)]; // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than lp(2) execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), lp(2))'); // "deriv_af" for output layer, also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))' + ... ' + lp(4) * ones(N(L),1)'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; // using the transposed of extended z vector here grad_E_mod(1:N(L), 1:N(L-1)+1, L-1) = ... err_dz_deriv_af * [1, z(1:N(L-1), L-1)']; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 2:N(l)+1, l))'; // new "deriv_af" execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))' + ... ' + lp(4) * ones(N(l),1)'); // same as for layer "L", "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(l), 1:N(l-1)+1, l-1) = ... err_dz_deriv_af * [1, z(1:N(l-1), l-1)']; end; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e. on next loop or next call to this function) Delta_W_old = -lp(1) * grad_E_mod + lp(3) * Delta_W_old; W = W + Delta_W_old; // go trough "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Std_batch.sci0000644000175000017500000000156011441407762021646 0ustar sylvestresylvestrefunction W = ann_FF_Std_batch(x, t, N, W, lp, T, af, ex, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on standard backpropagation algorithm (batch version). // see ANN_FF (help) // "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "af", "ex" and "err_deriv_y" if necessary if rsh < 7, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 8, ex = " ", end; if rsh < 9, err_deriv_y = 'ann_d_sum_of_sqr', end; // repeat T times for time = 1 : T // find gradient grad_E = ann_FF_grad_BP(x, t, N, W, lp(2), af, err_deriv_y); // update weights W = W - lp(1) * grad_E; // go trough "ex" execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_sum_of_sqr.sci0000644000175000017500000000052611441407762021556 0ustar sylvestresylvestrefunction E = ann_sum_of_sqr(y,t) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates sum-of-squares error between "y" and "t" patterns // see ANN_GEN (help) E = sum((y-t) .^ 2) / 2; endfunction scilab-ann-0.4.2.4/macros/ann_FF_run.sci0000644000175000017500000000240011441407762020551 0ustar sylvestresylvestrefunction y = ann_FF_run(x, N, W, l, af) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // runs the network, with biases, // with input pattern(s) "x" injected al layer "l(1)" // returning the activation at layer "l(2)" // (defaults to whole network) // see ANN_FF (help) // "l" and "af" are optional [lsh, rsh] = argn(0); // "l" defaults to whole network if rsh < 4, l = [2,size(N,'c')], end; // "af" defaults to logistic activation function if rsh < 5, af = 'ann_log_activ', end; // no. of present patterns P = size(x,'c'); // initialize "y" y = zeros(N(l(2)), P); // go trough all patterns for p = 1 : P // first "input" layer uses "x(:,p)" and calculate total input ... // (an "1" is added to the input vector to represent bias) z = W(1:N(l(1)), 1:N(l(1)-1)+1, l(1)-1) * [1; x(:,p)]; // ... then activation execstr("z = " + af + "(z)"); // propagate, same as above but use "z" for ll = l(1)+1 : l(2) // ... use old "z" to find total input z = W(1:N(ll), 1:N(ll-1)+1, ll-1) * [1; z]; // ... then compute activation execstr("z = " + af + "(z)"); end; // collect data y(:,p) = z; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Jacobian.sci0000644000175000017500000000145011441407762021457 0ustar sylvestresylvestrefunction J = ann_FF_Jacobian(x,N,W,dx,af) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates the Jacobian using a finite differences procedure [lsh,rsh] = argn(0); // optional parameters if rsh < 5, af = "ann_log_activ", end; // required for ann_FF_run l = [2,size(N,'c')]; // no. of patterns P = size(x,'c'); // initialize J J = hypermat([N(size(N,'c')), N(1), P]); // for each pattern for p = 1 : P // for each input for i = 1 : N(1) temp = x(i,p); x(i,p) = temp + dx; y_p = ann_FF_run(x(:,p), N, W, l, af); x(i,p) = temp - dx; y_n = ann_FF_run(x(:,p), N, W, l, af); J(:,i,p) = (y_p - y_n) / (2 * dx); end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_ConjugGrad.sci0000644000175000017500000000457711441407762022011 0ustar sylvestresylvestrefunction W = ann_FF_ConjugGrad(x, t, N, W, T, dW, ex, af, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // trains the network for T epochs using the Conjugate gradient algorithm // see ANN_FF (help) [lsh,rsh] = argn(0); // deal with default parameters if rsh < 7, ex = [" "], end; if rsh < 8, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 9, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N,'c'); //------------------------------------------------------------------- // first step for conjugate gradients is performed outside the loop, // for proper initialization // calculate first grad_E and initialize directly to grad_E_old grad_E = ann_FF_grad_BP(x, t, N, W, 0, af, err_deriv_y); // initialize direction to grad_E D = - grad_E; // iterate to T-1 (for T we need fewer calculations) for time = 1 : T-1 // calculate D^\T \circ Hessian D_circ_H = ann_FF_VHess(x, t, N, W, D, dW, af, err_deriv_y); // calculate D^\T \circ grad_E and D^\T \circ Hessian \circ D ... D_circ_grad_E = 0; D_circ_H_circ_D = 0; for l = 1 : L-1 // using "old" grad_E D_circ_grad_E = D_circ_grad_E + sum(D(:,:,l) .* grad_E(:,:,l)); D_circ_H_circ_D = D_circ_H_circ_D + sum(D_circ_H(:,:,l) .* D(:,:,l)); end; // ... and alpha alpha = - D_circ_grad_E / D_circ_H_circ_D; // new weights W = W + alpha * D; // execute ex if necessary execstr(ex); // new gradient grad_E = ann_FF_grad_BP(x, t, N, W, 0, af, err_deriv_y); // calculate D^\T \circ grad_E (new) ... D_circ_grad_E = 0; for l = 1 : L-1 D_circ_grad_E = D_circ_grad_E + sum(D(:,:,l) .* grad_E(:,:,l)); end; // ... and beta beta = - D_circ_grad_E / D_circ_H_circ_D; // new direction D = - grad_E + beta * D; end; // for T only // calculate D^\T \circ Hessian D_circ_H = ann_FF_VHess(x, t, N, W, D, dW, af, err_deriv_y); // calculate D^\T \circ grad_E and D^\T \circ Hessian \circ D ... D_circ_grad_E = 0; D_circ_H_circ_D = 0; for l = 1 : L-1 // using "old" grad_E D_circ_grad_E = D_circ_grad_E + sum(D(:,:,l) .* grad_E(:,:,l)); D_circ_H_circ_D = D_circ_H_circ_D + sum(D_circ_H(:,:,l) .* D(:,:,l)); end; // ... and alpha alpha = - D_circ_grad_E / D_circ_H_circ_D; // final weights W = W + alpha * D; // and final ex execstr(ex); endfunction scilab-ann-0.4.2.4/macros/ann_FF_SSAB_batch_nb.sci0000644000175000017500000000324011441407762022320 0ustar sylvestresylvestrefunction [W,Delta_W_old,Delta_W_oldold,mu]=ann_FF_SSAB_batch_nb(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on backpropagation with SuperSAB algorithm (batch version). // this function is to be used on networks without bias // see ANN_FF (help) // "mu", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // size of W hypermatrix, required in several places size_W = size(W)'; // define default parameters if necessary if rsh < 9, mu = lp(1) * hypermat(size_W,ones(prod(size_W),1)), end; if rsh < 10, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 11, ex = " ", end; if rsh < 12, err_deriv_y = 'ann_d_sum_of_sqr', end; // repeat T times for time = 1 : T // error gradient grad_E = ann_FF_grad_BP_nb(x,t,N,W,lp(2),af,err_deriv_y); // sign hypermatrix M = sign(sign(Delta_W_old .* Delta_W_oldold) ... + hypermat(size_W,ones(prod(size_W),1))); // mu hypermatrix update (former lp(1)) mu = ( (lp(4) - lp(5)) * M ... + lp(5) * hypermat(size_W,ones(prod(size_W),1)) ) .* mu; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) // same for Delta_W_oldold Delta_W_oldold = Delta_W_old; Delta_W_old = ... - mu .* grad_E ... - (lp(3) * Delta_W_old) .* (hypermat(size_W,ones(prod(size_W),1)) - M); W = W + Delta_W_old; // execute "ex" execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Mom_batch_nb.sci0000644000175000017500000000615111441407762022324 0ustar sylvestresylvestrefunction [W,Delta_W_old] = ann_FF_Mom_batch_nb(x,t,N,W,lp,T,Delta_W_old,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on backpropagation with momentum algorithm. // this function is to be used on networks without bias // see ANN_FF (help) // "Delta_W_old", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "Delta_w_old", "af", "ex" and "err_deriv_y" if necessary if rsh < 7, Delta_W_old = hypermat(size(W)'), end; if rsh < 8, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 9, ex = " ", end; if rsh < 10, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x,'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); size_W = size(W)'; // repeat T times for time = 1 : T // grad_E_mod is a hypermatrix with the same layout as W // because of flat spot elimination the modified grad_E is calculated here // reinitialize at each loop grad_E_mod = hypermat(size_W); // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1), l-1) * z(1:N(l-1), l-1); // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than lp(2) execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), lp(2))'); // "deriv_af" for output layer, also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))' + ... ' + lp(4) * ones(N(L),1)'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(L), 1:N(L-1), L-1) = ... grad_E_mod(1:N(L), 1:N(L-1), L-1) + ... err_dz_deriv_af * z(1:N(L-1), L-1)'; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 1:N(l), l))'; // new "deriv_af", also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))' + ... ' + lp(4) * ones(N(l),1)'); // same as for layer L, "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(l), 1:N(l-1), l-1) = ... grad_E_mod(1:N(l), 1:N(l-1), l-1) + ... err_dz_deriv_af * z(1:N(l-1),l-1)'; end; end; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) Delta_W_old = - lp(1) * grad_E_mod + lp(3) * Delta_W_old; W = W + Delta_W_old; // execute "ex" execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_pat_shuffle.sci0000644000175000017500000000116311441407762021677 0ustar sylvestresylvestrefunction [x,t] = ann_pat_shuffle(x,t) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // shuffles the patterns from "x" and the corresponding "t" // see ANN_GEN (help) // no. of patterns P = size(x,'c'); my_rand = ceil(P * rand(P,1)); for p = 1 : P // shuffle x temp = x(:,my_rand(p)); x(:,my_rand(p)) = x(:,p); x(:,p) = temp; // shuffle t same way (keep x(:,p) <-> t(:,p) correspondence) temp = t(:,my_rand(p)); t(:,my_rand(p)) = t(:,p); t(:,p) = temp; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_init.sci0000644000175000017500000000220611441407762020714 0ustar sylvestresylvestrefunction W = ann_FF_init(N, r, rb) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // generate the weight matrix for an feedforward ANN defined by N // see ANN_FF (help) // "r" and "rb" are optional arguments [lsh, rsh] = argn(0); // define "r" if necessary if rsh < 2, r = [-1,1], end; // "+1" -- to alow room for biases (first column in each W) // don't create weight entries for input neurons (from input layer 1) // i.e. no. of matrices W(:,:,*) is size(N,'c')-1 W = hypermat([max(N), max(N)+1, size(N,'c') - 1]); // initialize weights with random numbers between "r(1)" and "r(2)" // (only the required values, first column, i.e. bias, later) for l = 2 : size(N,'c') W(1:N(l), 2:N(l-1)+1, l-1) = ... (r(2) - r(1)) * rand(N(l), N(l-1)) + r(1) * ones(N(l), N(l-1)); end; // biases, if required, otherwise leave them 0 if rsh > 2 ... then for l = 2 : size(N,'c') W(1:N(l), 1, l-1) = ... (rb(2) - rb(1)) * rand(N(l), 1) + rb(1) * ones(N(l), 1); end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_VHess.sci0000644000175000017500000000143411441407762021003 0ustar sylvestresylvestrefunction VH = ann_FF_VHess(x, t, N, W, V, dW, af, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates the result of multiplication between a vector and Hessian // trough a finite differences procedure [lsh,rsh] = argn(0); // define default parameters if necessary if rsh < 7, af = ['ann_log_activ', 'ann_d_log_activ'], end; if rsh < 8, err_deriv_y = 'ann_d_sum_of_sqr', end; // calculate gradient to the + grad_p = ann_FF_grad_BP(x, t, N, W + dW * V, 0, af, err_deriv_y); // ... and to the - grad_n = ann_FF_grad_BP(x, t, N, W - dW * V, 0, af, err_deriv_y); // result, difference is 2 * dW VH = (grad_p - grad_n) / (2 * dW); endfunction scilab-ann-0.4.2.4/macros/ann_FF_Mom_batch.sci0000644000175000017500000000626511441407762021653 0ustar sylvestresylvestrefunction [W,Delta_W_old]=ann_FF_Mom_batch(x,t,N,W,lp,T,Delta_W_old,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, including biases // based on backpropagation algorithm with momentum (batch version). // see ANN_FF (help) // "Delta_W_old", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "Delta_W_old", "af", "ex" and "err_deriv_y" if necessary if rsh < 7, Delta_W_old = hypermat(size(W)'), end; if rsh < 8, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 9, ex = " ", end; if rsh < 10, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x, 'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); size_W = size(W)'; // repeat T times for time = 1 : T // grad_E_mod is a hypermatrix with the same layout as W // because of the flat spot elimination grad_E is calculated here // reinitialize at each loop grad_E_mod = hypermat(size_W); // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // adding "1" to "z(1:N(l-1),l-1)" to represent bias // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1)+1, l-1) ... * [1; z(1:N(l-1),l-1)]; // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than lp(2) execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), lp(2))'); // "deriv_af" for output layer, also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))' + ... ' + lp(4) * ones(N(L),1)'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; // using the transposed of extended z vector here grad_E_mod(1:N(L), 1:N(L-1)+1, L-1) = ... grad_E_mod(1:N(L), 1:N(L-1)+1, L-1) + ... err_dz_deriv_af * [1, z(1:N(L-1), L-1)']; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 2:N(l)+1, l))'; // new "deriv_af" execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))' + ... ' + lp(4) * ones(N(l),1)'); // same as for layer "L", "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(l), 1:N(l-1)+1, l-1) = ... grad_E_mod(1:N(l), 1:N(l-1)+1, l-1) + ... err_dz_deriv_af * [1, z(1:N(l-1), l-1)']; end; end; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e. on next loop or next call to this function) Delta_W_old = -lp(1) * grad_E_mod + lp(3) * Delta_W_old; W = W + Delta_W_old; execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_Mom_online_nb.sci0000644000175000017500000000576511441407762022541 0ustar sylvestresylvestrefunction [W,Delta_W_old] = ann_FF_Mom_online_nb(x,t,N,W,lp,T,Delta_W_old,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on backpropagation with momentum algorithm. // this function is to be used on networks without bias // see ANN_FF (help) // "Delta_W_old", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "Delta_w_old", "af", "ex" and "err_deriv_y" if necessary if rsh < 7, Delta_W_old = hypermat(size(W)'), end; if rsh < 8, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 9, ex = [" "," "], end; if rsh < 10, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x,'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); // grad_E_mod is a hypermatrix with the same layout as W // because of flat spot elimination the modified grad_E is calculated here grad_E_mod = hypermat(size(W)'); // repeat T times for time = 1 : T // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1), l-1) * z(1:N(l-1), l-1); // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than lp(2) execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), lp(2))'); // "deriv_af" for output layer, also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))' + ... ' + lp(4) * ones(N(L),1)'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(L), 1:N(L-1), L-1) ... = err_dz_deriv_af * z(1:N(L-1), L-1)'; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 1:N(l), l))'; // new "deriv_af", also add flat spot elimination execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))' + ... ' + lp(4) * ones(N(l),1)'); // same as for layer L, "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E_mod(1:N(l), 1:N(l-1), l-1) ... = err_dz_deriv_af * z(1:N(l-1),l-1)'; end; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) Delta_W_old = - lp(1) * grad_E_mod + lp(3) * Delta_W_old; W = W + Delta_W_old; // execute "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/cleanmacros.sce0000644000175000017500000000075611441407762021035 0ustar sylvestresylvestre// ==================================================================== // Allan CORNET // DIGITEO 2009 // This file is released into the public domain // ==================================================================== libpath = get_absolute_file_path('cleanmacros.sce'); binfiles = ls(libpath+'/*.bin'); for i = 1:size(binfiles,'*') mdelete(binfiles(i)); end mdelete(libpath+'/names'); mdelete(libpath+'/lib'); // ==================================================================== scilab-ann-0.4.2.4/macros/ann_FF_Std_batch_nb.sci0000644000175000017500000000163211441407762022325 0ustar sylvestresylvestrefunction W = ann_FF_Std_batch_nb(x, t, N, W, lp, T, af, ex, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on standard backpropagation algorithm (batch version). // this function is designed for networks without bias // see ANN_FF (help) // "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // define "af", "ex" and "err_deriv_y" if necessary if rsh < 7, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 8, ex = " ", end; if rsh < 9, err_deriv_y = 'ann_d_sum_of_sqr', end; // repeat T times for time = 1 : T // find gradient grad_E = ann_FF_grad_BP_nb(x, t, N, W, lp(2), af, err_deriv_y); // update weights W = W - lp(1) * grad_E; // execute "ex" execstr(ex); end; endfunction scilab-ann-0.4.2.4/macros/ann_d_sum_of_sqr.sci0000644000175000017500000000050711441407762022060 0ustar sylvestresylvestrefunction err_d = ann_d_sum_of_sqr(y,t) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates the derivative of sum-of-squares error // see ANN_GEN (help) err_d = y - t; endfunction scilab-ann-0.4.2.4/macros/ann_FF_grad_BP.sci0000644000175000017500000000463411441407762021256 0ustar sylvestresylvestrefunction grad_E = ann_FF_grad_BP(x, t, N, W, c, af, err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Calculate the error gradient considering all patterns // trough a backpropagation procedure // see ANN_FF (help) [lsh,rsh] = argn(0); // define default parameters if necessary if rsh < 5, c = 0, end; if rsh < 6, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 7, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of layers L = size(N, 'c'); // ... and patterns P = size(x,'c'); // initialize "z" to avoid resizing z = zeros(max(N), L); // initialize grad_E, W is a hypermatrix, grad_E have same layout grad_E = hypermat(size(W)'); // calculate grad_E // go trough all patterns for p = 1 : P // find all neuronal outputs (activation) for current input pattern // first "z" column is exactly "x(:,p)" z(1:N(1),1) = x(:,p); for l = 2 : L // adding "1" to "z(1:N(l-1), l-1)" to represent bias // first calculate total input (as column vector) ... z(1:N(l),l) = W(1:N(l), 1:N(l-1)+1,l-1) ... * [1; z(1:N(l-1), l-1)]; // ... then activation execstr('z(1:N(l),l) = ' + af(1) + '(z(1:N(l),l))'); end; // now for layer "L" (last), requiring special treatment on "err_dz" // "err_dz" for output layer, don't propagate smaller than lp(2) execstr('err_dz = clean(' + err_deriv_y + '(z(1:N(L),L),t(:,p)), c)'); // "deriv_af" for output layer execstr('deriv_af = ' + af(2) + '(z(1:N(L),L))'); // "err_dz_deriv_af" product is used twice err_dz_deriv_af = err_dz .* deriv_af; // adding contribution of pattern p // using the transposed of extended z vector here grad_E(1:N(L), 1:N(L-1)+1, L-1) = ... grad_E(1:N(L), 1:N(L-1)+1, L-1) + ... err_dz_deriv_af * [1, z(1:N(L-1), L-1)']; // backpropagate for l = L-1 : -1 : 2 // new "err_dz" based on previous one // transpose two vectors instead of W err_dz = (err_dz_deriv_af' * W(1:N(l+1), 2:N(l)+1, l))'; // new "deriv_af" execstr('deriv_af = ' + af(2) + '(z(1:N(l),l))'); // same as for layer "L", "err_dz_deriv_af" also used on next loop above err_dz_deriv_af = err_dz .* deriv_af; grad_E(1:N(l), 1:N(l-1)+1, l-1) = ... grad_E(1:N(l), 1:N(l-1)+1, l-1) + ... err_dz_deriv_af * [1, z(1:N(l-1), l-1)']; end; end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_init_nb.sci0000644000175000017500000000154711441407762021402 0ustar sylvestresylvestrefunction W = ann_FF_init_nb(N, r) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // generate the weight matrix for an feedforward ANN defined by N // this function is designed for networks without bias // see ANN_FF (help) // "r" is optional argument [lsh, rsh] = argn(0); // define "r" if necessary if rsh < 2, r = [-1,1], end; // don't create weight entries for input neurons (from layer 0), // i.e. no. of matrices W(:,:,*) is size(N,'c')-1 W = hypermat([max(N), max(N), size(N,'c') - 1]); // initialize (only the required values) // with random numbers between "r(1)" and "r(2)" for l = 2 : size(N,'c') W(1:N(l), 1:N(l-1), l-1) = ... (r(2) - r(1)) * rand(N(l), N(l-1)) + r(1) * ones(N(l), N(l-1)); end; endfunction scilab-ann-0.4.2.4/macros/ann_FF_SSAB_online_nb.sci0000644000175000017500000000350311441407762022525 0ustar sylvestresylvestrefunction [W,Delta_W_old,Delta_W_oldold,mu]=ann_FF_SSAB_online_nb(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu,af,ex,err_deriv_y) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Updates weight matrix of an ANN, // based on backpropagation with SuperSAB algorithm. // this function is to be used on networks without bias // see ANN_FF (help) // "mu", "af", "ex" and "err_deriv_y" are optional arguments [lsh, rsh] = argn(0); // size of W hypermatrix, required in several places size_W = size(W)'; // define default parameters if necessary if rsh < 9, mu = lp(1) * hypermat(size_W,ones(prod(size_W),1)), end; if rsh < 10, af = ['ann_log_activ','ann_d_log_activ'], end; if rsh < 11, ex = [" "," "], end; if rsh < 12, err_deriv_y = 'ann_d_sum_of_sqr', end; // no. of patterns P = size(x,'c'); // repeat T times for time = 1 : T // go trough all patterns, one at a time for p = 1 : P // error gradient grad_E = ann_FF_grad_BP_nb(x(:,p),t(:,p),N,W,lp(2),af,err_deriv_y); // sign hypermatrix M = sign(sign(Delta_W_old .* Delta_W_oldold) ... + hypermat(size_W,ones(prod(size_W),1))); // mu hypermatrix update (former lp(1)) mu = ( (lp(4) - lp(5)) * M ... + lp(5) * hypermat(size_W,ones(prod(size_W),1)) ) .* mu; // update weights // (the new Delta_W_old ! ;) will become old after weight update, // i.e on next loop or next call to this function) // same for Delta_W_oldold Delta_W_oldold = Delta_W_old; Delta_W_old = ... - mu .* grad_E ... - (lp(3) * Delta_W_old) .* (hypermat(size_W,ones(prod(size_W),1)) - M); W = W + Delta_W_old; // execute "ex" execstr(ex(1)); end; execstr(ex(2)); end; endfunction scilab-ann-0.4.2.4/macros/ann_log_activ.sci0000644000175000017500000000052511441407762021347 0ustar sylvestresylvestrefunction y = ann_log_activ(x) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // calculates logistic activation function for each component of "x" // see ANN_GEN (help) y = 1 ./ (1+%e^(-x)); endfunction scilab-ann-0.4.2.4/macros/ann_FF_grad_nb.sci0000644000175000017500000000341311441407762021346 0ustar sylvestresylvestrefunction grad_E = ann_FF_grad_nb(x,t,N,W,dW,af,ef) // This file is part of: // ANN Toolbox for Scilab 5.x // Copyright (C) Ryurick M. Hristev // updated by Allan CORNET INRIA, May 2008 // released under GNU Public licence version 2 // Calculates the error gradient following a finite difference procedure, // i.e. perturbing each weight in turn; // used for --- testing --- purposes only as is much slower than BP algorithm. // this function is designed for networks without bias // The gradient is calculated only for all patterns in "x" and "t" // see ANN_FF (help) [lsh,rsh] = argn(0); // define optional parameters if necessary if rsh < 6, af = 'ann_log_activ', end; if rsh < 7, ef = 'ann_sum_of_sqr', end; // create the return matrix grad_E = hypermat(size(W)'); // rl - run between layers, parameter for ann_FF_run function rl = [2,size(N,'c')]; // for each pattern for p = 1 : size(x,'c') // for each layer for l = 2 : size(N,'c') // for each neuron in layer for n = 1 : N(l) // for each connection to previous layer for i = 1 : N(l-1) // hold the old value of W temp = W(n,i,l-1); // change W value W(n,i,l-1) = temp - dW; // run the net y = ann_FF_run_nb(x(:,p),N,W,rl,af); // calculate new error, to the "left" execstr('err_n = ' + ef + '(y,t(:,p))'); // change W value W(n,i,l-1) = temp + dW; // run the net y = ann_FF_run_nb(x(:,p),N,W,rl,af); // calculate new error, to the "right" execstr('err_p = ' + ef + '(y,t(:,p))'); // "2" because \Delta w = 2 * dW grad_E(n,i,l-1) = ... grad_E(n,i,l-1) + (err_p - err_n) / (2 * dW); // restore W W(n,i,l-1) = temp; end; end; end; end; endfunction scilab-ann-0.4.2.4/ANN_toolbox.iss0000644000175000017500000001032211441407762017456 0ustar sylvestresylvestreÿþ;############################################################################################################## ; Inno Setup Install script for Toolbox_skeleton ; http://www.jrsoftware.org/isinfo.php ; Allan CORNET ; Copyright INRIA 2008 ;############################################################################################################## ; modify this path where is toolbox_skeleton directory #define BinariesSourcePath "E:\ANN_Toolbox_0.4.2.2" #define ANN_Toolbox_version "0.4.2.2" #define CurrentYear "2008" #define Toolbox_ANNDirFilename "ANN_Toolbox_0.4.2.2" ;############################################################################################################## [Setup] ; Debut Données de base à renseigner suivant version SourceDir={#BinariesSourcePath} AppName=ANN Toolbox 0.4.2.2 for Scilab 5.x AppVerName=ANN Toolbox 0.4.2.2 for Scilab 5.x DefaultDirName={pf}\{#Toolbox_ANNDirFilename} InfoAfterfile=readme.txt LicenseFile=license.txt WindowVisible=true AppPublisher=Your Company BackColorDirection=lefttoright AppCopyright=Copyright © {#CurrentYear} Compression=lzma/max InternalCompressLevel=normal SolidCompression=true VersionInfoVersion={#ANN_Toolbox_version} VersionInfoCompany=NONE ;############################################################################################################## [Files] ; Add here files that you want to add Source: loader.sce; DestDir: {app} Source: builder.sce; DestDir: {app} Source: license.txt; DestDir: {app} Source: etc\ANN_toolbox.quit; DestDir: {app}\etc Source: etc\ANN_toolbox.start; DestDir: {app}\etc Source: help\en_US\addchapter.sce; DestDir: {app}\help\en_US Source: jar\scilab_en_US_help.jar; DestDir: {app}\jar Source: macros\buildmacros.sce; DestDir: {app}\macros Source: macros\lib; DestDir: {app}\macros Source: macros\names; DestDir: {app}\macros Source: macros\*.sci; DestDir: {app}\macros Source: macros\*.bin; DestDir: {app}\macros Source: demos\*.*; DestDir: {app}\demos; Flags: recursesubdirs ; ;############################################################################################################## ; scilab-ann-0.4.2.4/etc/0000755000175000017500000000000011441407762015331 5ustar sylvestresylvestrescilab-ann-0.4.2.4/etc/ANN_toolbox.start0000644000175000017500000000305611441407762020576 0ustar sylvestresylvestre// ============================================================================= // Allan CORNET // Copyright DIGITEO 2010 // Copyright INRIA 2008 // ============================================================================= mprintf("Start ANN Toolbox 0.4.2.4\n"); if isdef("ANN_toolboxlib") then warning("ANN Toolbox 0.4.2.4 library is already loaded"); return; end etc_tlbx = get_absolute_file_path("ANN_toolbox.start"); etc_tlbx = getshortpathname(etc_tlbx); root_tlbx = strncpy( etc_tlbx, length(etc_tlbx)-length("\etc\") ); //Load functions library // ============================================================================= mprintf("\tLoad macros\n"); pathmacros = pathconvert( root_tlbx ) + "macros" + filesep(); ANN_toolboxlib = lib(pathmacros); clear pathmacros; // Load and add help chapter // ============================================================================= if or(getscilabmode() == ["NW";"STD"]) then mprintf("\tLoad help\n"); path_addchapter = pathconvert(root_tlbx+"/jar"); if ( isdir(path_addchapter) <> [] ) then add_help_chapter("ANN Toolbox 0.4.2.4", path_addchapter, %F); clear add_help_chapter; end clear path_addchapter; end // Load demos // ============================================================================= if or(getscilabmode() == ["NW";"STD"]) then mprintf("\tLoad demos\n"); pathdemos = pathconvert(root_tlbx+"/demos/ANN.dem.gateway.sce",%F,%T); add_demo("ANN Toolbox 0.4.2.4", pathdemos); clear pathdemos add_demo; end clear root_tlbx; clear etc_tlbx; scilab-ann-0.4.2.4/etc/ANN_toolbox.quit0000644000175000017500000000026711441407762020424 0ustar sylvestresylvestre// ==================================================================== // Allan CORNET // Copyright INRIA 2008 // ====================================================================scilab-ann-0.4.2.4/builder.sce0000644000175000017500000000246011441407762016702 0ustar sylvestresylvestre// ============================================================================= // Copyright INRIA 2008 // Copyright DIGITEO 2010 // Allan CORNET // ============================================================================= mode(-1); lines(0); TOOLBOX_NAME = "ANN_toolbox"; TOOLBOX_TITLE = "ANN toolbox"; toolbox_dir = get_absolute_file_path("builder.sce"); // Check Scilab's version // ============================================================================= try v = getversion("scilab"); catch error(gettext("Scilab 5.3 or more is required.")); end if v(2) < 3 then // new API in scilab 5.3 error(gettext('Scilab 5.3 or more is required.')); end clear v; // Check modules_manager module availability // ============================================================================= if ~isdef('tbx_build_loader') then error(msprintf(gettext('%s module not installed."), 'modules_manager')); end // Action // ============================================================================= tbx_builder_macros(toolbox_dir); tbx_builder_help(toolbox_dir); tbx_build_loader(TOOLBOX_NAME, toolbox_dir); tbx_build_cleaner(TOOLBOX_NAME, toolbox_dir); // Clean variables // ============================================================================= clear toolbox_dir TOOLBOX_NAME TOOLBOX_TITLE; scilab-ann-0.4.2.4/demos/0000755000175000017500000000000011441407762015665 5ustar sylvestresylvestrescilab-ann-0.4.2.4/demos/enc858_ssab_nb.sce0000644000175000017500000000105011441407762021056 0ustar sylvestresylvestre// ================================================== // Loose 8-5-8 encoder // on a backpropagation network without biases, with SuperSAB // (Note that the tight 8-3-8 encoder will not work without biases) // (The 8-4-8 encoder have proven very difficult to train on SuperSAB) // ================================================== FILENAMEDEM = "enc858_ssab_nb"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/ANN.dem.gateway.sce0000644000175000017500000000273611441407762021211 0ustar sylvestresylvestre// ==================================================================== // Copyright INRIA 2008 // Allan CORNET // ==================================================================== demopath = get_absolute_file_path("ANN.dem.gateway.sce"); subdemolist = [ "encoder 4-3-4 on ANN without biases", "encoder_nb.sce" ; .. "tight encoder 4-2-4 on ANN with biases", "encoder.sce" ; .. "encoder 4-3-4 on ANN without biases compare with encoder_nb.sce", "encoder_m_nb.sce" ; .. "tight encoder 4-2-4 on ANN with biases compare with encoder.sce", "encoder_m.sce" ; .. "encoder 8-4-8 on ANN without biases", "enc848_m_nb.sce" ; .. "encoder 8-3-8 on ANN with biases", "enc838_m.sce" ; .. "encoder 8-5-8 on ANN without biases", "enc858_ssab_nb.sce" ; .. "encoder 8-4-8 on ANN with biases", "enc848_ssab.sce" ; .. "tight encoder 4-2-4 on ANN with biases uses a mixed standard/conjugate gradients method", "encoder_cc.sce" .. ]; subdemolist(:,2) = demopath + subdemolist(:,2); // ==================================================================== scilab-ann-0.4.2.4/demos/enc848_m_nb.sci0000644000175000017500000000161111441407762020370 0ustar sylvestresylvestre// Loose 8-4-8 encoder // on a backpropagation network without biases, with momentum // (Note that the tight 8-4-8 encoder will not work without biases) rand('seed',0); // network def. // - neurons per layer, including input N = [8,4,8]; // inputs x = [1,0,0,0,0,0,0,0; 0,1,0,0,0,0,0,0; 0,0,1,0,0,0,0,0; 0,0,0,1,0,0,0,0; 0,0,0,0,1,0,0,0; 0,0,0,0,0,1,0,0; 0,0,0,0,0,0,1,0; 0,0,0,0,0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [2.5,0.1,0.9,0.25]; // init randomize weights between: r = [-1,7]; W = ann_FF_init_nb(N,r); Delta_W_old = hypermat(size(W)'); // 250 epochs are enough to ilustrate T = 250; [W,Delta_W_old] = ann_FF_Mom_online_nb(x,t,N,W,lp,T,Delta_W_old); // full run ann_FF_run_nb(x,N,W) // encoder encoder = ann_FF_run_nb(x,N,W,[2,2]) // decoder decoder = ann_FF_run_nb(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_m_nb.sci0000644000175000017500000000142211441407762020776 0ustar sylvestresylvestre// Loose 4-3-4 encoder // on a backpropagation network without biases, with momentum // (Note that the tight 4-2-4 encoder will not work without biases) rand('seed',0); // network def. // - neurons per layer, including input N = [4,3,4]; // inputs x = [1,0,0,0; 0,1,0,0; 0,0,1,0; 0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [2.5,0.05,0.9,0.25]; // init randomize weights between: r = [-10,15]; W = ann_FF_init_nb(N,r); Delta_W_old = hypermat(size(W)'); // 50 epochs are enough to ilustrate T = 50; [W,Delta_W_old] = ann_FF_Mom_online_nb(x,t,N,W,lp,T,Delta_W_old); // full run ann_FF_run_nb(x,N,W) // encoder encoder = ann_FF_run_nb(x,N,W,[2,2]) // decoder decoder = ann_FF_run_nb(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/enc858_ssab_nb.sci0000644000175000017500000000214611441407762021071 0ustar sylvestresylvestre// Loose 8-5-8 encoder // on a backpropagation network without biases, with SuperSAB // (Note that the tight 8-3-8 encoder will not work without biases) // (The 8-4-8 encoder have proven very difficult to train on SuperSAB) rand('seed',0); // network def. // - neurons per layer, including input N = [8,5,8]; // inputs x = [1,0,0,0,0,0,0,0; 0,1,0,0,0,0,0,0; 0,0,1,0,0,0,0,0; 0,0,0,1,0,0,0,0; 0,0,0,0,1,0,0,0; 0,0,0,0,0,1,0,0; 0,0,0,0,0,0,1,0; 0,0,0,0,0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [0.4, 0, 0.85, 1.004, 0.9999]; // init randomize weights between: r = [-1,2]; W = ann_FF_init_nb(N,r); mu = lp(1) * hypermat(size(W)',ones(prod(size(W)'),1)); Delta_W_old = hypermat(size(W)'); Delta_W_oldold = hypermat(size(W)'); // 350 epochs are enough to ilustrate T = 350; [W, Delta_W_old, Delta_W_oldold, mu] ... = ann_FF_SSAB_online_nb(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu); // full run ann_FF_run_nb(x,N,W) // encoder encoder = ann_FF_run_nb(x,N,W,[2,2]) // decoder decoder = ann_FF_run_nb(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_nb.sce0000644000175000017500000000071311441407762020460 0ustar sylvestresylvestre// ================================================== // Loose 4-3-4 encoder on a backpropagation network without biases // (Note that the tight 4-2-4 encoder will not work without biases) // ================================================== FILENAMEDEM = "encoder_nb"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/enc848_m_nb.sce0000644000175000017500000000073611441407762020373 0ustar sylvestresylvestre// ================================================== // Loose 8-4-8 encoder // on a backpropagation network without biases, with momentum // (Note that the tight 8-4-8 encoder will not work without biases) // ================================================== FILENAMEDEM = "enc848_m_nb"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/encoder_m.sci0000644000175000017500000000126311441407762020322 0ustar sylvestresylvestre// Tight 4-2-4 encoder // on a backpropagation ANN with biases and momentum rand('seed',0); // network def. // - neurons per layer, including input N = [4,2,4]; // inputs x = [1,0,0,0; 0,1,0,0; 0,0,1,0; 0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [2.5,0,0.9,0.25]; // init randomize weights between: r = [-1,7]; W = ann_FF_init(N,r); Delta_W_old = hypermat(size(W)'); // 200 epochs are enough to ilustrate T = 200; [W,Delta_W_old] = ann_FF_Mom_online(x,t,N,W,lp,T,Delta_W_old); // full run ann_FF_run(x,N,W) // encoder encoder = ann_FF_run(x,N,W,[2,2]) // decoder decoder = ann_FF_run(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/enc848_ssab.sci0000644000175000017500000000212511441407762020406 0ustar sylvestresylvestre// Loose 8-5-8 encoder // on a backpropagation network without biases, with SuperSAB // (Note that the tight 8-3-8 encoder will not work without biases) // (The 8-4-8 encoder have proven very difficult to train on SuperSAB) rand('seed',0); // network def. // - neurons per layer, including input N = [8,4,8]; // inputs x = [1,0,0,0,0,0,0,0; 0,1,0,0,0,0,0,0; 0,0,1,0,0,0,0,0; 0,0,0,1,0,0,0,0; 0,0,0,0,1,0,0,0; 0,0,0,0,0,1,0,0; 0,0,0,0,0,0,1,0; 0,0,0,0,0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [2, 0, 0.85, 1.003, 0.9999]; // init randomize weights between: r = [-1,1]; W = ann_FF_init(N,r); mu = lp(1) * hypermat(size(W)',ones(prod(size(W)'),1)); Delta_W_old = hypermat(size(W)'); Delta_W_oldold = hypermat(size(W)'); // 300 epochs are enough to ilustrate T = 300; [W, Delta_W_old, Delta_W_oldold, mu] ... = ann_FF_SSAB_online(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T,mu); // full run ann_FF_run(x,N,W) // encoder encoder = ann_FF_run(x,N,W,[2,2]) // decoder decoder = ann_FF_run(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_cc.sci0000644000175000017500000000126611441407762020456 0ustar sylvestresylvestre// Tight 4-2-4 encoder using a mixed standard/conjugate gradients algorithm rand('seed',0); x = [1,0,0,0; 0,1,0,0; 0,0,1,0; 0,0,0,1]'; t = x; N = [4,2,4]; W = ann_FF_init(N, [-1,1], [-1,1]); // --- standard BP algorithm --- // learning parameter for standard BP part lp = [2.5,0]; printf("Standard BP ..."); // standard BP for first 20 steps T = 20; W = ann_FF_Std_online(x,t,N,W,lp,T); // --- Conjugate Gradients algorithm --- printf("Conjugate Gradients ..."); T = 20; dW = 0.00001; W = ann_FF_ConjugGrad(x, t, N, W, T, dW); // --- test --- // full run ann_FF_run(x,N,W) // encoder encoder = ann_FF_run(x,N,W,[2,2]) // decoder decoder = ann_FF_run(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_m.sce0000644000175000017500000000061711441407762020320 0ustar sylvestresylvestre// ================================================== // Tight 4-2-4 encoder // on a backpropagation ANN with biases and momentum // ================================================== FILENAMEDEM = "encoder_m"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/enc838_m.sce0000644000175000017500000000061611441407762017710 0ustar sylvestresylvestre// ================================================== // Tight 8-3-8 encoder // on a backpropagation ANN with biases and momentum // ================================================== FILENAMEDEM = "enc838_m"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/README.examples0000644000175000017500000000152211441407762020362 0ustar sylvestresylvestreBackpropagation - Standard algorithm encoder_nb.sce : 4-3-4 encoder on ANN without biases encoder.sce : 4-2-4 tight encoder on ANN with biases - Momentum encoder_m_nb.sce : 4-3-4 encoder on ANN without biases compare with encoder_nb.sce encoder_m.sce : 4-2-4 tight encoder on ANN with biases compare with encoder.sce enc848_m_nb.sce : 8-4-8 encoder on ANN without biases enc838_m.sce : 8-3-8 encoder on ANN with biases - SuperSAB enc858_ssab_nb.sce : 8-5-8 encoder on ANN without biases enc848_ssab.sce : 8-4-8 encoder on ANN with biases (Note that the more tight encoders are very difficult to train with this algorithm) - Conjugate Gradients encoder_cc.sce : 4-2-4 tight encoder on ANN with biases uses a mixed standard/conjugate gradients method scilab-ann-0.4.2.4/demos/encoder_nb.sci0000644000175000017500000000133611441407762020466 0ustar sylvestresylvestre// Loose 4-3-4 encoder on a backpropagation network without biases // (Note that the tight 4-2-4 encoder will not work without biases) // ensure the same random starting point rand('seed',0); // network def. // - neurons per layer, including input N = [4,3,4]; // inputs x = [1,0,0,0; 0,1,0,0; 0,0,1,0; 0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [8,0]; // init randomize weights between r = [-1,1]; W = ann_FF_init_nb(N,r); // 500 epochs are enough to ilustrate T = 500; W = ann_FF_Std_online_nb(x,t,N,W,lp,T); // full run ann_FF_run_nb(x,N,W) // encoder encoder = ann_FF_run_nb(x,N,W,[2,2]) // decoder decoder = ann_FF_run_nb(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_m_nb.sce0000644000175000017500000000073711441407762021002 0ustar sylvestresylvestre// ================================================== // Loose 4-3-4 encoder // on a backpropagation network without biases, with momentum // (Note that the tight 4-2-4 encoder will not work without biases) // ================================================== FILENAMEDEM = "encoder_m_nb"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/enc838_m.sci0000644000175000017500000000147511441407762017720 0ustar sylvestresylvestre// Tight 8-3-8 encoder // on a backpropagation ANN with biases and momentum rand('seed',0); // network def. // - neurons per layer, including input N = [8,3,8]; // inputs x = [1,0,0,0,0,0,0,0; 0,1,0,0,0,0,0,0; 0,0,1,0,0,0,0,0; 0,0,0,1,0,0,0,0; 0,0,0,0,1,0,0,0; 0,0,0,0,0,1,0,0; 0,0,0,0,0,0,1,0; 0,0,0,0,0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [1.5, 0.07, 0.8, 0.1]; // init randomize weights between: r = [-10,15]; rb = r; W = ann_FF_init(N,r,rb); Delta_W_old = hypermat(size(W)'); // 500 epochs are enough to ilustrate T = 500; [W,Delta_W_old] = ann_FF_Mom_online(x,t,N,W,lp,T,Delta_W_old); // full run ann_FF_run(x,N,W) // encoder encoder = ann_FF_run(x,N,W,[2,2]) // decoder decoder = ann_FF_run(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/encoder_cc.sce0000644000175000017500000000062011441407762020443 0ustar sylvestresylvestre// ================================================== // Tight 4-2-4 encoder using a mixed standard/conjugate gradients algorithm // ================================================== FILENAMEDEM = "encoder_cc"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/encoder.sci0000644000175000017500000000111411441407762020001 0ustar sylvestresylvestre// Tight 4-2-4 encoder on a backpropagation ANN // ensure the same starting point each time rand('seed',0); // network def. // - neurons per layer, including input N = [4,2,4]; // inputs x = [1,0,0,0; 0,1,0,0; 0,0,1,0; 0,0,0,1]'; // targets, at training stage is acts as identity network t = x; // learning parameter lp = [2.5,0]; W = ann_FF_init(N); // 400 epochs are enough to ilustrate T = 400; W = ann_FF_Std_online(x,t,N,W,lp,T); // full run ann_FF_run(x,N,W) // encoder encoder = ann_FF_run(x,N,W,[2,2]) // decoder decoder = ann_FF_run(encoder,N,W,[3,3]) scilab-ann-0.4.2.4/demos/enc848_ssab.sce0000644000175000017500000000104511441407762020402 0ustar sylvestresylvestre// ================================================== // Loose 8-5-8 encoder // on a backpropagation network without biases, with SuperSAB // (Note that the tight 8-3-8 encoder will not work without biases) // (The 8-4-8 encoder have proven very difficult to train on SuperSAB) // ================================================== FILENAMEDEM = "enc848_ssab"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/demos/encoder.sce0000644000175000017500000000056111441407762020002 0ustar sylvestresylvestre// ================================================== // Tight 4-2-4 encoder on a backpropagation ANN // ================================================== FILENAMEDEM = "encoder"; lines(0); scepath = get_absolute_file_path(FILENAMEDEM+".sce"); exec(scepath+FILENAMEDEM+".sci",1); clear scepath; clear FILENAMEDEM; // ================================================== scilab-ann-0.4.2.4/help/0000755000175000017500000000000011441407762015506 5ustar sylvestresylvestrescilab-ann-0.4.2.4/help/builder_help.sce0000644000175000017500000000044011441407762020636 0ustar sylvestresylvestre// ==================================================================== // Copyright INRIA 2008 // Allan CORNET // ==================================================================== tbx_builder_help_lang(["en_US"], .. get_absolute_file_path("builder_help.sce")); scilab-ann-0.4.2.4/help/en_US/0000755000175000017500000000000011441407762016517 5ustar sylvestresylvestrescilab-ann-0.4.2.4/help/en_US/ann_FF_Jacobian_BP.xml0000644000175000017500000000365111441407762022544 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Jacobian_BP computes Jacobian trough backpropagation. CALLING SEQUENCE J = ann_FF_Jacobian_BP(x,N,W[,af]) PARAMETERS J The Jacobian hypermatrix: each J(:,:,p) have same structure as z(:,p)*x(:,p)', where z(:,p) is the network output given input x(:,p). x Matrix of input patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. af The activation function to be used. This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. Description This function calculates the Jacobian trough a backpropagation algorithm, for all patterns presented in x. See Also ANN ANN_GEN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_Mom_online.xml0000644000175000017500000001276211441407762022554 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Mom_online online backpropagation with momentum. CALLING SEQUENCE [W,Delta_W_old]= ann_FF_Mom_online(x,t,N,W,lp,T[,Delta_W_old,af,ex,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_FF_init). lp Learning parameters [lp(1),lp(2),lp(3),lp(4)]. lp(1) is the well known learning parameter of standard backpropagation algorithm, W is changed according to the formula: W(t+1) = W(t) - lp(1) * grad E + momentum term where t is the (discrete) time and E is the error. Typical values: 0.1 ... 1. Some networks train faster with even greater learning parameter. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. lp(3) is the momentum parameter. The momentum term added to W is: momentum term = lp(3) * Delta_W_old Typical values: 0 ... 0.9999... (smaller than 1). lp(4) is the flat spot elimination constant added to the derivative of activation function, when computing the error gradient, to help the network pass faster over areas where the error gradient is small (flat error surface area). I.e. the derivative of activation is replaced: f'(total neuronal input) --> f'(total neuronal input) + lp(4) when computing the gradient. Typical values: 0 ... 0.25. T The number of epochs (training cycles trough all pattern set). Delta_W_old The previous weight adjusting quantity. This parameter is optional, default value is: Delta_W_old = hypermat(size(W)') NOTE: When calling ann_FF_Mom_online for the first time you should either: - not give any value to Delta_W_old - initialize it to zero using "Delta_W_old=hypermat(size(W)')" On subsequent calls to ann_FF_Mom_online you should give the value of Delta_W_old returned by the previous call. af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y = f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight matrix have been updated, after each pattern (not whole set), using execstr. ex(2), same as ex(1), but is executed once after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is online backpropagation with momentum. Delta_W_old holds the previous W update (usefull for subsequent calls). See Also ANN ANN_GEN ANN_FF ann_FF_init ann_FF_run scilab-ann-0.4.2.4/help/en_US/build_help.sce0000644000175000017500000000043211441407762021321 0ustar sylvestresylvestre// ==================================================================== // Copyright INRIA 2008 // Copyright DIGITEO 2010 // Allan CORNET // ==================================================================== tbx_build_help(TOOLBOX_TITLE,get_absolute_file_path("build_help.sce")); scilab-ann-0.4.2.4/help/en_US/ann_FF_Jacobian.xml0000644000175000017500000000374211441407762022164 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Jacobian computes Jacobian by finite differences. CALLING SEQUENCE J = ann_FF_Jacobian(x,N,W,dx[,af]) PARAMETERS J The Jacobian hypermatrix: each J(:,:,p) have same structure as z(:,p)*x(:,p)', where z(:,p) is the network output given input x(:,p). x Matrix of input patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. dx The quantity used to perturb each x(i,p) in turn. af The activation function to be used. This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. Description This function calculates the Jacobian trough a finite differences procedure, for all patterns presented in x. See Also ANN ANN_GEN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_log_activ.xml0000644000175000017500000000316611441407762022052 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_log_activ logistic activation function CALLING SEQUENCE y = ann_log_activ(x) PARAMETERS y Matrix containing the activation, one pattern per column. For each pattern: 1 y(i) = -------------- 1 + exp[-x(i)] x Matrix containing the total neuronal input, one pattern per column. For each pattern: each x(i) for the corresponding i-th neuron on the current layer. Description This function is the default neuronal activation function. Any other, user defined, function should have the same input and output format for variables. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_run_nb.xml0000644000175000017500000000431111441407762021732 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_run_nb run patterns trough a feedforward net (without bias). CALLING SEQUENCE y = ann_FF_run_nb(x,N,W[,l,af]) PARAMETERS y Matrix of outputs, one pattern per column. Each column have a correspondent column in x. x Matrix of input patterns, one pattern per column N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector. W The weight hypermatrix (initialized first trough ann_BP_init_nb). l Defines the "injection" layer and the output layer. x patterns are injected into layer l(1) as coming from layer l(1) - 1. y outputs are collected from the outputs of layer l(2). This parameter is optional, default value is [2,size(N,'c')], i.e. the whole network. Warning: l(1)=1 does not make sense. af String containing the name of activation function. This parameter is optional, default value "ann_log_activ", i.e. logistic activation function. Description This function is used to run patterns trough a feedforward network as defined by N and W. This function is to be used on networks without biases. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_sum_of_sqr.xml0000644000175000017500000000305611441407762022256 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_sum_of_sqr calculates sum-of-squares error CALLING SEQUENCE E = ann_sum_of_sqr(y,t) PARAMETERS E The sum-of-squares error. y Matrix containing the actual network outputs, one pattern per column, each column have a correspondent in t. t Matrix containing the targets, one pattern per column, each column have a correspondent in y. Description This function calculates the sum-of-squares error given y and t. Any other, user defined, error function should have the same input and output format for variables. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_grad.xml0000644000175000017500000000503311441407762021366 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_grad error gradient trough finite differences. CALLING SEQUENCE grad_E = ann_FF_grad(x,t,N,W,dW[,af,ef]) PARAMETERS grad_E The error gradient, same layout as W. x Input patterns, one per column. t Target patterns, one per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. dW The quantity used to perturb each W parameter. af The name of activation function to be used (string). This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. ef The name of error function to be used (string). This parameter is optional, default value "ann_sum_of_sqr", i.e. the sum-of-squares error function. Description Calculates error gradient trough a (slow) finite differences procedure. Each element W(n,i,l) is changed to W(n,i,l)-dW then the error is calculated and the process is repeated for W(n,i,l)+dW. From the values obtained the partial derivative of the sum-of-squares error function, with respect to W(n,i,l), is calculated and the value of gradient returned. This process is very slow (compared to the backpropagation algorithms) so it is to be used only for testing purposes. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_init.xml0000644000175000017500000000402411441407762021413 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_init initialize the weight hypermatrix. CALLING SEQUENCE W = ann_FF_init(N[,r,rb]) PARAMETERS N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). r Two component row vector defining the smallest and the largest value for initialization. Weights will be initialized with random numbers between these two values. r(1) the lower limit r(2) the upper limit This parameter is optional, default value is [-1,1]. rb Same as r but for biases. This parameter is optional, default value is [0,0], i.e. the biases are initialized to zero. W The weight hypermatrix, in the format used by ann_FF_Std, ann_BP_run and all other algorithms for feedforward nets (with biases). Description This function builds the weight hypermatrix according to network description N. Its format is detailed in ANN_FF. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_d_sum_of_sqr.xml0000644000175000017500000000371111441407762022557 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_d_sum_of_sqr derivative of sum-of-squares error CALLING SEQUENCE err_d = ann_d_sum_of_sqr(y,t) PARAMETERS err_d Matrix containing the sum-of-squares error derivative, one per column, each column have a correspondent in y and t. y Matrix containing the actual network outputs, one per column, each column have a correspondent in t and err_d. t Matrix containing the targets, one per column, each column have a correspondent in y and err_d. Description This function calculates the derivative of sum-of-squares error given y and t. Any other, user defined, error function should have the same input and output format for variables. Note: this function is extremly simple, it just calculates y-t. The only reason to provide it is to allow the option to change the default error function if so is desired. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_grad_BP.xml0000644000175000017500000000603011441407762021745 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_grad_BP error gradient trough backpropagation CALLING SEQUENCE W = ann_FF_grad_BP(x,t,N,W[,c,af,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column. t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_BP_init). c defines the threshold of error which is backpropagated: a error smaller that c (at one neuronal output) is rounded towards zero and thus not propagated. It have to be set to zero for exact calculation of gradient. This parameter is optional, default value 0. af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. Description Returns the error gradient hypermatrix (it have the same layout as W) of a feedforward ANN, using the whole given training set. The algorithm used is standard backpropagation. EXAMPLES This function is used as a low level engine in other algorithms (thus the reason for existence of c parameter), e.g. see implementation of ann_FF_ConjugGrad function. See Also ANN ANN_FF ANN_FF_init scilab-ann-0.4.2.4/help/en_US/ann_FF_Mom_online_nb.xml0000644000175000017500000001301111441407762023217 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Mom_online_nb online backpropagation with momentum. CALLING SEQUENCE [W,Delta_W_old]= ann_FF_Mom_online_nb(x,t,N,W,lp,T[,Delta_W_old,af,ex,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first with ann_FF_init_nb). lp Learning parameters [lp(1),lp(2),lp(3),lp(4)]. lp(1) is the well known learning parameter of standard backpropagation algorithm, W is changed according to the formula: W(t+1) = W(t) - lp(1) * grad E + momentum term where t is the (discrete) time and E is the error. Typical values: 0.1 ... 1. Some networks train faster with even greater learning parameter. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. lp(3) is the momentum parameter. The momentum term added to W is: momentum term = lp(3) * Delta_W_old Typical values: 0 ... 0.9999... (smaller than 1). lp(4) is the flat spot elimination constant added to the derivative of activation function, when computing the error gradient, to help the network pass faster over areas where the error gradient is small (flat error surface area). I.e. the derivative of activation is replaced: f'(total neuronal input) --> f'(total neuronal input) + lp(4) when computing the gradient. Typical values: 0 ... 0.25. T the number of epochs (training cycles trough all pattern set). Delta_W_old The previous weight adjusting quantity. This parameter is optional, default value is: Delta_W_old = hypermat(size(W)'). NOTE: When calling ann_FF_Mom_online_nb for the first time you should either: - not give any value to Delta_W_old - initialize it to zero using "Delta_W_old=hypermat(size(W)')" On subsequent calls to ann_FF_Mom_online_nb you should give the value of Delta_W_old returned by the previous call. af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y = f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight hypermatrix have been updated, after each pattern (not whole set), using execstr. ex(2), same as ex(1), but is executed after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns. The algorithm used is online backpropagation with momentum. Delta_W_old holds the previous weight update (useful for subsequent calls). This function is to be used on networks without biases. See Also ANN ANN_GEN ANN_FF ann_FF_init_nb ann_FF_run_nb scilab-ann-0.4.2.4/help/en_US/ann_FF_run.xml0000644000175000017500000000371211441407762021257 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_run run patterns trough a feedforward net. CALLING SEQUENCE y = ann_FF_run(x,N,W[,l,af]) PARAMETERS y Matrix of outputs, one pattern per column. Each column have a correspondent column in x. x Matrix of input patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector. W The weight hypermatrix (initialized first trough ann_FF_init). l Defines the "injection" layer and the output layer. x patterns are injected into layer l(1) as coming from layer l(1) - 1. y outputs are collected from the outputs of layer l(2). This parameter is optional, default value is [2,size(N,'c')], i.e. the whole network. Warning: l(1)=1 does not make sense. af String containing the name of activation function. This parameter is optional, default value is "ann_log_activ", i.e. logistic activation function. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ANN_GEN.xml0000644000175000017500000000432611441407762020353 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ANN_GEN General utility functions OBJECTIVE To provide for some functions wich are usefull for a larger number of architectures. This is their general man page. USER INTERFACE: PARAMETERS AND FUNCTIONS In alphabetical order, not all functions require all parameters: x Matrix containing input patterns, one per column. There shall be a unique column corespondence with the matrix of targets t and/or y. y,z Matrix containing actual output patterns, one per column. There shall be a unique column corespondence with the matrix of targets t and/or x. t Matrix containing target patterns, one per column. There shall be a unique column corespondence with the matrix of targets x and/or y. y = ann_log_activ(x) Logistic activation function. Here y and x have to be column vectors. z = ann_d_log_activ(y) Derivative of logistic activation function, expressed in terms of actual output (not in terms of total input !). E = ann_sum_of_sqr(y,t) Sum-of-squares error function. err_d = ann_d_sum_of_sqr(y,t) Derivative, with respect to network outputs of sum-of-squares error function. Here y and t have to be column vectors. [x,t] = ann_pat_shuffle(x,t) Shuffles patterns randomly for better network training, the corespondence between x and t is preserved See Also ANN scilab-ann-0.4.2.4/help/en_US/ann_FF_Std_batch.xml0000644000175000017500000000266511441407762022354 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Std_batch standard batch backpropagation. CALLING SEQUENCE W = ann_BP_Std_batch(x,t,N,W,lp,T[,af,ex]) PARAMETERS This function have same parameters as ann_FF_Std_online except for: ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is standard batch backpropagation. See Also ann_FF_Std_online scilab-ann-0.4.2.4/help/en_US/ann_d_log_activ.xml0000644000175000017500000000400711441407762022350 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_d_log_activ derivative of logistic activation function Description This function is the derivative of default neuronal activation function. Any other, user defined, function should have the same input and output format for variables. Particularly note that the derivative is expressed in terms of activation, not total neuronal input. CALLING SEQUENCE z = ann_d_log_activ(y) PARAMETERS z Matrix containing the derivative of activation, one pattern per column. For each pattern (column): z(i) = y(i) * [1 - y(i)] y Matrix containing the current neuronal activation, one pattern per column. For each pattern: each y(i) for the corresponding i-th neuron (on the current layer). COMPONENTS General support functions See ANN_GEN for support functions for ANN. Feedforward networks See ANN_FF for detailed description. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_SSAB_online.xml0000644000175000017500000001224611441407762022551 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_SSAB_online online SuperSAB training algorithm. CALLING SEQUENCE [W,Delta_W_old,Delta_W_oldold,mu]= ann_FF_SSAB_online(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T[,mu,af,ex,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_FF_init). lp Learning parameters [lp(1),lp(2),lp(3),lp(4),lp(5)]. lp(1) is the well known learning parameter of vanilla backpropagation algorithm, it is used only to initialize mu if not given (mu being optional) Typical values: 0.1 ... 1. Note that initial values may be strongly amplified in some circumstances. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. lp(3) is the momentum parameter, used to "cancel" a previously wrong weight adaptation. Typical values: 0 ... 0.9999... (smaller than 1). lp(4) is the adaptive increasing factor. Typical values: 1.1 ... 1.3. lp(5) is the adaptive decreasing factor. Typical values: lp(5) = 1/lp(4) (lp(5) < 1). Delta_W_old The previous weight adjusting quantity. On first call this parameter should be initialized to zero using e.g. Delta_W_old = hypermat(size(W)') On subsequent calls to ann_FF_SSAB you should give the value of Delta_W_old returned by the previous call. Delta_W_oldold The weight adjusting quantity used two steps back. Same remarks as for Delta_W_old (above) apply. T The number of epochs (training cycles trough all pattern set). mu This is the hypermatrix of learning constants who replaces lp(1) and is adapted at each step. This parameter is optional, default value is "mu = lp(1)*hypermat(size(W)',ones(prod(size(W)'),1))", i.e. it is initialized uniformly to lp(1). af Activation function and its derivative. Row vector of strings: af(1) name of activation function (string). af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight matrix have been updated, after each pattern (not whole set), using execstr. ex(2) - same as ex(1) - but is executed once after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns. The algorithm used is online backpropagation with SuperSAB. See Also ANN ANN_GEN ANN_FF ann_FF_init ann_FF_run scilab-ann-0.4.2.4/help/en_US/ann_FF_INT.xml0000644000175000017500000000523611441407762021110 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_INT internal implementation of feedforward nets. Description This man page describes the internals of implementation, it is of interest only to those wanting to modify/adapt the functions provided (in case they do not fit the need :-) INTERNAL VARIABLES (IN ALPHABETICAL ORDER) The variables described here are used internally only in the functions body. They are: err_dz the partial derivative of error with respect to neuronal outputs, on current layer. err_dz_deriv_af the element-wise product between err_dz and deriv_af. deriv_af the derivative of activation function for current layer. grad_E gradient of error, same hypermatrix structure as W. E.g. grad_E(n,i,l) destined to hold the partial derivative of error with respect to W(n,i,l). grad_E_mod similar to grad_E but it is not the real grad_E because is modified in some significant way. l, ll current layer number (different from l(1), l(2)) (if l=[l(1),l(2)] is used as parameter then ll is used as layer counter) L total number of layers, including input. p current pattern number. P total number of patterns. y holds the network output(s), one per column. z matrix of neuronal outputs, one column for each layer, e.g. z(1:N(1),1) contains the inputs z(1:N(2),2) contains the outputs of first hidden layer z(1:N(L),L) contains the output BIAS Bias is simulated trough an additional neuron with constant output "1". The "bias neuron" is present on all layers, except output, as the first neuron. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_Std_batch_nb.xml0000644000175000017500000000303111441407762023017 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Std_batch_nb standard batch backpropagation (without bias). CALLING SEQUENCE W = ann_BP_Std_batch_nb(x,t,N,W,lp,T[,af,ex]) PARAMETERS This function have same parameters as ann_FF_Std_online_nb except for: ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is standard batch backpropagation. This function is to be used on networks without biases. See Also ann_FF_Std_online_nb scilab-ann-0.4.2.4/help/en_US/ann_FF_VHess.xml0000644000175000017500000000471211441407762021504 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_VHess multiplication between a "vector" V and Hessian CALLING SEQUENCE VH = ann_BP_VHess(x, t, N, W, V, dW, [af, err_deriv_y]) PARAMETERS VH The result of multiplication - hypermatrix with the same layout as W. x Matrix of input patterns, one pattern per column. t Matrix of target patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. V The "vector" by which Hessian have to be multiplied, actually is a hypermatrix with same layout as W (is from same space). dW Size of "finite difference". af The activation function to be used. This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. Description This function calculates the product between a vector and Hessian trough a (fast) finite difference procedure. The error gradient is calculated (trough backpropagation) at W+dW*V and at W-dW*V and then VH is calculated directly from here (this require only two backpropagations, explicit computation of Hessian is avoided). See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_Std_online_nb.xml0000644000175000017500000001023011441407762023221 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Std_online_nb online standard backpropagation CALLING SEQUENCE W = ann_BP_Std_online_nb(x,t,N,W,lp,T[,af,ex,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_BP_init_nb). lp Learning parameters [lp(1),lp(2)]. lp(1) is the well known learning parameter of standard backpropagation algorithm, W is changed according to the formula: W(t+1) = W(t) - lp(1) * grad E where t is the (discrete) time and E is the error. Typical values: 0.1 ... 1. Some networks train faster with even greater learning parameter. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. T The number of epochs (training cycles trough all pattern set). af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight hypermatrix have been updated, after each pattern (not whole set), using execstr. ex(2) - same as ex(1) - but is executed after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after online training with a given set of patterns. The algorithm used is online standard backpropagation. This function is to be used on networks without biases. See Also ANN ANN_GEN ANN_FF ann_FF_init_nb ann_FF_run_nb scilab-ann-0.4.2.4/help/en_US/ANN_FF.xml0000644000175000017500000003270611441407762020240 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ANN_FF Algorithms for feedforward nets. OBJECTIVE To provide engines for feedforward ANN exploration, testing and rapid prototyping. Some flexibility is provided (e.g. the possibility to change the activation or error functions). ARCHITECTURE DESCRIPTION (a) The network is visualized as follows: inputs at the left and data (signals) propagating to the right. (b) N is a row vector containing the number of neurons per layer, input included. (c) First layer is input (despite the fact that it does not process data it makes implementation clearer). Layer no. 1 2 ... size(N,'c') . -- o o -> \/ /\ i . -- o o -> o n \ |\ u p \ =====| > t u |/ p t u s / t / s . - o o -> input first output hidden Note that connections do not jump over layers, they are only between adjacent layers (fully interconnected). (d) The dimension of N is size(N,'c') so: - input layer have N(1) neurons - first hidden layer have N(2) neurons, ... - the output layer L have N(size(N,'c')) neurons (e) The input vector/matrix is x, each pattern is represented by one column. Only constant size input patterns are accepted. NOTE: Internally the patterns will be worked with, individually, as column vectors, i.e. each pattern vector is a column of the form: x(:,p), (p being the pattern order number). (f) Each neuron on first hidden layer have N(1) inputs, ... for layer l in [2, ..., size(N,'c')] each neuron have N(l-1) inputs from previous layer plus one simulating the bias (where applicable, most algorithms assume existence of bias). (g) The network is fully connected but a connection can be canceled by zeroing the corresponding weight (note that a training procedure may turn it back to a non-zero value, this is one reason for which some "hooks" are provided, see "ex" parameter below). USER INTERFACE (1): PARAMETERS This subsection describes the parameters taken by the various functions defined within the toolbox, not all functions require all parameters. In alphabetical order: af gives activation function and, if required, its (total) derivative. It is either: - a string giving the name of activation function - a two element row vector of strings where af(1) is the string with the name of activation function and af(2) is the name of the derivative. NOTE: Given an activation function y = f(x), the derivative have to be expressed in terms of y not x). E.g. given the logistic activation function: 1 y = ----------- 1 + exp(-x) the derivative will be expressed as: dy -- = y (1 - y) dx This form reduces the memory usage and, in the particular case of the logistic activation function, increases speed as well. This parameter is optional, default value is either "ann_log_activ" or ["ann_log_activ","ann_d_log_activ"] (depending on the function using it), i.e. the activation function, described above, and its derivative (the default functions are already defined in this toolbox). WARNING: Be very careful how to define a new activation function. These functions accept patterns as column vectors within y, each element representing the total input on a neuron, and return a similar matrix representing the activation of the whole layer. I.e. the logistic is defined as: y = 1 ./ (1 + %e^(-x)) and note the space between 1 and ./ The derivative of activation function is defined similarly: z = y .* (1 - y) and note the ".*" operator. Delta_W_old The quantity by which W was changed on previous training pattern. dW, dW2 the amount of variations of each W element for calculating the error derivatives trough a finite difference approach (see ann_FF_grad and ann_FF_Hess for more information). ef error function. This parameter is optional, default value is "ann_sum_of_sqr", i.e. the sum-of-squares (already defined within this toolbox). err_deriv_y the error derivative with respect to network outputs. Returns a matrix each column containing the error derivative corresponding to the appropriate pattern. This parameter is optional, default value is "ann_d_sum_of_sqr" (already defined within this toolbox), i.e. the derivative of sum-of-squares error function. ex is a Scilab program sequence, executed after the weight hypermatrix for each training pattern have been updated. Its main purpose is to provide hooks in order to change the learning function without having to rewrite it. Typical usages would be: checking for a stop criteria, pruning. This parameter is optional, default value is [" "] or [" "," "] (some functions have two hooks), i.e. empty string, does nothing. l range of layers between which the network is run. Two component row vector: l(1) layer into which a pattern will be injected, presented as it would have come from previous layer: l(1)-1. l(2) layer from which the outputs are collected. E.g.: l = [3,3] means input is injected into neurons from layer 3 and their outputs (l(2)=3) are collected to give the result. l = [2,3] means input is injected into first hidden layer (exactly as it would have come from input layer) and output is collected at the outputs of neurons on layer 3. This parameter is optional, default value is [2,size(N,'c')] (whole network). WARNING: l(1) = 1 does not make sense as it represents the input layer. lp represents the learning parameters, is a row vector [lp(1), lp(2), ...] The actual significance of each component may vary, see the respective man pages for representation and typical values. N row vector, defines the network, i.e. no. of neurons per layer. N(l) represents the number of neurons on layer l. E.g.: N(1) is the size of input vector, N(size(N),'c') is the size of output vector r range of random numbers based on which the connection weights (not biases) are initialized. Is a two component row vector: r(1) gives the lower limit r(2) gives the upper limit This parameter is optional, default value is [-1,1]. rb range of random numbers based on which the biases (not other weihts) are initialized. Is a two component row vector: rb(1) gives the lower limit rb(2) gives the upper limit This parameter is optional, default value is [0,0], i.e. biases are initialized with 0. t matrix of targets, one pattern per column. E.g. t(:,p) represents pattern no. p. x matrix of inputs, one pattern per column. E.g. x(:,p) represents pattern no. p USER INTERFACE (2): FUNCTIONS The function names are built as follows: ann prefix for all function names within this toolbox. _FF prefix for all function names designed for feedforward nets. defines the type of algorithm: online uses one pattern at a time, batch uses all patterns at once. _nb postfix for all function names within this toolbox designed for networks without bias. ann_FF_init Build and initialize the weight hypermatrix. ann_FF_Std Standard (vanilla, delta rule) backpropagation algorithm. ann_FF_Mom Backpropagation with momentum. ann_FF_run Runs the network. ann_FF_grad Calculate the error gradient trough a finite difference approach. It is provided for testing purposes only. ann_FF_Jacobian Calculate the Jacobian trough a finite difference approach. It is provided for testing purposes only. ann_FF_Jacobian_BP Calculate the Jacobian trough a backpropagation algorithm. ann_FF_Hess Calculate the Hessian trough a finite difference approach. It is provided for testing purposes only. ann_FF_VHess Calculate the multiplication between a vector and the Hessian trough an efficient finite difference approach. ann_FF_ConjugGrad Conjugate gradients algorithm. ann_FF_SSAB Backpropagation with SuperSAB algorithm. TIPS AND TRICKS - Do not use the no-bias networks unless you know what you are doing. - The most efficient (by far) algorithm is the "Conjugate Gradient", however it may require bootstrapping with another algorithm (see the examples). - Reduce as much is possible the number of loops and the number of function calls, use instead as much is possible the matrix manipulation capabilities of Scilab. - You can do a shuffling of training patterns between two calls to the training procedure, use the "ex" hooks provided. - Be very careful when defining new activation and error functions and test them to make sure they do what are supposed to do. - don't use sparse matrices unless they are really sparse (< 5%). IMPLEMENTATION DETAILS - Each layer have associated a hypermatrix of weights. NOTE: Most algorithms assume existence of bias by default. For each layer l, except l=1, the weight matrix associated with connections from l-1 to l is W(1:N(l),1:N(l-1),l-1) for networks without biases and W(1:N(l),1:N(l-1)+1,l-1) for networks with biases, i.e. biases are stored in first column: W(1:N(l),1,l-1). The total input to a layer l is: = W(1:N(l),1:N(l-1),l-1)*z(1:N(l-1)) for network without biases = W(1:N(l),1:N(l-1)+1,l-1)*[1; z(1:N(l-1))] for network with biases where z(1:N(l-1)) is output of previous layer (column vector), i.e. bias is simulated as neuron no. 0 on each layer with constant output 1. W is initialized to: hypermat(max(N),max(N),size(N,'c')-1) for networks without biases, "hypermat(max(N),max(N)+1,size(N,'c')-1)" for networks with biases; the unused entries from W are initialized to zero and left untouched. - Pattern vectors are passed as columns in a matrix representing a set (of patterns). OBJECTIVE - No sanity checks are performed as this will greatly hurt speed. It is assumed that (as least to some extent) you know what you are doing ;-) You can implement them yourself if you wish. The following conditions have to be met: + targets have to have the same size as output layer, i.e. size(target,'r') = N(size(N,'c')) + inputs have to have the same size as input layer, i.e. size(input,'r') = N(1) + all N(i) have to be positive integers of course (am I paranoid here ? :-) + lp parameter is a row-vector of numbers (actual dimension depends on the algorithm used). + af is a row-vector of string(s) defining a valid activation function (and its derivative) as appropriate for the algorithm used. + ex is a string or a two row-vector of strings, representing valid Scilab set of instructions. + l(1) <= l(2) (see definition of l above) and l(1) >= 2 + r(1) <= r(2) and rb(1) <= rb(2) (see definition of r and rb above), if not then the program may run but you may not get what you would expect when initializing the weights. Warning: In some particular cases this may lead to very subtle errors (e.g. your program may even run without generating any Scilab errors but the results may be meaningless). - The algorithms themselfs are not described here, there are many books which describes them (e.g. get mine "Matrix ANN" wherever you may find it ;-). - Hypermatrices are slow, however there is no other reasonable way of doing things; tests performed by myself show that using embedded matrices may increase speed but the manipulation of submatrices "by hand" is very tedious and error prone. Of course you may rewrite the algorithms for yourself using embedded matrices if you want to. If you really need speed then go directly to C++ or whatever. See Also ANN ANN_GEN ANN_FFT_INT scilab-ann-0.4.2.4/help/en_US/ann_FF_init_nb.xml0000644000175000017500000000376311441407762022103 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_init_nb initialize the weight hypermatrix (without bias). CALLING SEQUENCE W = ann_BP_init_nb(N[,r]) PARAMETERS N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). r Two component row vector defining the smallest and the largest value for initialization. Weights will be initialized with random numbers between these two values. r(1) the lower limit r(2) the upper limit This parameter is optional, default value is [-1,1]. W The weight hypermatrix, in the format used by ann_BP_Std_nb, ann_BP_run_nb and other functions working with feedforward nets (without bias). Description This function builds the weight hypermatrix according to network description N. The format of it is detailed in ANN_FF. This function is to be used on networks without biases. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_grad_BP_nb.xml0000644000175000017500000000623411441407762022432 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_grad_BP_nb error gradient trough backpropagation (without bias) CALLING SEQUENCE W = ann_FF_grad_BP_nb(x,t,N,W[,c,af,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column. t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_BP_init). c defines the threshold of error which is backpropagated: a error smaller that c (at one neuronal output) is rounded towards zero and thus not propagated. It have to be set to zero for exact calculation of gradient. This parameter is optional, default value 0. af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. Description Returns the error gradient hypermatrix (it have the same layout as W) of a feedforward ANN, using the whole given training set. The algorithm used is standard backpropagation. This function is to be used on networks without biases. EXAMPLES This function is used as a low level engine in other algorithms (thus the reason for existence of c parameter), e.g. see implementation of ann_FF_ConjugGrad function. See Also ANN ANN_FF ANN_FF_init_nb scilab-ann-0.4.2.4/help/en_US/ann_FF_SSAB_batch_nb.xml0000644000175000017500000000304011441407762023015 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_SSAB_batch_nb batch SuperSAB algorithm (without bias). CALLING SEQUENCE [W,Delta_W_old,Delta_W_oldold,mu]= ann_FF_SSAB_batch_nb(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T[,mu,af,ex,err_deriv_y]) PARAMETERS This function have same parameters as ann_FF_SSAB_online_nb except for: ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is batch SuperSAB. See Also ann_FF_SSAB_online_nb scilab-ann-0.4.2.4/help/en_US/ann_FF_Mom_batch_nb.xml0000644000175000017500000000300511441407762023016 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Mom_batch_nb batch backpropagation with momentum (without bias). CALLING SEQUENCE [W,Delta_W_old]= ann_FF_Mom_batch_nb(x,t,N,W,lp,T[,Delta_W_old,af,ex,err_deriv_y]) PARAMETERS ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is batch backpropagation with momentum. This function is to be used on networks without biases. See Also ann_FF_Mom_online_nb scilab-ann-0.4.2.4/help/en_US/ann_pat_shuffle.xml0000644000175000017500000000344111441407762022377 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_pat_shuffle shuffles randomly patterns for an ANN CALLING SEQUENCE [x,t] = ann_pat_shuffle(x,t) PARAMETERS x Matrix containing the training inputs, one per column, each column have a correspondent in t. t Matrix containing the targets, one per column, each column have a correspondent in x. Description This function randomly shuffles the columns in matrices x and t, i.e. randomly shuffles the patterns. Some ANN train better if the training set is presented repeatedly but in random order. This function is intended to be called between two epochs. The correspondence between the columns in x and t is preserved. Note that x and t may change place, i.e. you may call this function as "[t,x]=ann_pat_shuffle(t,x)" as long as you keep your order. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_ConjugGrad.xml0000644000175000017500000000650211441407762022476 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_ConjugGrad Conjugate Gradient algorithm. CALLING SEQUENCE W = ann_BP_ConjugGrad(x,t,N,W,T,dW[,ex,af,err_deriv_y]) PARAMETERS W The weight hypermatrix; it have to be initialized with ann_BP_init function. x Matrix of input patterns, one pattern per column. t Matrix of target patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). T Number of training cycles (epochs, discrete time, steps). dW The quantity used to calculate the product between a direction and the Hessian trough a finite differences algorithm; this parameter is passed to ann_BP_VHess function (see its man page for details). ex string representing a valid Scilab sequence. It is executed after the weight hypermatrix have been updated (i.e. after each epoch), using "execstr". This parameter is optional, default value: " " (empty string, i.e. does nothing). af The name of activation function to be used (string). This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. err_deriv_y The name of error function derivative with respect to network outputs (string). This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. Description This function performs training of a feedforward net using the conjugate gradients algorithm. The computation of Hessian is avoided by calculating directly the product between a direction and Hessian trough a finite difference approach which require just two error gradients (see function ann_BP_VHess). The error gradient is calculated across all training patterns at once given in x (respectively t). See Also ANN ANN_GEN ANN_FF ANN_FF_init ANN_FF_run ANN_FF_grad_BP ANN_FF_VHess scilab-ann-0.4.2.4/help/en_US/ann_FF_Hess.xml0000644000175000017500000000545111441407762021357 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Hess computes Hessian by finite differences. CALLING SEQUENCE H = ann_FF_Hess(x, t, N, W, dW, dW2 [,af ,ef]) PARAMETERS H The Hessian hypermatrix, have the same layout as W.*.W (not W.*.W', as usually found in literature); H(n1,i1,l1,n2.i2,l2) is the second derivative of error with respect to weights W(n1,i1,l1) and W(n2,i2,l2). x Matrix of input patterns, one pattern per column. t Matrix of target patterns, one pattern per column. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. dW, dW2 The quantities used to perturb each W parameter; dW is used for non-diagonal parameters, for diagonal parameters the supplemental perturbations quantities dW2 * dW are also used (note that very small values may easily lead to numerical instabilities) af The activation function to be used. This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. ef The error function to be used. This parameter is optional, default value "ann_sum_of_sqr", i.e. the sum-of-squares error function. Description This function calculates the Hessian trough a finite differences procedure. This process is very slow and is provided for testing purposes only. Internals: or off-diagonal elements: two weights are perturbed at +/- dW; for on-diagonal elements the perturbing quantities are: +/- (1 +/- dW2) * dW for one weight; the corresponding Hessian term is calculated from the four resulting errors (at four W points). See Also ANN ANN_GEN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_grad_nb.xml0000644000175000017500000000471411441407762022052 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_grad_nb error gradient trough finite differences CALLING SEQUENCE grad_E = ann_FF_grad_nb(x,t,N,W,dW[,af,ef]) PARAMETERS grad_E The error gradient, have same layout as W. x Input patterns, one per column. t Target patterns, one per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix. dW The quantity used to perturb each W parameter. af The activation function to be used. This parameter is optional, default value "ann_log_activ", i.e. the logistic activation function. ef The error function to be used. This parameter is optional, default value "ann_sum_of_sqr", i.e. the sum-of-squares error function. Description Calculates error gradient trough a (slow) finite difference procedure. Each element W(n,i,l) is changed to W(n,i,l)-dW then the error is calculated and the process is repeated for W(n,i,l)+dW. From the values obtained the partial derivative of the sum-of-squares error function, with respect to W(n,i,l), is calculated and the value of gradient returned. This process is very slow (compared to the backpropagation algorithms) so it is to be used only for testing purposes. This function is to be used on networks without biases. See Also ANN ANN_FF scilab-ann-0.4.2.4/help/en_US/ann_FF_SSAB_batch.xml0000644000175000017500000000277411441407762022353 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_SSAB_batch batch SuperSAB algorithm. CALLING SEQUENCE [W,Delta_W_old,Delta_W_oldold,mu]= ann_FF_SSAB_batch(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T[,mu,af,ex,err_deriv_y]) PARAMETERS This function have same parameters as ann_FF_SSAB_online except for: ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is batch SuperSAB. See Also ann_FF_SSAB_online scilab-ann-0.4.2.4/help/en_US/ann_FF_Std_online.xml0000644000175000017500000001007111441407762022545 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Std_online online standard backpropagation. CALLING SEQUENCE W = ann_BP_Std_online(x,t,N,W,lp,T[,af,ex]) PARAMETERS x Matrix of input patterns, one pattern per column. t Matrix of targets, one pattern per column. Each column have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_FF_init). lp Learning parameters [lp(1),lp(2)]. lp(1) is the well known learning parameter of standard backpropagation algorithm, W is changed according to the formula: W(t+1) = W(t) - lp(1) * grad E where t is the (discrete) time and E is the error. Typical values: 0.1 ... 1. Some networks train faster with even greater learning parameter. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. T The number of epochs (training cycles trough all pattern set). af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight hypermatrix have been updated, after each pattern (not whole set), using execstr. ex(2) - same as ex(1) - but is executed after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after online training with a given set of patterns. The algorithm used is online standard backpropagation. See Also ANN ANN_GEN ANN_FF ann_FF_init ann_FF_run scilab-ann-0.4.2.4/help/en_US/ann_FF_Mom_batch.xml0000644000175000017500000000276411441407762022352 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_Mom_batch batch backpropagation with momentum. CALLING SEQUENCE [W,Delta_W_old]= ann_FF_Mom_batch(x,t,N,W,lp,T[,Delta_W_old,af,ex,err_deriv_y]) PARAMETERS This function have same parameters as ann_FF_Mom_online except for: ex string representing a valid Scilab program sequence, executed after each epoch trough execstr. Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns, T times. The algorithm used is batch backpropagation with momentum. See Also ann_FF_Mom_online scilab-ann-0.4.2.4/help/en_US/ann_FF_SSAB_online_nb.xml0000644000175000017500000001236511441407762023232 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ann_FF_SSAB_online_nb online backpropagation with SuperSAB CALLING SEQUENCE [W,Delta_W_old,Delta_W_oldold,mu]= ann_BP_SSAB(x,t,N,W,lp,Delta_W_old,Delta_W_oldold,T[,mu,af,ex,err_deriv_y]) PARAMETERS x Matrix of input patterns, one pattern per column. t Matrix of targets, one pattern per column. Each row have a correspondent column in x. N Row vector describing the number of neurons per layer. N(1) is the size of input pattern vector, N(size(N,'c')) is the size of output pattern vector (and also target). W The weight hypermatrix (initialized first trough ann_FF_init_nb). lp Learning parameters [lp(1),lp(2),lp(3),lp(4),lp(5)]. lp(1) is the well known learning parameter of standard backpropagation algorithm, it is used only to initialize mu if not given (mu being optional) Typical values: 0.1 ... 1. Note that initial values may be strongly amplified in some circumstances. lp(2) defines the threshold of error which is backpropagated: a error smaller that lp(2) (at one neuronal output) is rounded towards zero and thus not propagated. Typical values: 0 ... 0.1. E.g. assume that output of neuron n have the actual output 0.91 and the target (for that particular neuron, given the corresponding input) is 1. If lp(2) = 0.1 then the error term associated to n is rounded to 0 and thus not propagated. lp(3) is the momentum parameter, used to "cancel" a previously wrong weight adaptation. Typical values: 0 ... 0.9999... (smaller than 1). lp(4) is the adaptive increasing factor. Typical values: 1.1 ... 1.3. lp(5) is the adaptive decreasing factor. Typical values: lp(5) = 1/lp(4) (lp(5) < 1). Delta_W_old The previous weight adjusting quantity. On first call this parameter should be initialized to zero using e.g. Delta_w_old = hypermat(size(W)') On subsequent calls to ann_FF_SSAB_online_nb you should give the value of Delta_W_old returned by the previous call. Delta_W_oldold The weight adjusting quantity used two steps back. Same remarks as for Delta_W_old (above) apply. T The number of epochs (training cycles trough all pattern set). mu This is the hypermatrix of learning constants who replaces lp(1) and is adapted at each step. This parameter is optional, default value is "mu = lp(1)*hypermat(size(W)',ones(prod(size(W)'),1))", i.e. it is initialized uniformly to lp(1). af Activation function and its derivative. Row vector of strings: af(1) name of activation function. af(2) name of derivative. Warning: given the activation function y=f(x), the derivative have to be expressed in terms of y, not x. This parameter is optional, default value is "['ann_log_activ', 'ann_d_log_activ']", i.e. logistic activation function and its derivative. err_deriv_y the name of error function derivative with respect to network outputs. This parameter is optional, default value is "ann_d_sum_of_sqr", i.e. the derivative of sum-of-squares. ex two-dimensional row vector of strings representing valid Scilab sequences. ex(1) is executed after the weight matrix have been updated, after each pattern (not whole set), using execstr. ex(2) - same as ex(1) - but is executed after each epoch. This parameter is optional, default value is [" "," "] (do nothing). Description Returns the updated weight hypermatrix of a feedforward ANN, after training with a given set of patterns. The algorithm used is online backpropagation with SuperSAB. This function is to be used on networks without biases. See Also ANN ANN_GEN ANN_FF ann_FF_init_nb ann_BP_run_nb scilab-ann-0.4.2.4/help/en_US/ANN.xml0000644000175000017500000000342011441407762017654 0ustar sylvestresylvestre $LastChangedDate: 2008-03-26 09:50:39 +0100 (mer., 26 mars 2008) $ ANN Toolbox for neural networks Description This ANN toolbox is designed to provide ANN engines for exploration and easy prototyping.This man page gives an overview. The above main objective have influenced the overall design, i.e. (1) this toolbox will not be translated to C or Fortran for speed (see also the sci2for function), (2) except for the main algorithms (described in various books, e.g. see mine "Matrix ANN", it is heavily commented and explained such that it will be easy to adapt to particular needs if necessary, (3) at some points clarity was preferred over speed, when this do not lead to large speed loss. COMPONENTS General support functions See ANN_GEN for support functions for ANN. Feedforward networks See ANN_FF for detailed description. See Also ANN_GEN ANN_FF