Artificial Neural Network

An artificial neural network (ANN) is a class of models, inspired by the central nervous system, used in machine learning and pattern recognition and classification. These models are nonlinear parametric regression models with either automatic, unsupervised training (setting of the model parameters) or supervised training from some training set of known input output relations, depending on the type of network.

An ANN consists of a collection of neurons (processing units) and connections between these neurons. Usually these neurons accumulate “information” from neighboring neurons and either fire or not, depending on some local threshold level.

The simplest type of ANN is the feed-forward net, in which all the information flows in a single direction. Figure 1 shows a four-layer feed-forward net, with an input layer, followed by two hidden layers, followed in turn by an output layer.

Figure 1 A Simple Four-Layer Feed-Forward Net

Many neurons are modeled (and constructed or simulated) to have binary outputs (and often binary inputs as well). Each neuron has some rule (called a [Page 47]firing rule) for deciding which combination of inputs results in which output. One particularly simple rule would be to sum all the inputs (multiplied perhaps by weights) and check to see if the sum was more than some threshold value; if so, then fire, and if not, then do not fire (a binary output). Notice that this firing rule is discontinuous in that it has a sudden jump at the threshold. This rule is sometimes called the hardlimit firing rule.

As a simple example (for the neuron illustrated in Figure 2), let w 1 = 0.3, w 2 = 0.5, w 3 = 0.3, and the threshold q = 1.5. Then for the inputs 1.2, 2.1, and 0.7, we get accumulated sum (.3)(1.2) + (.5)(2.1) + (.3)(0.7) = 1.62 > 1.5, so the neuron would fire.

The parameters of a feed-forward net include the weights and thresholds for each neuron. These parameters must be set during the training phase of the network. During training (a supervised learning situation), one uses a set of known input-output relationships to set the parameters of the network. One common technique for training is the backpropagation algorithm. This algorithm basically computes a gradient of the error with respect to the weights in order to adjust the weights. The computation proceeds by propagating influences backwards in the network (and hence the name).

Figure 2 A Single Neuron

Many other kinds of firing rules are possible, including a sigmoid (a continuous version of the hardlimit rule), Gaussian rules, and others. The sigmoid firing rule has the functional form

Many other kinds of connection topologies are possible. A Hopfield net, for example, is a neural network with bidirectional connections (and with the same weight in each direction). These networks act as associative memories in the sense that the network can store a set of patterns and fire if a similar pattern is presented to it.

The presence of loops in a network allows for feedback, so these types of networks are sometimes called recurrent networks.

Many popular software systems, including Matlab, SPSS, SAS, and R (a open source version of S), have neural network toolboxes. It is also fairly easy to get software packages (as either executables or source) from the Internet to simulate various neural networks; one such place is http://www.neural-networks-at-yourfingertips.com/.

...