What do neural networks have to do with real neurons?
Hi there guys and girls, for once again, I am publishing a new post after a long period of inactivity. Of course, the time I didn’t dedicate to writing posts was invested into some academic research which has left me with many thoughts, some of which I’d like to share with you. Today’s topic is neural networks and their relation with real neurons and intelligence.
First of all, by neural networks I am referring to artificial neural networks. What exactly artficial neural networks are? They are a computational model of real neurons and an integral part of the disciplines of machine learning and computational intelligence. They first appeared in 1944 by McCulloch and Pitts (also more at McCulloch-Pitts Neurons). The main idea that neural networks were based upon at the time, was that real neurons work in a binary way. A neuron can send a signal (also called a spike) or not. A neuron’s spike is never modulated according to the strength of the signal. A neuron either fires, or it doesn’t. There is not such thing as firing "very much" or firing "weakly" (this is called the all-or-none law).
The picture above makes it more clear. On the horizontal time we have time and on the vertical axis voltage. The neuron is at a resting state at time 0. There is a certain point called a threshold. Once the input voltage the neuron accepts gets pass this threshold, then the neuron fires and sends an action potential. As long as it has fired it returns to a resting state.
Of course this is not as simple as it seems. The action potential is transmitted in an electrochemical way via neurotransmitters. But this is the general concept of a neuron, and, since we are talking about the first artificial neuron model here, we shouldn’t expect anything more complicated in 1944.
So, McCulloch and Pitts proposed the first model of an artificial neurons. The model was the following:
McCulloch-Pitts neuron architecture
A neuron accepts a number of inputs n. Each input is modified by a weight w. Then, all the inputs are summed and get through an activation function. If the output of the function gets over the threshold the neuron fires. In the original model the activation function was the Heaviside function and the threshold was 0. The Heaviside function gives one for any input over zero.
The Heaviside function
Anyway, this was pretty much the whole idea. At the beginning, many people believed that neural networks could be used as the basis for any computation or as a means of memory storage in computers. The direct analogy between the binary nature of the neuron and that of the computers reinforced these ideas. In addition, it was thought that maybe neural networks could provide the means to build computers that could mimic human intelligence. However, all this was simply proven wrong.
Neural networks faded for the next decades and resurfaced in the 80s. The reasons for the fade out were some technical problems that neural networks faced and the re-appearance was due to the solution of these problems (the most famous is the XOR problem). We we won’t delve any further into technical issues (go here for more information: http://en.wikipedia.org/wiki/Feedforward_neural_network). Anyway, through the 80s neural networks have been successfuly applied to a multitude of problems like image recognition, sound recognition, classification and regression. What neural networks actually do, to put it in simpler terms, is pattern recognition. They can be trained to some patterns and then check if some other entity matches to that pattern. A popular task, for example, is face recognition.
A feedforward neural network
To recognize a face the network pictured above would do the following. First the input layer would receive the image. Then the input layer would send its signal to the hidden layer which would process the input and send it to the output layer. In the output layer, each neuron would represent one image from a given set, and the neuron with the highest activation would signal this particular image. For example, if neuron 2 had the highest activation of all neurons, then the network recognizes the input as image 2. This is a simplified example. If you are interested in this kind of application of neural networks visit Computer Programs that Can See where you can find some very instructive information on the subject.
Now, what do neural networks have to do with real neurons and with intelligent systems in general? First of all, it is important to state, that neural networks, besides being very useful tools, are also the main component of the theory of mind called connectionism. Connectionism is known also as the opposite theory of computationalism.
According to computationalists, the mind is like a computer and works by processing symbols. According to connectionists, the mind can be modelled by a network, where information in the network is not described by any specific entity, but it is rather distributed in the network and its connections. The connections of the network are a direct analog of the synapses of real neurons (for more information on synapses you can visit: Introduction to Neurons, Synapses, Action Potentials, and Neurotransmission).
As you saw, the first neural network was based on a very loose idea of how the brain really works. Neural networks can be seperated into three generations. The first one was the one we introduced. The second one, is the most popular, and has few differences from the first. The main difference is that in the second generation, neural networks use a continuous activation function and not a discrete one like the Heaviside step function. The most popular choice is the hyperbolic tangent function.
Hyperbolic tangent function
What is interesting is that the second generation that was more powerful than the first introduced an extra element of realism. The continuous activation functions are thought to represent the firing rate of neurons. For most of neuroscience’s history it was believed that neurons transmitted information through the use of the firing rate. This is now believed to be wrong. However, it is very interesting how an extra element of realism to what is otherwise an abstraction over the nervous system can provide it with extra functionality.
The inspiration for the introduction of continuous functions over discrete ones was mainly mathematics and not biology. However, the fact described in the paragraph above, was observed by scientists working in the field of the third generation of neural networks also known as spiking neural networks.
Spiking neural networks give an even greater role in biological realism. Underlying the structure of spiking neural networks is the assumption that neurons do not transmit information through the firing rate, but through some other information coding scheme. The modelling of the spiking neural networks usually includes the modelling of the action potential of a single neuron and, usually, realism is a very desirable element, even though there are models that range from the utterly realistic to the more abstract. A very well known model that has been implemented as a spiking neural network is the Hodgkin-Huxley model of the neuron, which is considered the best model of a neuron thus far, and for which, Alan Lloyd Hodgkin and Andrew Huxley won a nobel in 1952. Spiking neural networks are said to be even more powerful than second generation neural networks, even though the current state of research is still not up to par with the research done with second generation neural networks.
Alan Hodgkin and Andrew Huxley
So, to revisit the question in the title of this article, what do neural networks have to do with real neurons? Why each level of biological realism brings more power to the artificial neural network? McCulloch and Pitts believed that neurons were simple binary machines. Using this assumption they managed to get some results, even though nothing close to human intelligence. However, neurons also work in an analog way, since they transmit electrochemical signals. By making adjustments to the original model of McCulloch and Pitts we created the second generation of neural networks, that could perform some of the tasks that humans can perform. However, they are still not up to par with human intelligence. The third generation of neural networks promises even greater capabilities.
So, at which point do we need to stop imitating the neurons’ architecture? Up to which level of realism do we need to go to simulate the human brain? What are the main components of the neuron and the brain that we must copy in order to reach the functionality of the brain?
That’s the question that the new generation of scientists will have to answer.
More sources:
Pulsed neural networks: A book on spiking neural networks
An introduction to neural networks: Self-explanatory. A very nice e-book to help you learn the basics of the field.
Connectionism: By Stanford encyclopedia of philosophy
