Multilayer Networks

• Multiple layers of linear units still produce only linear functions
• Perceptrons have a discontinuous threshold which is undifferentiable and therefore unsuitable for gradient descent
• We want a unit whose output is a nonlinear, differentiable function of the inputs
• One solution is a sigmoid unit
• Like perceptrons it computes a linear combination of its inputs and then applies a threshold to the result. But the threshold output is a continuous function of its input which ranges from 0 to 1.
• It is often referred to as a squashing function.