Stochastic Gradient Descent

• Approximate gradient descent search by updating weights incrementally, following the calculation of the error for each individual example
• delta rule: (same as LMS algo, but only similar to perceptron training rule)
• error function:
• If is sufficiently small, stochastic gradient descent (SGD) can be made to approximate true gradient descent (GD) arbitrarily closely