‘Learning’ the Stochastic Gradient Descent Algorithm

Cartoonish Depiction of Gradient Descent — optimizing a set of weights and minimizing the loss function
  • covering more broad topics next time,
  • reading over applications,
  • practicing more problems..etc.

Before we add the word “Stochastic”

Before we dive into what Stochastic Gradient Descent is, lets take a look at what the term “Gradient Descent” means.

Cost Function

We can use a Cost Function to analyze the model’s ability to understand and learn patterns and relationships between inputs and outputs. A cost function is usually a function of the error difference, and our goal in machine learning problems is to MINIMIZE the cost function. There are many cost functions we can use to optimize models. Examples are Mean Squared Error (MSE), Mean Absolute Error (MAE), and a lot of other more math-heavy trickier ones :)

The Gradient and the Descent

We can first set arbitrary values to our weights, for instance, starting from 0. Then, from that point, we need to find out which direction to go towards in order to reach the minimum value of the cost function. The gradient of the cost function tells us exactly that. The gradient, also denoted with a upside down triangle, represents the slope of the function with respect to each weight. The gradient of the cost function is basically a vector of partial derivatives with respect to each weight (ex. w1, w2,..etc).

Back to the ‘Stochastic’

Ok, so now that we looked into Gradient Descent, what does the term “Stochastic” mean? Stochastic is all about randomness. What can we do to this algorithm that involves randomness though?

The Circle Maps represent the cost function, from the top-eye perspective or point of view. The path to finding the minimum point of the loss function looks very different for Gradient Descent and SGD.

Applications

The Stochastic Gradient Descent algorithm is one of the most used optimization algorithms in Machine Learning. It is very popular in deep learning and neural networks as well!

--

--

Hello! I’m a student at the University at Texas in Austin. Welcome to my collection of thoughts. I like to write and blog. rushiblogs.weebly.com

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aarushi Ramesh

Hello! I’m a student at the University at Texas in Austin. Welcome to my collection of thoughts. I like to write and blog. rushiblogs.weebly.com