Gradient Descent in Visual Basic fitting a Quadtratic Function

In the last tutorial, I demonstrated gradient descent that fitted a linear function to the data. Now I will post an example of it fitting a quadratic function. The data set is as folllow (y = x^2 - 4) Note that the below is code for Form_Load (You will need one Command button, two listboxes, one Label, and one Picture for the code to run. Set the Picture width height and scalewidth and scaleheight to 5000)

List1.AddItem ("2")
List1.AddItem ("3")
List1.AddItem ("4")
List1.AddItem ("5")
List1.AddItem ("6")
List1.AddItem ("7")
List1.AddItem ("8")
List1.AddItem ("9")
List1.AddItem ("10")

List2.AddItem ("0")
List2.AddItem ("5")
List2.AddItem ("12")
List2.AddItem ("21")
List2.AddItem ("32")
List2.AddItem ("45")
List2.AddItem ("60")
List2.AddItem ("77")
List2.AddItem ("96")


Label1.Caption = "Click button to run gradient descent"
Command1.Caption = "Click here"
Form1.Caption = "Gradient Descent"

Notice how the data fits the function above. Hopefully gradient descent will pick up on this. The key change from the previous example is changing the minimization function.

Here is the code for Command1_Click (when clicked, will run and graph gradient descent)

Private Sub Command1_Click()
Dim i, j, m As Integer
Dim x(24) As Long
Dim y(24) As Long
Dim yIndex As Long
Dim theta0, temp0, theta1, temp1, theta2, temp2 As Long


theta0 = 0 'initialize all variables to 0
theta1 = 0
theta2 = 0
temp0 = 0
temp1 = 0
theta2 = 0

m = List1.ListCount - 1 'number of samples
yIndex = 50

For i = 0 To List1.ListCount - 1 'populate arrays with data
    x(i) = CLng(List1.List(i))
    y(i) = CLng(List2.List(i))
Next

Dim iterations As Long
iterations = 168000 'set iterations high

Dim alpha As Double
alpha = 0.001 'set learning rate

Dim h0, h1, h2 As Long
h0 = 0
h1 = 0

For i = 0 To iterations
    h0 = 0
    h1 = 0
  
    For j = 0 To m
    h0 = h0 + ((theta0 + x(j) * x(j) * theta1) - y(j))
    h1 = h1 + ((theta0 + x(j) * x(j) * theta1) - y(j)) * x(j)
    Next j
  
    temp0 = theta0 - (alpha * h0) / CDbl(m)
    temp1 = theta1 - (alpha * h1) / CDbl(m)
    theta0 = temp0
    theta1 = temp1
   
Next i

Label1.Caption = CStr(theta1) & "x^2 + " & CStr(theta0)

Dim xGrid As Long
Dim yGrid As Long
Dim c, d As Long
xGrid = Picture1.Width / yIndex
yGrid = Picture1.Height / yIndex
c = 0
d = 0
''Set up grid
For i = 0 To Picture1.Height
   If (c > xGrid) Then
        For j = Picture1.Width / 2 - xGrid / 2 To Picture1.Width / 2 + xGrid / 2
            Picture1.PSet (j, i)
        Next j
    c = 0
    End If
    c = c + 1
Next i

For i = 0 To Picture1.Width
    If (d > yGrid) Then
        For j = Picture1.Height / 2 - yGrid / 2 To Picture1.Height / 2 + yGrid / 2
            Picture1.PSet (i, j)
        Next j
    d = 0
    End If
    d = d + 1
Next i

For i = 0 To Picture1.Width
    Picture1.PSet (i, Picture1.Height / 2)
Next

For i = 0 To Picture1.Height
    Picture1.PSet (Picture1.Width / 2, i)
Next i


Picture1.DrawWidth = 2
Dim yPos, counter As Double
yPos = 0
counter = 0#
For i = 2500 To 5000
    yPos = (theta1 * counter) ^ 2 + (theta0 * xGrid / 100)
    Picture1.PSet (i, Picture1.Height / 2 - yPos * xGrid)
    counter = counter + 0.01
Next i

counter = 0#
For i = 2500 To 0 Step -1
    yPos = (theta1 * counter) ^ 2 + (theta0 * xGrid / 100)
    Picture1.PSet (i, Picture1.Height / 2 - yPos * xGrid)
    counter = counter + 0.01
Next i
End Sub

The part of interest is here:

For i = 0 To iterations
    h0 = 0
    h1 = 0
  
    For j = 0 To m
    h0 = h0 + ((theta0 + x(j) * x(j) * theta1) - y(j))
    h1 = h1 + ((theta0 + x(j) * x(j) * theta1) - y(j)) * x(j)
    Next j
  
    temp0 = theta0 - (alpha * h0) / CDbl(m)
    temp1 = theta1 - (alpha * h1) / CDbl(m)
    theta0 = temp0
    theta1 = temp1
   
Next i

Label1.Caption = CStr(theta1) & "x^2 + " & CStr(theta0)


h0 and h1 are set as the minimization of a quadratic function. This is what gives us a quadratic solution. It fits the data much better. The bottom part is complicated as I didn't use a graphing tool but instead wrote a simple graph myself. Playing around with the input data set can give interesting results.