Logistic regression in Java with plotting

This tutorial will show how to implement an easy example of logistic regression in Java. It assumes that JavaPlot is installed. See my other tutorial (link at bottom of page) or google how to import Javaplot (it is trivial).

The major change being made here from linear regression is the hypothesis function uses the sigmoid function.

Here is the data set we will be using:

 double x1[] = {0.0, 0.5, 1.0, 1.5, 2.0, 0.1, 3.0, 3.1, 3.5, 3.2, 2.5, 2.8};   
 double x2[] = {0.0, 1.0, 1.1, 0.5, 0.3, 2.0, 3.0, 2.3, 1.5, 2.2, 3.6, 2.8};   
 double y[]  = {0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};

Here is a graph of the data set. We are trying to determine the boundary line, which is a line that will "split" the two groups in half.

The slight change we make will be too call the sigmoid function during gradient descent. We take the proper partial derivatives and step through m amount of examples. Here is the relevant code:

for(i = 0; i < iterations; i++)
            {
         
                h0 = 0.0;
                h1 = 0.0;
                h2 = 0.0;
               
                for(j = 0; j<m; j++)
                {
                    h0 = h0 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j]);
                    h1 = h1 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j])*x1[j];
                    h2 = h2 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j])*x2[j];
                }    
        temp0 = theta0 - (alpha*h0)/(double)m;
        temp1 = theta1 - (alpha*h1)/(double)m;
        temp2 = theta2 - (alpha*h2)/(double)m;
        theta0 = temp0;
        theta1 = temp1;
        theta2 = temp2;

}

Note that the equation we get is in two variables, x1 and x2. We will find the line by setting each to 0 and finding the intercepts. From there, we use the equation for a line to determine our boundary line. This is done in order to plot the line.

The method sigmoid(double x) is pretty straightforward. The function is rather complex but the explanation for it isn't hard to understand. See the machine learning course notes for coursera for the explanation. The code is:

private static double sigmoid(double x)
    {
        return 1 / (1 + Math.exp(-1*x));
    }
   

The resulting function is two variable as I said before. First, we need to set x2 = 0 and then x1 = 0. When those are the cases, we are left with bother theta1*x1 = theta0 and theta2*x2 = theta0. The code is as follows:

double point1 = -1 * theta0 / theta1; //x intercept

double point2 = -1 * theta0 / theta2; //y intercept

Now using these points, we can find a line using the equation for a line which is:

y - y1 = m(x-x1)

Our points are (0, point2) and (point1, 0)

Subbing in the values and isolating y we get the function for the line:

String outputFunction = String.valueOf(String.valueOf((-1*point2)/point1) + "*x+" + String.valueOf(point2));

The slope m is negative because theta0 has to be brought to the other side. Working it out on paper might make it clearer.

Here is a plot of the resulting line:

A good fit

Here is the full code. I added a plot of the cost function for the first 1000 iterations to check that it is going down. JavaPlot must be added for the plot to work. I discussed that in my post here

import com.panayotis.gnuplot.JavaPlot;
import com.panayotis.gnuplot.style.PlotStyle;
import com.panayotis.gnuplot.style.Style;

public class GradDescent {

    public static void main(String[] args) {
   
       
         double x1[] = {0.0, 0.5, 1.0, 1.5, 2.0, 0.1, 3.0, 3.1, 3.5, 3.2, 2.5, 2.8};   
         double x2[] = {0.0, 1.0, 1.1, 0.5, 0.3, 2.0, 3.0, 2.3, 1.5, 2.2, 3.6, 2.8};   
         double y[]  = {0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0};
         int counter = 0;
       
         JavaPlot p = new JavaPlot();
       
         /*number of examples*/
         int m = 12;
       
          /*thetas and temp thetas*/
            double theta0, temp0, theta1, temp1, theta2, temp2;
            theta0 = 0.0; temp0 = 0.0;
            theta1 = 0.0; temp1 = 0.0;
            theta2 = 0.0; temp2 = 0.0;
         
        /*# of iterations and learning rate*/
            int iterations = 2181800;
            float alpha = 0.009f;

            int j = 0;
            double h0 = 0.0;
            double h1 = 0.0;
            double h2 = 0.0;
           
            int i = 0;
            for(i = 0; i < iterations; i++)
            {
         
                h0 = 0.0;
                h1 = 0.0;
                h2 = 0.0;
               
                for(j = 0; j<m; j++)
                {
                    h0 = h0 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j]);
                    h1 = h1 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j])*x1[j];
                    h2 = h2 + (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2) - y[j])*x2[j];
                }    
        temp0 = theta0 - (alpha*h0)/(double)m;
        temp1 = theta1 - (alpha*h1)/(double)m;
        temp2 = theta2 - (alpha*h2)/(double)m;
        theta0 = temp0;
        theta1 = temp1;
        theta2 = temp2;
       
        counter = counter + 1;
        if(counter < 1000)
        {
            for(j = 0; j<m; j++)
            {
                h0 = h0 + y[j]*Math.log(sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2)) + (1-y[j])*(1 - (sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta2)));                                                            //+ Math.pow(( sigmoid(theta0 + x1[j]*theta1 + x2[j]*theta1) - y[j]), 2.0);
            }
            h0 = (h0 / m) * -1;
            float[][] cost = {{(float)counter, (float) h0}};
            p.addPlot(cost);
       
            System.out.println("Cost at " + counter + " is " + h0);
           
        }
       
        }
         
       p.plot();
       System.out.println(theta2 + "x2 + " + theta1 + "x1 + " + theta0);
      

    }
   
   
    private static double sigmoid(double x)
    {
        return 1 / (1 + Math.exp(-1*x));
    }
   
    private static void testGradientDescent(float n, double theta0, double theta1, double theta2)
    {
         double x3[][] = {{0.0, 0.0}, {0.5, 1.0}, {1.0, 1.1}, {1.5, 0.5}, {2.0, 0.3}, {0.1, 2.0}};   
         double x4[][] = {{3.0, 3.0}, {3.1, 2.3},{3.5, 1.5}, {3.2, 2.2}, {2.5,3.6}, {2.8, 2.8}};  
       
        JavaPlot p = new JavaPlot();
        p.set("title", "'Gradient Descent'");
        p.set("xrange", "[0:4]");
        //p.addPlot(outputFunction);
       
        p.addPlot(x3);
        p.addPlot(x4);
       
        double point1 = -1 * theta0 / theta1;
        double point2 = -1 * theta0 / theta2;
       
        String outputFunction = String.valueOf(String.valueOf((-1*point2)/point1) + "*x+" + String.valueOf(point2));
        System.out.println("Plotting " + outputFunction);
       
        PlotStyle myPlotStyle = new PlotStyle();
        myPlotStyle.setStyle(Style.LINES);
        myPlotStyle.setLineWidth(2);
       
        double boundaryLine[][] = {{point1, 0},{0, point2}};
        //p.addPlot(boundaryLine);
        p.addPlot(outputFunction);
        p.plot();
       
    }

}

Checking if gradient descent is working

This is a continuation on my post on how to run gradient descent using a quadratic fitting function.

One way to check if gradient descent is working is to compute the cost function and see if it is going down given theta values that have been being updated. The way I did this was to check during gradient descent every 1000 iterations. I computed the cost function, and if it is going down, then that means gradient descent is converging. Here is the code I put in the iteration loop:

 counter = counter + 1;
  if(counter % 1000 == 0)
        {
            for(j = 0; j<m; j++)
            {
                h0 = h0 + Math.pow(((theta0 + (x[j]*x[j]*theta1)) - y[j]), 2.0);
            }
            h0 = h0 / (2.0 * m);
            System.out.println("Cost at " + counter + " is " + h0);
        }

The first thing I did was have a variable that increments by one every iteration. I now notice that I could simply use the iteration variable that is in the loop but whatever. The if statement says that if counter is divisible by 1000, then do the following code. This is because I have 30000 iterations so I want to check every 1000 iterations that it is going down.

The code below computes this function:

 

 The code that can perform this is as follows:

for(j = 0; j<m; j++)
            {
                h0 = h0 + Math.pow(((theta0 + (x[j]*x[j]*theta1)) - y[j]), 2.0);
            }
            h0 = h0 / (2.0 * m);

As you can see, it simply calculating the summation of the squared error where h(x) is equal to:

 theta0 + (x[j]*x[j]*theta1)

 and then dividing by 2*m

If this number goes down, that means that gradient descent is converging.

 

Improving the fit of gradient descent using better functions

This is a continuation of my previous post simple gradient descent example using Codeblocks-EP in C language

Here is an example of trying to fit a linear function "y = theta1*x + theta0" to a data set that mimics
y = x^2

The data set:

float x[] = {1,2,3,4,5,6,7,8,9,10};
float y[] = {1,4,9,16,25,36,49,64,81,100};

As you can see, the training set suggests that the data should follow y=x^2

However, when we plot it using a linear function, the plot is wildly inaccurate for most values of x. It is a terrible fit:

Not good. But now let's change our hypothesis function to a quadratic function and see if we get a better fit. The code as of now is currently:

for(j = 0; j<m; j++)
            {
                h0 = h0 + ((theta0 + x[j]*theta1) - y[j]);
                h1 = h1 + ((theta0 + x[j]*theta1) - y[j])*x[j];
            }

Here we are going through all of our training examples and updating them using a linear function. By adding another x[j] term, we can hopefully get a quadratic fit that more closely fits the data. So we will replace it with:

for(j = 0; j<m; j++)
            {
                h0 = h0 + ((theta0 + x[j]*x[j]*theta1) - y[j]);
                h1 = h1 + ((theta0 + x[j]*x[j]*theta1) - y[j])*x[j];
            }

Now let's see the plot and values for theta0 and theta1:

A much better fit. Note that the learning rate had to be changed as well as the iterations. We went from:

int iterations = 1000;

float alpha = 0.03;

to an increasre in iterations and a decrease in the learning rate (at this value, it fails to converge)

int iterations = 10000;

float alpha = 0.003;

Here is the full code:

#include "koolplot.h"

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main(int argc, char**argv) {

    /*training set1.2
    1.1
    2.4 2.2
    3.1 3.3
    4.4 4.3
    5.3 5.4
    6.3 6.7
    7.6 7.7
    8.7 8.6
    9.7 9.8*/
    float x[] = {1,2,3,4,5,6,7,8,9,10};
    float y[] = {1,4,9,16,25,36,49,64,81,100};

    /*number of examples*/
    int m = 10;

    /*thetas and temp thetas*/
        float theta0, temp0, theta1, temp1;
        theta0 = 0.0; temp0 = 0.0;
        theta1 = 0.0; temp1 = 0.0;

    /*iterations and learning rate HIGHLY VARIABLE*/
        int iterations = 1000;
        float alpha = 0.03;

        int j = 0;
        float h0 = 0.0;
        float h1 = 0.0;
        int i = 0;
        for(i = 0; i < iterations; i++)
        {

            h0 = 0.0;
        h1 = 0.0;
            for(j = 0; j<m; j++)
            {
                h0 = h0 + ((theta0 + x[j]*x[j]*theta1) - y[j]);
                h1 = h1 + ((theta0 + x[j]*x[j]*theta1) - y[j])*x[j];
            }

    //h0 = h0 / (float)m;
    //h1 = h1 / (float)m;
    temp0 = theta0 - (alpha*h0)/(float)m;
    temp1 = theta1 - (alpha*h1)/(float)m;
    theta0 = temp0;
    theta1 = temp1;
    }

    printf("Theta0: %f\n", theta0);
    printf("Theta1: %f\n", theta1);

    Plotdata xx(0.0, 10.0), yy = theta0 + xx*xx*theta1;
    plot(xx, yy);

    return 0;
}

Simple gradient descent example using Codeblocks-EP in C language

The intention of this tutorial is to show you how to set up Codeblocks-EP and use the plotting software called KoolPlot to run a simple gradient descent example and plot it. This is workable in Windows.

First, download CodeBlocks-EP at http://codeblocks.codecutter.org/

Note that this is a different program than the regular CodeBlocks but I am pretty sure it has the same functionality. It seems to be a version to learn from. Install it and open it.

Step 1: Create a new project

Step 2: Select "koolplot project" and click "Go"

Step 3: Select "Add Console" It is not the default so make sure that you select it and then hit next.

Step 4: Name your project "GradientDescent" and click next. On the next screen, click finish.

Step 5: Go to File -> New -> Empty file. Name it GradientDescent.c

Step 6: Right click and select edit and paste the following code:

#include "koolplot.h"

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

int main(int argc, char**argv) {

    /*training set1.2
    1.1
    2.4 2.2
    3.1 3.3
    4.4 4.3
    5.3 5.4
    6.3 6.7
    7.6 7.7
    8.7 8.6
    9.7 9.8*/
    float x[] = {1.2, 2.4, 3.1, 4.4, 5.3, 6.3, 7.6, 8.7, 9.7};
    float y[] = {1.1, 2.2, 3.3, 4.3, 5.4, 6.7, 7.7, 8.6, 9.8};

    /*number of examples*/
    int m = 9;

    /*thetas and temp thetas*/
        float theta0, temp0, theta1, temp1;
        theta0 = 0.0; temp0 = 0.0;
        theta1 = 0.0; temp1 = 0.0;

    /*iterations and learning rate HIGHLY VARIABLE*/
        int iterations = 1000;
        float alpha = 0.03;

        int j = 0;
        float h0 = 0.0;
        float h1 = 0.0;
        int i = 0;
        for(i = 0; i < iterations; i++)
        {

            h0 = 0.0;
        h1 = 0.0;
            for(j = 0; j<m; j++)
            {
                h0 = h0 + ((theta0 + x[j]*theta1) - y[j]);
                    h1 = h1 + ((theta0 + x[j]*theta1) - y[j])*x[j];
            }

    //h0 = h0 / (float)m;
    //h1 = h1 / (float)m;
     temp0 = theta0 - (alpha*h0)/(float)m;
     temp1 = theta1 - (alpha*h1)/(float)m;
     theta0 = temp0;
    theta1 = temp1;
    }

    printf("Theta0: %f\n", theta0);
    printf("Theta1: %f\n", theta1);

    Plotdata xx(0.0, 3.0), yy = theta0 + xx*theta1;
    plot(xx, yy);

    return 0;
}

 Step 7: Select Build -> Build and then Build -> Run

Lastly, if you followed all of these steps it should run. The command console will show the values of theta0 and theta1 and the plot with plot out "y = theta0 + x*theta1" Note that I had to use "xx" and "yy" at the end because these are the variables that koolplot is using to plot and x and y were already used to store our training data. If you click on the plot, you will be able to see the x and y values above. Here is what the output should look like:

And that's it.

 

Multi-feature gradient descent in java

public class GDescent {

    public static void main(String[] args) {
  
         float x1[] = {1f,1f,1f,1f,2f,2f,2f,3f,3f,3f};  
         float x2[] = {6f,6f,6f,6f,9f,9f,9f,13f,13f,13f};
         float y[] = {2f,2f,3f,4f,5f,5f,6f,5.5f,6f,7f};  
      
         /*number of examples*/
         int m = 10;
      
          /*thetas and temp thetas*/
            float theta0, temp0, theta1, temp1, theta2, temp2;
            theta0 = 0.0f; temp0 = 0.0f;
            theta1 = 0.0f; temp1 = 0.0f;
            theta2 = 0.0f; temp2 = 0.0f;
        
        /*# of iterations and learning rate*/
            int iterations = 3430000;
            float alpha = 0.003f;

            int j = 0;
            float h0 = 0.0f;
            float h1 = 0.0f;
            float h2 = 0.0f;
            int i = 0;
            for(i = 0; i < iterations; i++)
            {
        
                h0 = 0.0f;
                h1 = 0.0f;
                h2 = 0.0f;
                for(j = 0; j<m; j++)
                {
                    h0 = h0 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j]);
                    h1 = h1 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j])*x1[j];
                    h2 = h2 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j])*x2[j];
                }   
        temp0 = theta0 - (alpha*h0)/(float)m;
        temp1 = theta1 - (alpha*h1)/(float)m;
        temp2 = theta2 - (alpha*h2)/(float)m;
        theta0 = temp0;
        theta1 = temp1;
        theta2 = temp2;
        }
          
       System.out.println("" + theta2 + "x2 + " + theta1 + "x1 + " + theta0);
     
       testGradientDescent(2f, theta0, theta1, theta2);

    }
  
    private static void testGradientDescent(float n, float theta0, float theta1, float theta2)
    {
        float result = theta0 + (theta1*1) + (theta2*6);
        //float x[] = {1f,2f,3f,4f,5f,6f,7f,8f,9f,10f};  
        //float y[] = {2f,5f,10f,17f,26f,37f,50f,65f,82f,101f};
        //int marks[][]={{1,2},{2,5},{3,10},{4,17},{5,26},{6,37},{7,50},{8,65},{9,82},{10,101}};
      
        System.out.println("Result: " + result);
        //String outputFunction = String.valueOf(String.valueOf(theta0)  + "+" + String.valueOf(theta2) + "*x*x");
        //System.out.println("Plotting " + outputFunction);
        //JavaPlot p = new JavaPlot();
        //p.addPlot(marks);
      
      // // p.addPlot(outputFunction);
       // p.setPersist(true);
       // p.plot();
        //p.setPersist(true);
      
    }

}

Notice that the partial derivatives are calculated here:

h0 = h0 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j]);
h1 = h1 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j])*x1[j];
h2 = h2 + ((theta0 +  x1[j]*theta1 + x2[j]*theta2) - y[j])*x2[j];

Nibbler - Pacman/Snake game in Java+Libgdx+Universal Tween Engine

Prototype of the game (with terrible graphics) can be downloaded here

Source code can be downloaded here

Nibbler game tutorial - Java, libgdx, Universal Tween Engine Part 2

Welcome to part 2 of this tutorial. By the end of today we will have the basic game loop set up. First, double click on "DesktopLauncher.java" and replace the code with this:

package com.mygdx.nibbler.desktop;

import com.badlogic.gdx.backends.lwjgl.LwjglApplication;
import com.badlogic.gdx.backends.lwjgl.LwjglApplicationConfiguration;
import com.mygdx.nibbler.nibblergame;

public class DesktopLauncher {
    public static void main (String[] arg) {
        LwjglApplicationConfiguration config = new LwjglApplicationConfiguration();
        config.title = "Nibbler";
        config.width = 480;
        config.height = 640;
        new LwjglApplication(new nibblergame(), config);
    }
} 

All this is doing is changing the size of the window when we run the game on Windows. We want the window to be more of a "portrait" than a "landscape."

Now go back to the "core" directory and open "nibblergame.java" Replace all of the code with:

package com.mygdx.nibbler;
package com.mygdx.nibbler;


import com.badlogic.gdx.Game;
import com.badlogic.gdx.Gdx;

public class nibblergame extends Game {

    @Override
    public void create() {
        System.out.println("Game Created");
       
    }
   
}

This creates the game and now we will create a screen for it. Right click on "Nibbler-core" and select New -> Package. Name it com.mygdx.Screen

Add the following code:

package com.mygdx.Screen;
public class GameScreen implements Screen {
} 

Eclipse will show an error and it will say to import and then to add unimplemented functions. Do both. The resulting code should be:

package com.mygdx.Screen;

import com.badlogic.gdx.Screen;

public class GameScreen implements Screen {

    @Override
    public void show() {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void render(float delta) {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void resize(int width, int height) {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void pause() {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void resume() {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void hide() {
        // TODO Auto-generated method stub
       
    }

    @Override
    public void dispose() {
        // TODO Auto-generated method stub
       
    }

}
 
Now return back to "nibblergame.java" and add this line:


setScreen(new GameScreen());

Remember to import "GameScreen" and you are done!