Logistics regression Regularization (2/4)

Ok, so what is the problem…. By now you understand logistics regression (although called regression) is actually a classification algorithm. If you have not done so already, have a look at my previous article for an introduction.

The problem

So, with linear regression, we do not always end up with a linear decision boundary. This can occur in the following cases:

1. Have many features eg. 10 or more
2. If we have 2 features, its not a straight line which decides yes or no.

In these cases, visualizing the data will often look like this:

An important point above is we need to watch out for overfitting. Take note below, we want to fall into that ‘Just right’ space. If we end up overfitting, then our our classification may work great on training data, but may fall short on new data.

Solution

The solution is to regularise our theta’s as we go through our optimisation process. This process involves reducing the magnitude of the values of the theta’s. This allows us move from ‘overfitting’ to ‘just right’.

In essence we are applying a regularisation parameter (lambda) to each theta and reduces them as we calculate each one. This then has the effect of smoothing out the decision boundary.

From the previous article, calculating our cost will change slightly as follows to apply the lambda to each theta.

While, calculating the gradients, our equation would change as follows. Take note, that we do not penalise theta0.

In Matlab code, above equations will look as follows:

function [J, grad] = computeCost(theta,x,y,lambda)
  [m,n] = size(x);
  x = [ones(m,1), x];
  h = sigmoid(x * theta);
  % excluded the first theta value
  theta1 = [0 ; theta(2:size(theta), :)];
  % reduce magnitude of the thetas
  p = lambda*(theta1'*theta1)/(2*m);
  J = ((-y)’*log(h) — (1-y)’*log(1-h))/m + p;
  % calculate grads
  grad = (x’*(h — y)+lambda*theta1)/m;
end

Conclusion

I trust that gives you a good overview of why we apply regularisation. You can go onto the next article where go through a good example.