Logistic Regression Model for classification problem

Week 3 of the Machine Learning course, we learnt about Logistic Regression Model.

Classification problems can be binary or multi-class. A decision boundary that can be either linear or non-linear is used to classify the inputs of such problem. For the multi-class classification, we can use "1 versus all" method that choose one class and then lump others into a second class then apply binary logistic regression to each case until a minimum value is achieved. 

Tip: Don't use linear regression for the classification problem as it may give hypothesis that exceed the range.

Instead, use Logistic Regression Model that uses sigmoid / logistic function that always produce hypothesis between 0 and 1. 

The hypothesis produced by Logistic Regression Model is always in terms of probability of the inputs being in a specific class while the hypothesis produced by linear regression is based on an output value (e.g. price of housing property). Hence, Cost function in Logistic Regression Model is different from that of linear regression.

Note: The word 'features' refers to characteristics of the input that contributes to the output. Example: price of the housing property (i.e. output) is determined by the location, size, number of bedrooms etc (i.e. features). 

When the input have too few features, it can lead to "under-fitting" issue that cause the hypothesis to have high bias. On the other hand, if the input have too many features, using a high degree polynomial can lead to accurate predictions on training data but cannot provide accurate predictions on new unseen data. This is known as the "over-fitting" issue that cause the hypothesis to have high variance. 

Two ways to resolve the problem of "over-fitting":
  1. Reduce the number of redundant features
  2. Regularization 
    • This method aims to reduce the 'weights' of each feature so that it can decrease the cost of the machine learning algorithm 
If the hypothesis is a function based on too many features, linear and logistic regression models would be too inefficient. In other words, these models would have difficulties in finding a good hypothesis. Therefore, Neural Network Model is preferred (Week 4). 

Comments

Post a Comment

Popular posts from this blog

How to connect Python to MySQL Workbench

Image
Step-by-Step Guide 1) Download MySQL: For Windows user , you can download  MySQL Installer that allows you to install all the products in one package. Link as follows https://dev.mysql.com/downloads/windows/installer/8.0.html Download MySQL Installer (For Windows only!) Begin download Otherwise, you can download the product 1 by 1  (Remember to select your Operating System for all!) MySQL Workbench  MySQL Connector / Python   (Note: You have to download Python first. And remember to select the OS and Python version!) MySQL Community Server For Windows users... 2) How to setup MySQL Installer We are going to show how to download and install MySQL Community Server only... Double click the file and install it Once it has finished installing, you will see this window below and click on the 'Add' option Click 'Add' option Agree on their License terms and click 'next' Agree on License Terms Selec...
Read more

Predict EPL results (Part 2: Neural Network example)

Image
Neural Network explained Artificial Neural Network is a model that receives input in vector/matrix form and uses them to compute the output  It is made up of 3 different layers namely input layer, hidden layer and output layer.  Each layer is made up of 1 or more artificial neurons (aka nodes)  The nodes are interconnected with one another Each node is assigned to a weight value denoted by  w  that determines how much emphasis is given to the learned information extracted from the raw training data  Note: A   Neural Network with 1 hidden layer is known as  Shallow Neural Network Types of Neural Networks Feedforward Neural Network Convolutional Neural Network (CNN) Recurrent Neural Network - Long Short Term Memory (LSTM) Networks Recursive Neural Network Neural Network Architecture The figure below shows a  Neural Network Architecture  that is made up of n  features,  h  hidden layers and ...
Read more

Bias and Variance in Machine Learning

Sorry for the lack of updates...Have been rushing to complete the Machine Learning course as fast as possible... Good news!!! We managed to survive through the whole course!!!! Hahaha Summary of Week 6 lesson: Some definitions first before delving into the details... Bias:  The hypothesis is known to be biased if it tends to overestimate or underestimate a parameter Variance:  The hypothesis is known to have high variance if the data are far away from its average value Given the complexity of some data, a linear regression may not be able to formulate an accurate prediction. As the linear regression only factors in 1 specific feature of the data, the 'best-fit' line does not fit the data well enough. Such under-fitting problem leads to high training costs and also causes the hypothesis to be biased. Ways to reduce the bias: Add more features / Increase the degree of the polynomial Decrease λ  Use larger neural network As mentioned a...
Read more