Important Concepts of Regression Analysis

The first algorithm while starting to learn machine learning is regression to be precise linear and logistic regression . Below are answers to few questions which I found conceptually important .

The residual of the data is normally distributed .

There is should be no correlation among the residual term i.e Autocorrelated data .

The independent variables should not be correlated i.e Data should be Multicollinear .

There should be constant variance i.e data should be homoskedastic .

There should be a linear relationship between the dependent(response) variables and independent (response)variables.

I believed for a long time that scaling improves accuracy and I guess most of you to, but that is not the case scaling does not guarantee to increase our accuracy , what it does is it reduces the time required to train our model .

Since we mostly find a sigmoid function that gives a linear boundary we may assume that its linear but actually Logistic regression may/may not be linear it all depends on the Decision boundary i.e our exponential term .

We can increase our training dataset , but how does increasing dataset actually reduce overfitting , the concept here is if we consider our data to be a combination of a signal and noise , so if we increase amount of data we are increasing the impact of signal in it , since the noise is a random factor our model is being trained to have more characteristics as of the main signal (cancelling the noise randomness) and hence aiming towards generalization .

Another way to reduce overfitting is to reduce the number of features .The more the number of features ,we are training our model to understand and memorize more about our dataset as we reduce the number of features its tending towards generalization , but one must make sure to do this reduction smartly so that the problem of overfitting does not convert to that of an underfitting .

Regularization also helps to reduce overfitting .There are basically two kinds of regularization one is LASSO and the other one is RIDGE .

Lasso performs L1 regularization by adding penalty equivalent to sum of absolute value of magnitude of coefficients , where as Ridge performs L2 regularization by adding penalty equivalent to sum of square of magnitude of coefficients .Lasso can be used for feature selection and cases where data is sparse and there is no multicollinearity whereas Ridge can be used in case there is multicollinearity ,also the computation is much faster .

Related posts:

Are you looking to get into product management? You’re not alone, according to an Agile Insider report The hiring trend for Product Manager’s (PM) grew over 30% between 2017 to 2019. While growth…

Widespread space tourism seems like science fiction, but there are several companies working hard to make it a reality. It won’t happen today, not even the day after that, but after writing about…

After watching Ad Astra on a long flight, I was reminded of the power of a good story and was pleasantly surprised that movies can still retain the concept of love, yet not have it be between a man…