So in the assignment 1 notebook it was explicitly mentioned to use Ridge regression model. And we were provided with a video link to understand why we are using it.
I saw the video what I understood is ridge regression is useful when we do not have large number of data points relative to no. of features cause there is a chance of overfitting the data hence we make a somewhat non-accurate model to not fit the data perfectly instead we introduce a bias term lambda * m^2.
But for the given dataset for house salesprice prediction we have fairly good amount of data , i.e. we have 1995 data points. So why are we using it in this case? Or my understanding of Ridge regression is wrong?
Actually your Understanding of Ridge is super-clear…
We are using it here because in terms of data-points and it’s huge scale, 1995 data points are not very much that’s why we are using it here…as this is not a good amount of data…
in terms of data-points and it’s huge scale
by this do u mean no. of data points are not enough comparison to no. of features (relatively)
No I am talking about the whole dataframe…
the values are not enough to predict accurate result
For example if your friend gives you choclate today and tomorrow you cannot predict certainly that he will give you chocolate day after tomorrow…but if he gives you chocolate for 10 days then maybe you can say thet he will give you chocolate the 11th day…
And if he gives you chocolate for 100 days then you can certainly say that he will give you chocolate the 101th day…and siimlarly if he gives you chocolate for 1000 days then you can with absoute no doubt can say that he will give to chocolate 1001th day
Similary when we are talking about data in general, 1995 data-points are not enough to predict anything with certainity and that’s why it is considered small and thus we can use ridge regression on this
But if the data-points were 12,345,567 then we can predict want we want to with a huge enough accuracy…and maybe we will not use ridge this time
I know it’s not a good way of explaining but let me know if doubt persists…
This clears all my doubts
Thanks a lot