Please am trying to run a code amd am getting memory error.
Kindl assist oo.
You just need to have 32 GB of RAM on your computer.
More useful answer: it might help to ask yourself if you can decrease somehow the size of this dataset which you’re trying to load.
It seems like your dataset has 52436 columns, which is kinda weird.
We need to know:
The data am using is
The data doesn’t seem to have that much columns.
One possible solution (which doesn’t involve removing anything) I could offer (already described in the link I gave in previous answer, which seems to be neglected anyway): change the encoded columns to an
int8 datatype by using
dtype argument of
OneHotEncoder (that’s what I assume you’re using).
72983 rows * 52436 columns requires around 3.8 billion values.
If they would require only a byte of information, this is 3.8 billion bytes (around 3.56 GB of RAM). But the result would be probably a bit bigger than that, because not all of these columns are categorical encodings.
If the above doesn’t work, then the only option is to remove columns which might be unnecessary for the outcome. You can use feature importance (from second assignment) to decide which columns you might decide without harming the result too much.