Effect of One Hot Encoded _nan columns

When we encoded the columns using One Hot encoder, we can see the new columns are created suffixes with _<category_name> for the particular feature. We can see that for categorical columns containing the nan values, the columns will be created with _nan suffix. I trained model on the data with these columns, and got a training loss of $ 19630.11 and validation loss of $ 27994.80
Now, when I see the parameters having most effect on my model, I see that this column with nan values have 2nd highest effect, isn’t it misleading?, that an encoded column with nan values is having 2nd highest effect?
Ok, so I decide to remove the encoded columns for nan category. But, it increases my training loss to $ 19660.95 and validation loss to $ 28002.15
Do someone has an idea, what’s going on?

You are essentially getting rid of the information whether that initial column had a value set or not, so loss would increase. Also, you mentioned that it was second most important parameter so of course removing it will affect the loss.