Genuine question
Edit: change the title from “Why can Socialist Market Economy be categorized as socialism, and why is NEP categorized as state capitalism?” to “Why can Socialist Market Economy be categorized as socialism meanwhile NEP is categorized as state capitalism?”
These are not mutually exclusive and could be used interchangeably.
“State capitalism” means that the state, under a dictatorship of the proletariat, holds all the political power not the capitalist class. Both the USSR during the NEP, and China now, can be called state capitalism. Both can also be called socialist because the state directs the economy towards the development of production through a market system.
Personally, I would say state capitalism, in the NEP/Lenin usage, and socialism, are different.
The proletariat seizes power by expropriating the largest enterprises which already have socialized production to use as the basis of socialist society. However, it logically follows that in order for this to lead to the proletariat having a dominant position in the economy, for public enterprises that operate for the interests all people to be the principal aspect of the society, then those large enterprises must have already dominated society prior to their expropriation.
If you nationalize the biggest enterprises in a country where there really are no big enterprises and so industrial big bourgeois capital does not actually dominate society, then you will not end up in a dominate economic position after nationalizing them. You would be nationalizing what is ultimately a secondary, subordinate set of enterprises which play limited role in the economy as a whole.
When Lenin talked about the NEP being capitalist, he said that Russia at the time was overwhelmingly dominated by “petty-bourgeois production.” That means even if he nationalized the biggest enterprises, the dominant aspect of the economy will still be the small enterprises and not the big enterprises, and even those “big” enterprises, he said many were not even currently operational due to the war.
The socialist market economy exists in a country where big enterprise does dominate society so there is actually a material foundations for building a socialist society, but small enterprise still exists in a significant degree, just in a secondary, subordinate position.