How does it damage the PN junction of the panel is open circuit or barely loaded? It doesn’t seem logical that this would damage the panel, but I’m open to being proven wrong.
There are all kinds of follow up questions to ask as well, but I think the main one is how big an effect are we talking?
I run a similar setup, but with syncthing as the syncing system. Every time I connect the phone to the charger it just syncs the database and I can even sync it outside the home network. Works like a charm. Worst case you get a sync conflict which is easy to solve.