The problem is GTP is learning from our existing knowledge base. If legislation is trying to amend a broken system, we don’t want AI to be modeling that system. This case seems fairly harmless, an AI takeover isn’t what we should be worries about.
Something like institutional racism being replicated in a more insidious manner is the concern. Relying on these closed systems potentially gives the type of people who implemented the discrimination being modeled to turn around and say, ‘See, we were right all along!’ If results are held up on a pedestal and AI is integrated into our political and legal systems, it may make changing society for the better much harder.
We shouldn’t universally condemn tools like ChatGTP being used in this way, but we should tread very carefully when it comes to large scale societal changes.
More linguistic history.