You know, given automated proof checkers, I was naively assuming mathematics was one field that gen-AI would have a hard time screwing up. Even programming is too difficult to write thorough testing for. But a proof (or counter example to a conjecture) seems like it would have to be solid if it passes lean or whatever system for validating it.

But the threats #3 and #5 the declaration lists make me consider the bigger picture. Academic fields without clear capitalist payouts are already underfunded and under respected. Pure mathematics could, at least up until now, draw on the respect STEM gets, but with math proofs getting used as fuel for the LLM hype machine, there are a variety of unpleasant ways things could twist.

Threat #1 makes me wonder… if LLMs+formal verification systems manage to pluck lots of low hanging fruit, and we are left with harder stuff that not enough literature exists as training data for LLMs, it seems like the entire educational pipeline for producing mathematicians could end up screwed up.

he chickened out at the last minute and self-censored it

You know, on net, the internet would be a better place if more people did this, but I wonder why the author even bothered posting anything at all?

2017, three years post gamergate

You know, I was wondering if the author elided the gap between early 2000s and 2017 because of the cringey and sometimes outright awful stuff that came out of the rationalists… but you’ve reminded me the skeptic movement also had some nasty bits both within it and as offshoots.

That was kind of a nothing post? I think the author is misrepresenting the growth (or at least failing to connect the most important dots) of the lesswrong rationalist by portraying them as a natural evolution of the skeptics movement and not the deliberate cultivation of an audience by Eliezer looking for people to spread his ideology to and by Thiel looking to influence farm and create a cult incubator. And then the post is titled “my falling out with the rationalist”, and they write they "wanted to offer a more personal perspective " but they don’t actually discuss that much of the personal angle either…

Even /r/singularity, which has a high percentage of boosters and often buys OpenAI/Anthropic talking points uncritically (or with only a few un-upvoted naysayers) eviscerated his comments as the shallow attempt to beg for regulations on China to protect OpenAI that they are. Also, China’s consistent open sourcing and lower relative cost per performance (well insofar as you believe the benchmarks and such indicate performance) has the /r/singularity audience questioning the “China bad” narrative US media tries to sell.

That is an unfair simplification! Some of this audience is there for the closeted (well not so closeted over the past few years) race science and are annoyed by his AI doomer posting!

I read through it myself, but I was too busy laughing at the absurd parts (secret Chinese data centers inside mountains) and suspending disbelief at the overall premise to track all the individually nonsensical bits.

That’s not how international cooperation works.

They also completely ignore the fact that the US has burned away half a century of soft power and (relative) credibility. A lot of Scott’s (and the other AI:2027/2040 authors) readers are Trumpers or at least so libertarian-brain-rotted they view Trump as not that bad a tradeoff, so it’s pretty obvious to me why they would avoid dwelling on this little problem for their scenario. (I think this is also lowkey why the put the scenario off until 2029, even their insane imaginations can’t imagine international cooperation under Trump, but they don’t want to draw attention to this.)

That’s not how economics works.

Yeah, everyone is somehow superrich because carefully limited AIs are producing so much value? Even with the lesswrong favorite trope of magic nanotech (which I was disappointed to see AI: 2040 fail to include), equitable distribution is an extreme problem.

nothing that AI is revolutionizing is really impacting the kinds of stuff that people would be buying save that there’s more of it and that people are no longer being paid to produce it

I actually know the rationalist canon answer to this from other crap Scott has posted! He expects humanoid robots to takeoff exponentially basically, with an initial bootstrapping of buying some car companies’ manufacturing lines solely off the insane valuations of the robot companies.

and of a kind that isn’t even interesting

It is also boring because there is not enough magic-scifi super tech, or competing rival God-AGIs playing 40 dimensional chess against each other as they jockey for power and control over the humans, but your complaints are also true!

utopia they outline is not meaningfully connected to the policies and values they advocate for

Their doom/salvation scenario really needs centralized planning acting for the common good to land on the salvation side, and they are deeply allergic to the idea, hence all the complicated neoliberal workarounds.

apparently he’s having a bit of a hard time selling his crowd on the really obvious totalitarian implications of instituting a global surveillance system to track and remotely disable chipsets

I thought I would agree with his libertarian audience about this much at least, but reading their comments… some of the angles they take and line of argument they make actually makes me more in favor of totalitarian dystopian regulation on computer chips.

Awww, but I like the pure science fiction.

Even as sci-fi, Ai: 2040 is lackluster! Where is the classic lesswrong trope of drexler-style nanotech? Where are the cybernetic augmentations!?

The second is, things happen because of ideas and agreements rather than actions. You can speak into existence billion year plans for what happens to whole galaxies, and it happens because that is what you said.

It is the same flaw that underlies the cryptobros delusions about blockchains and NFTs. You can make some cryptography ledger, it doesn’t mean anyone actually follows it or that the ledger tracks reality.

Oh hey, a full top level post complaining AI-2027 wasn’t sci-fi enough (they literally use the words “I don’t think it is science-fictional enough”)! https://www.lesswrong.com/posts/Lc8v92pGPtjF3gsCv/some-quick-thoughts-on-ai-2027

The impression I’ve gotten it that it is relatively less toothless than some of the other rationalist spin-off movements aimed at preventing doom. And it is willing to deign to try to coalition build with normies that want to stop AI for normie reasons (except apparently they’ve made even that a bad thing, from the tweet it is willing to unapologetically coalition build with fascists!) But yes, it still has a lot of rationalist hangups and quirks getting in the way of actually acting effectively.

Lesswrong has some funny objections to AI: 2040. They buy the overall premise, sure, but even that much still leaves lots of gaps in the “scenario”. And they have even more galaxy brain takes… My favorite complaint: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=54dZes9uNqDdbNGBu

tldr; AI: 2040 is leaving out all the other sci-fi singularity tech just around the corner like genetic augmentations and cybernetic brain implants!

That and treating ARR as a reliable indicator of revenue. Which isn’t as bad as the valuations, but is still pretty bad. Ed Zitron has explained all the ways they game ARR, and, more importantly, it doesn’t matter how much your revenue grows if you are spending 2 dollars for every dollar you make (or spending 20 dollars for every dollar they make, as seems to be the case with their subscription plans).

Maybe the boosters will shut-up once OpenAI and Anthropic finally run out of venture capital to burn on subsidizing subscriptions, but actually, judging by the way the AI 2027 authors are still claiming credit for being right, they will probably just look for someone else to blame for the high costs.

Sometimes they also glue a bunch of “tools” to help the model reason. The model can call by extruding tokens with the right syntax and then get back information from the “tool” shoved in its context! That way, the model can at least handle stuff like basic arithmetic correctly! Except only sometimes, because the models frequently screw up calling the tools correctly or skip using the tool or any number of other completely dumb mistakes. Oh, and if the tool connects the model to the internet (or any insecure source of text) in any way, shape, or form, congratulations, you’ve now got a massive security vulnerability!

And for anyone skimming the comments instead of reading the articles, that is just one of two people Apple’s lawsuit references, there is another high level OpenAI person with a completely different set of egregious corporate espionage actions!

From the headlines I had been assuming it was a normal tech company patent troll slap fight but no… from the 9to5mac article:

When interviewing Apple employees for jobs at OpenAI, Mr. Tan uses Apple’s confidential information to gain access to even more insider knowledge. He has used an Apple internal project codename to ask, “What’s the plan[?]” for an unannounced Apple product.

He has directed job candidates still working for Apple to bring “Actual parts” from Apple to their interviews for “show and tell” sessions in which he and his team at OpenAI can elicit still more Apple confidential information. These directions to bring Apple’s parts to OpenAI job interviews surprised at least one of the candidates, who commented that he “didn’t even know we could take those from the office.”

OpenAI has been instructing Apple employees to bring “CAD/design artifacts” and “prototypes” to their interviews and to divulge details about their work such as “subsystem and component selection,” the “tools or methodologies you use for system integration, such as CAD software, simulation tools,” and “Vendor selection and communication/collaboration with vendors.”

That is absolutely batshit. Like I’ve heard the interview process has been getting worse and worse, but “steal shit from your current company” is on another level. I wonder if he lead in with any plausible deniability to it or just straight up jumped to that. Also, I have a hard time imagining being willing to steal from a company just to get through an interview at another company. Not out of company loyalty, but because I wouldn’t trust the interviewers to actually pay-up with a cushy job and salary.

Isn’t UK’s usually problem the other way around, they use "Protecting the Children"^tm as an excuse to ban sensible things or take away privacy from adults? This would be a first, taking away something from Children to protect the adults.

Lesswronger is mad the current pipeline to sanewash and legitimize lesswrong forum posts into academic content takes too long and thinks about ways to accelerate it: https://www.lesswrong.com/posts/wn5jTrtKkhspshA4c/michaeldickens-s-shortform?commentId=gyJWhLjq5fPh9Bv3b

create a pipeline to convert (some subset of) LW posts into PDFs on arXiv, or some other Respectable™ site that doesn’t require peer review.

The benefit of putting your predictions 15-30 years in the future is that people forget them by the time the deadline comes around and you can pivot more easily and also claim credit for the few successful predictions while all your failures are forgotten. The AI: 2027 could learn from Kurzweils’ example, except apparently you can just keep claiming credit for being right even when your wrong prediction is only a few years old!

Yeah, they’ve internalized their own fiction and fantasies so heavily they forget when they are talking about something they made up and not something real. And in turn, they forget to explain their terms for normies and ridicule commenters that can’t keep up with their lore.

Like in the comments and discussion for AI: 2040 on lesswrong, in response to a heavily downvoted comment, they treated “neuralese” like an obviously real thing just around the corner that will solve chain of thought. Or assuming continual learning is obviously just one or two inventions away and not a fundamentally missing feature of LLMs that no one has a good solution to.