…and I still don’t get it. I paid for a month of Pro to try it out, and it is consistently and confidently producing subtly broken junk. I had tried doing this before in the past, but gave up because it didn’t work well. I thought that maybe this time it would be far along enough to be useful.
The task was relatively simple, and it involved doing some 3d math. The solutions it generated were almost write every time, but critically broken in subtle ways, and any attempt to fix the problems would either introduce new bugs, or regress with old bugs.
I spent nearly the whole day yesterday going back and forth with it, and felt like I was in a mental fog. It wasn’t until I had a full night’s sleep and reviewed the chat log this morning until I realized how much I was going in circles. I tried prompting a bit more today, but stopped when it kept doing the same crap.
The worst part of this is that, through out all of this, Claude was confidently responding. When I said there was a bug, it would “fix” the bug, and provide a confident explanation of what was wrong… Except it was clearly bullshit because it didn’t work.
I still want to keep an open mind. Is anyone having success with these tools? Is there a special way to prompt it? Would I get better results during certain hours of the day?
For reference, I used Opus 4.6 Extended.
my experience with LLM’s and numerical computations like with MATLAB or GNU octave, has been poor. I assume its more of an issue that the data isn’t there, MATLAB has it’s own proprietary AI (which I don’t believe is trained on users code) and Octave has no AI associated on it’s end so the major LLM’s only get trained by the data it is prompted by users online or otherwise. Which is why if you prompt it to do a 3D plot, it will almost always pull something out of it’s ass.
your feeling of a “mental-fog” is my experience with AI in general, the language model explains the ideas well, but then the code editor does some obscure move that makes no fucking sense. also, because you’re not programming it and learning from your mistakes it makes you uncertain of your code. its unfortunate to see search engines are going to shit because of AI, because AI is not ready.
The solutions it generated were almost write every time
Did you vibe code this post? 😂
I have a full pro model for Kiro at work. It does actually work, but we have custom MCP servers for all the internal tools, context on how to use these tools, style guidelines, etc. and then on top of that we have a lot of AI context files in the code base to help the AI understand the code base and make the correct changes.
I’ve been using it on a side project and it works if you know how to constrain it. It does get things wrong a lot. But the big thing about it is doing spec driven development where you give it a write up and it makes a requirements doc and a design doc with a lot of correctness properties in them to follow when generating and making the tasks.
I don’t believe people can vibe code unless they can actually code. It’s a whole different way of coding. I still manually edit what it does a lot.
A lot of people explain it like it’s a brand new junior developer. You need to give it as much context as possible, tell it to exactly what you want, tell it what you don’t want, tell it why, etc. and it still may not listen exactly.
In my experience there are three ways to be successful with this tool:
- write something that already exists so it doesn’t need to think
- do all the thinking for it upfront (hello waterfall development)
- work in very small iterations that doesn’t require any leaps of logic. Don’t reprompt when it gets something wrong, instead reshape the code so it can only get it right
The issue with debugging is that it doesn’t actually think. LLMs pattern match to a chain of thought based on signals, not reasoning. For it to debug you need good signals in your code that explicitly tell what it is doing and the LLMs do not write code with that level of observability by default.
Edit: one of my workflows that I had success with is as follows:
- write a gherkin feature file describing desired functionality, maybe have the LLM create multiple scenarios after I defined one to copy from
- tell the LLM to write tests using those feature files, does an okay job but needs help making tests run in parallel.
- if the feature is simple, ask the LLM to make a plan and review it
- if the feature is complex then stub out the implementation in code and add TODOs, then direct the LLM to plan. Giving explicit goals in the code itself reduces token consumption and yield better plans
write something that already exists so it doesn’t need to think
If something already exists, it shouldn’t need to be rewritten.
Doing otherwise is a sign that something has gone wrong.
That was the case before LLMs and it is still the case today.
What they mean is rewrite something that has a LICENSE my company can’t use.
If the rewrite is based on something which has a license that your company can’t use, then the rewrite likely can’t be used either
I’m pretty sure if code is AI generated it’s likely considered original, but I’m not a lawyer by any stretch.
Only something created by a human can be copyrightable. (See the copyright status of monkey who took a selfie for precedent).
Any code written by an LLM is not copywritable because a human did not write it.
Also the company that trained the LLM is likely in breach of the licenses the code palls under.
Absolutely. It’s amazing how many articles showcasing vibe coding is just people reinventing things like a password generator.
I use it and it works. It doesn’t give you the right result in one shot, but neither does manual coding. You iterate and prompt again and again. In the end, it saves a ton of time. Engineers are definitely going to lose their jobs because fewer people are needed. I know its tough to accept this and people will go through denial. Part of that is saying the AI code is junk. But, you’ll find it can produce junk and quickly fix it into the right solution faster than an engineer can. It sucks, but this is the new reality. The one thing that is cool once you embrace it is that you realize you can customize your favorite apps or even build anything you want from scratch.
It sucks, but this is the new reality.
Sorry mate, but you drank the AI koolaid from Sam Altman and the other tech oligarchs. The reality is that all of the major AI companies are deep in the red, OpenAI isn’t even making a profit with the 200$ subscription.
The only reason people are able to burn thousands of tokens to vibecode their apps is that they don’t have to pay the price for that, the companies are. This money will run out soon and then we will see the real cost for the bigger models.
If a subscription for Claude Code costs 500$ or even 1000$, will companies still pay for it or let actual humans do the work? We will see. I seriously doubt it, and I don’t want to depend on a subscription-based service to do my work while my skills are atrophying. Thank god my employer doesn’t force me to use AI.
Engineers are definitely going to lose their jobs
This kind of fear-mongering is what I despise most about the whole bubble.
I haven’t drank Koolaid. I’m talking from my experience using it in my professional software engineering job where I lead software projects. I’ve built things that used to take 20 weeks in 1 week with Claude. My employer does not really care about the cost of the tokens. And, when they can have one engineer do 20 weeks of work in 1 week, that to them is actually a cost savings. I already ask myself the question … Should I give this task to another engineer or just vibecode it myself?
OpenAI may not survive because they do have financial issues from overspending, but that barely matters. The company with the strongest coding LLM is Anthropic and it doesn’t sound like they’re having financial difficulty. Either way, now that it is clear what is possible, some company will succeed.They have incentives to do it.
Like I said, it will suck for some people, but its hard to deny the reality at this point.
I’ve built things that used to take 20 weeks in 1 week with Claude.
That’s ridiculous. You’ve either been a bad coder even before the AI hype or you’re simply lying. I have used these tools and they’re not that good or make you that fast - except when you’re just merging all of the proposed code blind and hope for the best. I fear for the future colleagues who will have to work with the raging dumpster fire you have created for them.
The company with the strongest coding LLM is Anthropic and it doesn’t sound like they’re having financial difficulty
Oh yes, they have the same problems OpenAI has. Just look into the vibecoding subreddits, you can see many people complaining about excessive rate limits and their models getting dumber. A healthy company wouldn’t try to put a cap on the token useage and introduce peak-hour throttling, that’s a big warning sign that they’re overspending as well.
its hard to deny the reality at this point
I only see one person here denying reality. You will be effed in a major way when your employer one day decides that the subscriptions are too expensive or tell you to limit your token useage.
I know it is a big change and will take some time to come to terms with it. But, it is here. I’m not going to argue anymore. It’s pointless.
Did you just pull a random infographic out of your ass without even mentioning the source? I reverse-searched it and it comes from Anthropic, of all places - the guys that run Claude Code.
Forbes took a look at that study, I love this money quote from it:
These flaws turn Anthropic’s dataset into an overstated labor-market conclusion. The study’s findings do not have the level of reliability required to sustain the breadth of the headline framing, because each conclusion rests on an exposure measure whose scope (1), construction (2, 3, 4, 5, 7), and interpretation (6, 8, 9, 10) remain contested.
So yeah, an AI company telling us that AI will theoretically replace our jobs, based on their own study with flawed data - damn, that’s trustworthy! /s
I’m not going to argue anymore. It’s pointless.
At least on this point we agree.
You still need programmers because you need people proficient in programming to be able to tell how to fix the junk that it generates into working code.
Sure, but like I said, it will be fewer.
And since the chat bot produces entry level code at best those are the positions that will be dropped, starving the field of newcomers in the long run.
I think the last part you said is the best way to use LLMs. I am not confident in it building complex architectures but if you want to make a dedicated single use script or a very customised basic application for personal use, it will do it well
customize your favorite apps
can you elaborate?
Github is full of open source apps. Some times the maintainer won’t add a feature you want. You can just clone the repo and ask Claude to do it and then run your own version of it.
Key is having it write tests and have it iterate by itself, and also managing context in various ways. It only works on small projects in my experience. And it generates shit code that’s not worth manually working on, so it kind of locks your project in to being always dependent on AI. Being always dependant on AI, and AI hitting a brick wall eventually means you’ll reach a point where you can’t really improve the project anymore. I.e. AI tools are nearly useless.
You just didn’t use the right prompts!!!
/s
you need to fully be able to program to work with these things, in my experience.
you have to explain what you want very specifically, in precise programming terms.i tried a preview of chatgpt codex and it’s working better than my free version of claude, but codex creates a whole virtual programming environment, you have to connect it to a github repository, then it spins up an instance with tools you include and actually tests the code and fixes bugs before sending it back to you.
but you still need to be able to find the bugs and fix them yourself.oh and i think they work best with python, but i’ve also used ruby and dart and it’s decent.
it’s kinda like a power tool, it’ll definitely help you a lot to fix a car but if you can’t do it with wrenches it won’t help very much.I recently started using Pro to debug a problem I couldn’t solve. The one thing I need from it is an extra insight, a second opinion (because I’m the only developer), and it allowing me to let it read the whole folder helps, it identified a problem I didn’t consider because it’s a file outside of where I was looking.
The trick about vibe coding is that you confidently release the messed up code as something amazing by generating a professional looking readme to accompany it.
The more Emojis in that Readme the better!
It’s a tool that you need to learn. Try some of claude.md files people share online for your programming area as a starter. You still need to review what it does but just asking for it to create tests as it creates code does a lot to improve output.
Don’t just use it as a drop in replacement for a programmer; use it to automate menial tasks while employing trust but verify with every output it produces.
A well written CLAUDE.md and prompt to restrict it from auto committing, auto pushing, and auto editing without explicit verification before doing anything will keep everything in your control while also aiding menial maintenance tasks like repetitive sections or user tests.
verify with every output it produces.
I agree that you can get quality output using these tools, but if you actually take the time to validate and fix everything they’ve output then you spend more time than if you’d just written it, rob yourself of experience, and melt glaciers for no reason in the process.
prompt to restrict it from auto committing, auto pushing, and auto editing without explicit verification
Anything in the prompt is a suggestion, not a restriction. You are correct you should restrict those actions, but it must be done outside of the chatbot layer. This is part of the problem with this stuff. People using it don’t understand what it is or how it works at all and are being ridiculously irresponsible.
repetitive sections
Repetitive sections that are logic can be factored down and should be for maintainability. Those that can’t be can be written with tons of methods. A list of words can be expanded into whatever repetitive boilerplate with sed, awk, a python script etc and you’ll know nothing was hallucinated because it was deterministic in the first place.
user tests.
Tests are just as important as the rest of the code and should be given the same amount of attention instead of being treated as fine as long as you check the box.
I agree it’s not perfect; I still only use it very sparingly, I was just just saying as an alternative to trusting everything it does out of the box.
I think it’s mostly going to be useful for boilerplate generation, and effectiveness is going to vary wildly based on what language you’re using. JS or Python? It’ll probably do OK. Plenty of open source for it to “learn” from. Delphi? Forget it.
Brief experimentation showed it liked to bullshit if it was wrong, rather than fix things.
I haven’t tried any Anthropic models personally.
So far, between the free online chats by OpenAI and DeepSeek, and the smaller models I’ve run on my own machine, the most useful things I have gotten from it were to treat it as an overeager student that lacks the first-hand experience needed to see the big picture, asking it questions that I’m pretty sure I already know the answer to and seeing if 1) it “understands” what I’m getting at and 2) it can surprise me with a viewpoint I hadn’t thought of before.
Using them to double-check my own ideas seems to be marginally useful, especially when there’s no qualified human being whose attention I can borrow. Using them as a sort of semantic web search can sometimes get me what I’m looking for faster than Google. If anything, they’re an opportunity to exercise critical thinking; if I can tell where it’s getting things wrong I can be fairly confident that my own understanding of the problem/subject is pretty solid.
Vibe coding, though? I have yet to see it work out. Maybe as some starting slop so that I can get to work refactoring code (and get the ideas flowing) instead of staring at a blank file.
Don’t jump right in to coding.
Take a feature you want, and use the plan feature to break it down. Give the plan a read. Make sure you have tests covering the files it says it’ll need to touch. If not, add tests (can use LLM for that as well).
Then let the LLM work. Success rates for me are around 80% or higher for medium tasks (30 mins–1 hour for me without LLM, 15–30 mins with one, including code review)
If a task is 5 mins or so, it’s usually a hit or miss (since planning would take longer). For tasks longer than 1 hour or so, it depends. Sometimes the code is full of simple idioms that the LLM can easily crush it. Other times I need to actively break it down into digestible chunks
