Very much. That’s why generative AI needs to create tests before it writes code. It doesn’t know the answer (it can’t know anything), so instead the agent creates the tests to define the shape of the answer, and then the generative system keeps regenerating the whole output with variations until it fits the answer.
It’s as if you baked a cake for your kid’s 10th birthday, realized that you only added 9 candles, and threw out the whole cake to make a new one. The new one has 10 candles, is perfect, but you’re looking at it from the wrong side so it looks upside down compared to the cookbook. In the garbage it goes, gotta make a new cake!
I was always convinced that the only reason we didn’t have this kind of tech is that it would just be prohibitively expensive to use.
Everyone hails transformers and attention heads as this huge breakthrough, but I’m here scratching my head how that’s a breakthrough when we’re just brute forcing through the problem…
Cory Doctorow explains it quite well in his podcast. The ‘innovation’ is the replacement of the problem domain model (in earlier machine learning systems) with the brute force. This allows the billionaires/ruling-class/fascists to do away with troublesome experts (needed to create the model, but who may have ‘ethics’ and can say ‘no’) and just throw money at it.
My brain does something similar when I’m trying to remember something. I just go alphabetically, hoping the first letter of the word or name I’m trying to recall will trigger the memory. It works more often than not. Brute force the memory bank
This sounds like a promising ADHD hack
But unlike the mind uses probability vs recall to find the next word.
I hate to be that guy, and i hate ai as much as the next, but an llm is a glorified, large neural network that is trained to handle text. Much like an embedding modal of a vector database. Those work exclusively on probability and weights, not recall ( and have existed pre-gpt boom)
That said, current ai is bruteforcing an issue like never before and its done so inefficiently its crazy…
Fucking. Ai. Bubble. BURST
That’s why it works on business majors
No. Not at all. Like, not even close. This is absurdly wrong.
You just don’t get it.
I don’t get it either. Does an LLM spit out a million answers but it only shows me one?
Yes. You can watch the wasteful iterations in a locally run AI.
It will take time to “tell” itself things that only a sleepy toddler would need to be told. And it may have to pause to “remind” itself of those things, later, if it does not “forget”.
It is really interesting the way brute-forcing through a space with a giant pile of associations can achieve some things that look like intelligence (to folks who fail the Turing test.)
Edit: Adding various air-quotes to emphasize that these things don’t think, per comment below.
That’s not how it works at all. You are talking about “thinking” models, which are just models prompting themselves.
LLMs are fancy next word predictors. It’s just returning what mathematically is the next likely word given their training set (really token since it doesn’t actually understand words).
I’m not sure what part of my "explain like I’m five " version you’re taking issue with enough to correct me. You may want to clarify if you feel I’ve misinformed.
To my read, you just restated what I was trying to make more accessible.
Edit: I added some air quotes to help clarify that the terms I used are for illustration only.
Modern online LLMs also expose the thinking process. A few years ago, they didn’t have that feature at all, so the first answer appeared to be the only answer. As far as I can tell, it really was the only answer. The obvious problem with that is the lack of processing. A human could come up with a few stupid responses, but would quickly filter out the worst ones and only say the one that seemed most reasonable. An old LLM didn’t seem do that at all. It would just spit out the first version and let the user deal with the stupidity.
Modern LLMs also spit out stupid idiot BS, but they have an internal processing step where the answer is refined. In some cases, it takes multiple steps to process the answer. If you want, you can also track the internal “train of thought” to see how the initial wrong answer got kicked out and replaced by something semi-useful.
Regardless, I don’t see that a brute forcing it. It looks more like the internal thinking process a human would go through before opening their mouth.
