ITS hot af, AQI is at 400 in Minneapolis, and I'm drinking a quart jar filled with gatorade powder mix, ice, and cheap whiskey, so now's as good time as any to post about LLMs.
(or, AI as people say)
When I was younger, I was an ideological usability zealot. I grew up with the old version of Mac OS (like Mac OS 9, etc) which did not feature any terminal whatsoever. I believed that all software, including and especially software that is used to produce more software, should be "usable", per the definitions and practices established by the ideas of Design, and as practiced by Silicon Valley in thier rise to domination over the tech industry. (See: 
)
In otherwords, pouring millions (billions??) of dollars into painstakingly and labor-intensively testing the software with 10s of new users, hundreds of times, eventually distilling processes and interfaces into a pure intuitive essence that approaches a mythical "kiosk which can't stump or bore anyone, literally anyone", when they try to use its most basic functions. (Think Gmail, when compared to, for example, self-hosting email).
Only problem is, in order to get there, you signed your soul away to get those billions of dollars, and now someone wants them back plus 1000 times more.
But there have been arguments online over the years, arguments about "worse is better," and such things.
I had believed that, basically, these people were smoking thier own reproductive organs, and that its all bluster; just tech elitism. A way to take a moral high ground and gatekeep the social hierarchy of tech (access to a lucrative career, etc).
Here's a case study:
RTFM (Read the F*ing Manual)
Say that I want to learn about the Linux operating system, because I care about Software Freedom, whatever that means to me. Okay. So, I will browse the man pages. Does this mean that I am browsing something which is only for men? Is this "no girls allowed", like Calvin's tree house?
No, of course not! Why would you think that? Something must be wrong with you if you ever had that idea. CLEARLY man is short for manual, we are talking about the Linux Manual here after all. What about man pages is confusing???
The "worse is better" argument is painted into a bit of a corner here... For example, you're supposed to already know what (3) means in exec(3). Its clear as day; right!?
Ok, back to reality:
We are probably going to ignore this and just move on, or, in an extreme, after years of brushing under the rug, we might send a google (or duckduckgo) search:
What do the numbers on man pages mean
And this search engine query (in 2026) will return a reasonable result:
https://unix.stackexchange.com/questions/3586/what-do-the-numbers-in-a-man-page-mean
But it wasn't always like that. unix.stackexchange.com only started existing in 2010. around that time, this type of google search would return diddly squat.
git gud, as Cranky Kong would say.
Initially, computers came with circuit wiring diagrams and manuals.
The idea was that the user would read the entire manual first, and then start trying to use the computer. Programming in BASIC, or whatever. (this was before my time, I either didn't exist or was probably shitting my pants on the regular at this point)
But Silicon Valley couldn't make as much money that way; the companies that pushed the boundaries of how accessible and usable thier computers were won in the market. "The Eternal September" happened before I even "joined the server". And we've never really looked back since.
Stack Overflow
Here's a thought experiment. Say you have a classroom full of highschool or undergrad college students. You arbitrarily split the classroom in half.
The one half gets the linux manual + the standard library documentation from 4 different programming languages, and the other half gets access to google or duckduckgo + Stack Overflow.
Then you ask each group to develop some software which does something useful, like a web application or something. Which one do you think will deliver more of a finished, usable product?
I was already working in "the industry" at the time that this question reached the managers and executives, and I remember there was definitely some hand-wringing and pearl-clutching around it: What if our junior engineers are just copy-pasting from Stack Overflow? How would we know? Can we penalize them for this? What if someone posts some malicious code on SO and it compromises our systems because someone copied it without understanding it?
As someone who "grew up" with Stack Overflow and learned programming that way, as a member of the Stack Overflow generation, this definitely rubbed me the wrong way and came off as misguided at best, condescending / deeply out of touch at worst. I labeled those concerns as "toxicity" and moved on, it was plain as day to me that Stack Overflow and things like it were a breath of fresh air compared to the alternative. A light in the dark; a place to start and a place to discover, a way to break through the gatekeeping and, through an earnest, genuine interest and work ethic, level up one's own capabilities faster, even to the point where it could be "on-demand" to some extent: anything I need to know to complete a limited task, I can learn in an afternoon. Not always a replacement for the manual, but a damn good entry point to it.
This made a real difference for me; it allowed me to have this career, to make this money, etc. I started making the joke that "the user interface of xyz software is located on StackOverflow", that is, most software can't afford the millions/billions to really give it the proper usability testing treatment; so the online community is left to take up the slack; developers won't bother to document / streamline thier software to make it understandable when there's already a ready-made adaptable hivemind alternative which can "route around" almost anything.
It's an unspoken rule / expectation that any issue a user might run into should be solved by looking up a workaround posted to a site like StackOverflow, instead of being the software author's responsibility to fix. Unless that software author happens to be a megacorporation w/ a public-facing product in a competitive market.
This permeated everything during the "aughties", the 2000s and 2010s; programming languages, compilers, IDEs, even your utility company's billing system. nothing was spared. It still continues to this day.
RTFM --> Stack Overflow --> LLM
Ok, so you might see where this is going... The LLM training companies scraped not just the man pages, but all of the Stack Overflow questions and answers, and used them to optimize thier Next Token Predictor.
Now we can run the same experiment again:
Say you have a classroom full of highschool or undergrad college students. You arbitrarily split the classroom in half.
The one half gets access to google or duckduckgo + Stack Overflow, and the other half gets access to a modern LLM.
Then you ask each group to develop some software which does something useful, like a web application or something. Which one do you think will deliver more of a finished, usable product?
I'm not gonna lie, it feels like the same pearl-clutching, fear, and bullshit from before.
Yes, people who copy and paste from an LLM without even making an attempt to understand what's going on will likely suffer the same fate as those who did the same with Stack Overflow answers.
At the end of the day, in order to be able to "Do Software" or "Make Software", someone has to want to be doing that, to have a legitimate interest in it, and to actually put in the hours and effort to learn WTF is going on inside the computer.
I'm suspicous of the claims that LLMs are actively preventing people from learning, or that they are making people less capable / "dumber". For who? And Relative to what?
LLMs seem to increase folks' ability to interact with the process of creating software. This dramatically increases the # of software creators and authors. And it also dramatically decreases the % of software creators and authors who deeply understand how thier software works. But does it decrease the total # of people who are curious and are actively learning more about how software works? I don't buy it.
It reminds me of the same gatekeepy worries from when StackOverflow was new. I think the accessibility effect is huge; just allowing more people to participate and make progress early on in thier journey will hugely benefit folks' motivation. Ultimately, motiviation is what grows the cohort of sickos who catch the bug and end up NEEDING to understand wtf is going on.
Thats the track to deeper understanding and mastery. Someone telling you to RTFM won't do it; you have to be motivated to seek out "TFM" yourself.
It wont happen for everyone! In fact, it might appear to happen for a smaller % of people than "the before times". But I think thats probably mostly due to the dramatic increase in the number of people who're willing to take the first step. And this isn't just because of LLMs; tools and learning materials for "doing software" have been getting more and more acessible for as long as I've been paying attention, over 20 years at this point.
Usability vs. Industry
I started my usability zealotry journey before I entered the workforce as a Corporate Software Schlub. I justified my fundemantalist beliefs by gesturing at the landscape around me: Where is the software community? Where can people go to learn about things, gain confidence / capability, and what is accessible?
As much as I have been "glazing" stack overflow in this post, really, stack overflow was a very toxic community. If you were posting there, you were practically guaranteed to be ignored or shitted on, even if you were just answering your own question or something innocuous like that.
College was also a bit of a joke; 90% of the students had no interest in learning and growing thier capabilities, they were just there to collect status and land a job which would pay the bills. Curriculum was heinously outdated and most profs were checked out; no interest in learning new things or providing an experience for students that could prepare them for the current state of the industry.
It seemed that actually getting the job and working on a team at a company was... the only realistic, accessible way to learn; besides a herculean self-teaching effort. Even then, teaching onesself without context was risky: What if what you learn isn't relevant or in-demand? It wasn't easy to tell, or easy to break into the high-demand disciplines.
So there's a bit of a chicken and egg problem: In order to get a job, you need to learn the skills that pay the bills. But in order to learn the skills that pay the bills, you need to get a job first so you have someone to learn from.
Also, naturally, companies would not invest in the usability of thier internal systems / development processes to make them easier to get started with / more accessible -- after all, isn't that what they are paying nosebleed salaries for? People who can use the computer without usability being setup beforehand?
Usability <---> Community Spectrum?
Eventually I kinda crashed out of the industry, became an alcoholic, quit for three years, and started doing something else entirely, namely, working with friends to establish Cyberia Computer Club.
It took a long time to bash this thru my thick "on the spectrum" skull, but I eventually realized:
Software can be used by groups of people, not just by individuals
What does this mean?
It should be familiar to anyone who has ever worked in a "good team" for a software company; you don't have to understand how everything works, you can just ask the person who sits next to you. (Before covid, back when we used to sit next to eachother). Even if that person doesn't know directly, they might know the right person to ask, or know how to search for the right person to ask, and eventually you will be connected with someone who can explain what the computer is doing and why it says SUPRE Messaging Error: no ball?, or whatever it may be.
My epiphany: unusable software probably mostly comes from organizations; from places where you always have someone to ask or to lean on when nothing makes sense. The documentation doesn't exist because its held in that organization's social fabric and in people's heads, not because the software authors hate you personally and want you to suffer. .
At this point, I finally understood maybe where the "worse is better" crowd was coming from: a tightly knit group with a healthy social environment can use software in ways that wouldn't be possible for an individual; the sum really is greater than its parts, and such arrangements can speed up development, allow new ideas to fourish faster with fewer barriers to entry, less friction, etc.
Alone with the LLM
Are LLMs replacing this social fabric? Are we being separated and isolated? Why? what does this mean for us? Who does this benefit?
I'm not sure if I'm ready to don a tin foil hat strong enough to claim that the LLMs were explicitly meant to isolate people and destroy the community connections which enabled organizations to use unusable software and systems.. But it sure seems like LLMs strongest proponents have that in mind.
The LLM might start acting as a replacement for the co-worker that you used to sit next to, and with the right tools / agent setup, it could potentially even serve as a frontline replacement for the technical-question-answering-function of your local social group.
I have seen this firsthand; there is a new friction in tech Q&A communities every time someone says "well, did you ask the LLM?" That person may not be technically wrong, but almost certainly its in bad taste. It's like the old "Let me google that for you" toxic in-joke. But at the same time, similarly to the lmgtfy toxicity, it can destroy the fabric and foundation of the community.
OK but I want to keep Google in a Shoebox under my bed tho.
At the same time, I can't completely discount and devalue the LLM.
There are LLMs that can run on many computers that folks already have; they may not be up-to-date with the latest info or have the same breadth of information that search engines do, but more often than not, they are good enough. 20-30GB of data sounds like a lot, but if it can approximate the whole of StackOverflow + most of the rest of the commonly-traveled internet, it might start to become something that we want to grab and hang onto.
Unfortunately; in order to run the LLMs at "interactive" speeds, we need both enough RAM to hold the 20-30GB size, and enough RAM bandwidth to read all of the active parameters each time the model generates the next token (Like, each time it generates a word).
In practice, this means you kinda need a fat GPU or an expensive shared memory system like a new Mac, a Strix Halo computer from AMD, or a DGX Spark from nvidia. All of these options cost more than $2-3k USD.
There is a lot of valid criticism of LLMs as a technology; they're unpredictable, can't tell whether they're right or not; will be confidently wrong etc.
But in the past, without that community social support, the only way to answer these questions was to send your query off to a third party; Google, DuckDuckGo, Ask Jeeves, whatever. The ability to do this without any internet connection and without sharing your queries with a third party is completely new. Despite the memory market buyouts and monopolization, it's still massively cheaper than trying to run your own search engine.
Qwen 3.6 27B
LLMs are also able to do more than just vomit out variably-legitimate BS about whatever you ask. It's like Google Search of the old days grew legs and could start awkwardly waddling around on its own.
For whatever reason, Alibaba has handed out a real spicy one with the Qwen 3 models, particularly the 27 billion parameter dense (27B) version and the 35 billion parameter with 3 billion active "Mixture of Experts" version (35B A3B MOE). These are "Open Weights", which means that while they're not "open source", at least they are "freeware"; the publisher allows you to copy the model and run it on your own hardware. It will run on a Macbook Pro or on a single GPU as long as it has at least 24GB of VRAM, and when it comes to programming-related uses, the Qwen models can definitely hold a candle to Anthropic and OpenAI's best, especially considering that folks can run it on hardware they either already have, or can reasonably afford.
(For comparison, the hardware to run a 1 trillion parameter LLM at the same speed would probably approach $100,000, while a reasonable minimum for these "best in class" Qwen models is more like $1500 even at the current elevated prices)
LLMs' strange memory bandwidth addiction
LLMs must read every single parameter from memory each time they want to generate the next token (next word) of thier "reasoning" output or response to the user's query.
Mixture of Experts (MOE) models improve on this by segmenting the model and only reading smaller parts of it at once, but memory bandwidth still usually acts as the primary limiting factor on the percieved speed of LLMs running on some given hardware.
This is the classic bottleneck of the Von Nuemann computer architechture that we have been using for over 50 years now; each time the CPU does the next unit of work; it must first fetch the work to be done from memory, and as we push it faster and faster, this becomes a major limiting factor.
The same is true for GPUs; only they have 1000s of individual cores all trying to load from main memory at once. What a mess. So of course, memory management and scheduling for LLMs is deep arcane computer science, even carting out the classics like the Hilbert Curve, just like I have on this blog.
But at the end of the day, the maximum speed that an LLM can generate text is determined by how fast it can read the model weights from memory, and it has to go thru every active weight for each token. There are some clever tricks employed to sidestep this as much as possible, but so far, the core problem remains.
But there's one thing I haven't mentioned yet;
Commercial Cloud LLM's reliance on Batching
To put it simply.... Why load every single one of the model weights from VRAM to generate each individual token for each individual request, when we can:
- Load the first 0.1% of the model weights from VRAM.
- Apply them to 100 requests in parallel.
- Then load the next 0.1% of the model weights from VRAM,
- Apply them to 100 requests in parallel.
- ???
- Obtain Venture Capital Funding
In practice, this kind of batching allows LLM inference in a datacenter to be 100s of times more efficient, or put another way, trying to do it any other way is economic suicide by comparison.
The only problem is, in order to pull this off, you need 100 people who are all trying to use your LLM at once. Not something that the "shoebox under your bed" is likely to experience.
This is why attempting to run LLMs locally is viewed with such derision.. You invest $1000s in hardware, only to perform an amount of work that would cost $10 on the open market. In order for the economics to work out, you need enough demand to saturate the hardware with concurrent requests. But right now, there's no path for individuals to get there. OpenRouter says anyone can become an inference provider, but really what they mean is, anyone who has $10M of capacity can become an inference provider.
This makes LLMs fundamentally different from other "Cloud Computing" solutions... In this case, economic actors are incentivized to give all of thier data and processes over to third parties not just because "operating servers is hard", but also because of this particular unique memory bandwidth bottleneck, and the batching of everyones requests thru the same GPUs as the only workable solution.
Again, it's tempting to look at this and say "Clearly this is intentional; it's Cloud 2.0, they just want more vendor lock-in and more ability to tie-down, surveil, and price gouge corporations and individuals for all they're worth."
But this time, I'm not sure it's by design; I think it's also just a neccesary consequence of how LLMs work within the established CPU <--> Memory framework of computing that we have been living in for over 50 years. There are potentially some "compute in memory" style ideas about how to kill this bottleneck, and one could even argue this is starting to peek out a bit in the SambaNova custom silicon that Intel demoed at Computex 2026... But it's clearly many years away from being something that people like you and I can just go to the store and buy.
I acknowledge that there are tons of reasons to dislike LLMs besides this, but at least for me, this hardware-limitation-enforced-centralization takes the cake and is the most troubling. It's hard enough to self-host or community-host a web server; throwing in a 100-times hardware cost multiplier just makes it that much less appealing to do the same for an LLM.
Fuck the Police
I ended up upping the ante from my $700 AMD GPU and buying a $4000 nvidia 5090 anyways. Do I think it was worth my money? Well, it's hard to say.
At the time, I rationalized it to myself by saying "surely someone else will want to use this and I can share it so it benefits more people."
But even tho that didn't really come to pass so far besides a couple folks here and there, I think that LLMs' capabilities are unique enough and valuable enough that I'm ok with grabbing what I can while I can: the future of the hardware and its availability is uncertain. I don't expect to see anything much better than Qwen 3.6 27B fit on this 32GB VRAM GPU any time soon, I assume that model authors will hesitate to release such a thing, lest it compete with thier commerical offerings. But even if that's the case, I can already see a lot of value from Qwen. It can read photos of signs / product packaging that are written in a different language and translate them to english for me, which is pretty neat.
Previously, I would have to submit to google for that. Yeah, its not perfect, but i'd rather have it than not have it.
And of course, it can also work surprisingly well with code, with the shell, error logs, etc. Maybe a subject for another post? IDK.
I don't want to come off as too much of an "AI booster", but at the same time, I can't unsee the parallels between the transition from RTFM to StackOverflow ~15 years ago.
So yeah if you want to use whatever LLM I have currently loaded on that GPU, hmu and I can probably hook you up.
Meh im not gonna edit this. aaaaand Post
Comments