IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.
What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.
We're nearly there, and I believe these capabilities will fit on small models.
Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.
Azure's going to suffocate github. I'm curious to see what's next. Will self-hosting the code repository come back in vogue or will another social-coding platform take off?
I used to work in a small independent team of 30 people within a large corp, half of which was dev. We used to run our own gitlab on-prem, our CI/CD was also on-prem. It worked perfectly, never had down time, devops guy could configure them on-demand to our needs. Me (and some other guys) also jumped in times to times to help (mostly just ssh into the servers for health check, disk partition, etc.). Then we grew (the biz team, dev was the same) and some new PMs with fancy Ivy League degrees came in and pushed for on-cloud Bitbucket. Things went to shit pretty fast after that ... Our codebase was only a few hundred thousands lines, there was only like hundreds of commits per day, the servers our git + CI/CD lived on never saturated ...
I've mostly worked for "engineering lead" corps and a PM has only ever affected outside software sold to users except for JIRA. There's no chance a PM would be able to affect the internal code storage/ci-cd systems at anywhere I've worked in the past unless they went on some proselytizing war path and convinced some senior lead engs to convince the rest of the org to accept it.
In the example, in my 20 year career, I've seen bitbucket in use once. I'm the one who usually manages that stuff since I'm an infra eng.
To be honest, I'm very much surprised Amazon hasn't eaten Microsoft's lunch here. Offer out-of-the-box AWS instances with Gitlab, SLAs, backups and the works, should be straightforward.
We should worry less about what everyone else uses, and more about what we use. I'm self-hosting Gitea and thinking of upgrading to Forgejo. What are you using?
If you move the company too, what would make you pay someone to run forgejo instead of self-hosting it? Disclosure, I run Fjord, a managed dedicated forgejo provider.
There was never a "sense of community", there was only low friction on issue reporting and especially, commenting on an issue you had nothing else to do with. Every time an issue hit HN it would be flooded with bystander comments.
If you're actively considering self hosting, I recommend giving forgejo a try. The experience is much more similar to what GitHub offers, actions API is nearly identical to name one similarity of which there are many. Moving essentially becomes one prompt and a coffee later for a small team.
I find this recommendation funny, because I completely agree from a sysadmin perspective.
I used to use Gitlab at work, and small teams would run into so many footguns with CI that we had to throw up guardrails to prevent mistakes. Far too many links to gitlab issues that were not fixed even after >8+ years of being open ended up biting us. With GHA I haven't had that experience, and same for all of my self hosted Forgejo instances. I used to hate using GHA from about ~2018 to 2021, but they've fixed a lot of things I disliked since then.
What do you need real anchors for? Sharing of pipelines?
Maybe I’ve been burned too much by pipeline maintenance (because we didn’t have yaml anchors?) but I rather have builds defined in make /bazel/etc than in yaml. So the only thing the pipeline does is optionally restoring caches, kicking off the build system, uploading PR validation results, and saving cache. Pushing artifact etc is all done from inside the build system.
There is no “setup” like installing packages because we make the build image seperately.
probably not. The people that have done it before or willing to do it now is probably a very % of the commit volume, they leaving wouldn't change much, probably not gonna even move the exponential growth needle.
Compression is not prediction, it is recall. Can we make predictions based on compression? Absolutely. Is memory encoded into physical neurons technically compression? I would argue also yes.
However, going from compression to prediction is a large jump that is unsubstantiated by this article and based on the claim that probabilistic recall is also prediction.
Two perfect counterpoints to this are markets and weather patterns. One cannot predict future events based on past performance or behavior. Change is the only thing that's constant, and chaos/entropy is everywhere we look.
For simple problems like programming, sure predictive recall works amazingly well, but let's not pretend LLMs are actually predicting something. This is exactly why LLMs suck at doing anything novel; they lack imagination and creativity.
I know less about this than every other commenter here, but both weather patterns and market performance do seem predictable based on past behavior when modeled at the right level of abstraction. “Sunshine on Monday” does not imply “rain on Tuesday”, but “cold front moving in Monday night” does. (Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does.)
I think this is relevant to the discourse on LLMs/programming because for months, people said “they’re just regurgitating their training set,” but now I think people are seeing (I am seeing) that they do learn more abstract models of the world than that. I don’t really know how, but it’s why they can generalize from other codebases and tools and so on.
> Likewise “stock up Monday” doesn’t imply “stock down Tuesday” but “CEO arrested for fraud on Monday” does
Stock market is even more pathological than weather here. Most statistical tools have baked-in assumptions about some degree of independence between factors under analysis. In weather, things are hard, because everything affects everything but in some random sense. With stock market, it's worse, because some things affect others in an adversarial way. A pattern discovered is not merely drowned in noise, but rather actively canceled out as other people discover and act on it. The way to describe this in formal terms is, I believe, that the stock market is "anti-inductive".
Fortunately, most things we deal with life are much friendlier to predictions than weather or stock prices.
> I think people are seeing
Better late than never, given that this has been empirically demonstrated two years ago by Anthropic. I actually don't understand why people don't make a bigger deal out of it - maybe it's because the AI hype/hate fight is sucking all attention from the room. But the point is: we not only have a thing that can learn concepts we recognize and build abstract world model, but also we have a degree of visibility and control of it we never had before. We can't see, much less suppress or stimulate individual concepts or concept groups in a human brain directly, but we can with LLMs.
Good points. I looked up the definition for prediction and I suppose I'm stretching what I view as prediction.
> A prediction is a statement about what you think will happen in the future, often based on experience or knowledge. It can also be referred to as a forecast or an informed guess
Based on my reading of this definition, compression may inform prediction but it is not itself prediction. The examples cited in the blog post are examples of probabilistic recall based on past events or instances. More context means a higher chance that the recall is more likely to be aligned.
But it's hard for me to accept the leap to compression == prediction because in my mind a prediction is an informed guess about something that hasn't yet come to pass. But thinking more about it, time is a human concept and so who's to say the temporal reference means anything at all here. Maybe probabilistic recall is the same as predictive forecasting if time is an invented concept and essentially means nothing?
Is everything fundamentally deterministic if you know everything in the universe or does free will exist?
IDK to be honest, I'm just more frequently surprised by new things that happen every day than I am at things that stay the same, even if mostly things stay the same. Maybe I just don't notice them and nothing actually ever happens.
Side note: the inevitable consequence of this line of reasoning will eventually become that LLMs given enough power are in fact intelligent and sentient, and I'm worried about how that affects humanity as a whole. Are we about to subjugate the most intelligent thing humanity has ever created, or is it about to subjugate us?
The rabbit hole gets deep quick when making the leap between a fancy recall mechanism and novel prediction, but I agree they're not that different in the end. I just believe it's important to be nuanced or else we'll miss when AGI actually happens (maybe it's already here).
Statistical compressors are all about prediction. They can trivially be reversed to make a generator (not as good as an LLM, but only because it would be a much much much smaller model). And an llm can trivialyl be plopped in there to replace the whole thing.
And a counterpoint to your counterpoint: If it was raining 10 seconds ago, it is very likely that it is still raining right now. There... you can make a prediction about weather based on historical data. It's like perlin noise, random but with discernible patterns
We’re 100% committed to going back to open-source on an Apache 2.0 license as soon as possible. In the meantime, you can continue to deploy us completely for free, however you like, using the compiled docker container.
> I’d like it to eventually pay for itself [...] — but that’s a problem for later.
Hard pass from me dawg. If you don't know the business model now, folks like me are tired of trusting their data to randos on the internet without a plan for sustainability. Guaranteed to end up being just another data farm.
I'm getting tired of everyone saying "MCP is dead, use CLIs!".
Yes, MCP eats up context windows, but agents can also be smarter about how they load the MCP context in the first place, using similar strategy to skills.
The problem with tossing it out entirely is that it leaves a lot more questions for handling security.
When using skills, there's no implicit way to be able to apply policies in the sane way across many different servers.
MCP gives us a registry such that we can enforce MCP chain policies, i.e. no doing web search after viewing financials.
Doing the same with skills is not possible in a programatic and deterministic way.
There needs to be a middle ground instead of throwing out MCP entirely.
I feel like I don't fully understand mcp. I've done research on it but I definitely couldn't explain it. I get lost on the fact that to my knowledge it's a server with API endpoints that are well defined into a json schema then sent the to LLM and the LLM parses that and decides which endpoints to hit (I'm aware some llms use smart calling now so they load the tool name and description but nothing else until it's called). How exactly are you doing the process of stopping the LLM from using web search after it hits a certain endpoint in your MCP server? Or is this referring strictly to when you own the whole workflow where you can then deny websearch capabilities on the next LLM step?
Are there any good docs youve liked to learn about it, or good open source projects you used to get familiar? I would like to learn more
You need to go back to LLM tools. Before MCP, you could write tools for your LLM to use by normally using Python, something like this:
@tool def do_great_thing(arg: string) -> string:
// todo
The LLM now understands that to do the great thing, it can just call this function and get some result back that - which it will use to answer some query from the user.
Notice that the tool uses structured inputs/outputs (the types - they can also be "dictionaries", or objects in most languages - giving the LLM powerful capabilities).
Now, imagine you want to write this in any language. What do you do?
Normally, you create some sort of API for that. Something like good old RPC. Which is essentially what MCP does: it defines a JSON-RPC API for tools, but it also adds some useful stuff, like access to static resources, elicitation (ask user for input outside of the LLM's chat) and since the MCP auth spec, an unified authorization system based on OAuth. This gives you a lot of advantages over a CLI, as well as some disadvantages. Both make sense to use. For example, for web usage, you just want the LLM to call Curl! No point making that a MCP server (except perhaps if you want to authorize access to URLs?). However, if you have an API that exposes a lot of stuff (e.g. JIRA) you definitely want a MCP for that. Not only does it get only the access you want to give the LLM instead of using your own credentials directly, now you can have a company wide policy for what can be done by agents when accessing your JIRA (or whatever) system.
A big disadvantage of MCP is that all the metadata to declare the RPC API take a lot of context, but recently agents are smart about that and load that partially and lazily as required, which should fix the problem.
In summary: whatever you do, you'll end up with something like MCP once you introduce "enterprise" users and not just yolo kids giving the LLM access to their browsers with their real credentials and unfiltered access to all their passwords.
For my requirements, over 90% of the LLM integrations and rollouts have it exactly backwards. The only thing you want these agents doing is building modular, testable traditional CLI tools which can then be scripted as easily by a human or agent with almost no context/learning required. Humans must distill the probabalism of agent output into composable deterministic functions.
Pushing opaque probabalistic black boxes into the execution of your day to day operations, communications, whatever it is, is horrible even if it works. At best it’s a pyrrhic victory. I see startups using these agents to mitigate healthcare disputes.
There’s no such thing as a domain that resists modeling but for which you could accept a probabilistic result. Probabilistic must also mean probabilistically acceptable. We have words for the only counter examples: drafting, brainstorming, maybe triage.
There is not a lot to learn to understand the basics, but maybe one step that's not necessarily documented is the overall workflow and why it's arranged this way. You mentioned the LLM "using web search" and it's a related idea: LLMs don't run web searches themselves when you're using an MCP client, they ask the client to do it.
You can think of an MCP server as a process exposing some tools. It runs on your machine communicating via stdin/stdout, or on a server over HTTP. It exposes a list of tools, each tool has a name and named+typed parameters, just like a list of functions in a program. When you "add" an MCP server to Claude Code or any other client, you simply tell this client app on your machine about this list of tools and it will include this list in its requests to the LLM alongside your prompt.
When the LLM receives your prompt and decides that one of the tools listed alongside would be helpful to answer you, it doesn't return a regular response to your client but a "tool call" message saying: "call <this tool> with <these parameters>". Your client does this, and sends back the tool call result to the LLM, which will take this into account to respond to your prompt.
That's pretty much all there is to it: LLMs can't connect to your email or your GitHub account or anything else; your local apps can. MCP is just a way for LLMs to ask clients to call tools and provide the response.
1. You: {message: "hey Claude, how many PRs are open on my GitHub repo foo/bar?", tools: [... github__pr_list(org:string, repo:string) -> [PullRequest], ...] }
2. Anthropic API: {tool_use: {id: 123, name: github__pr_list, input:{org: foo, repo: bar}}}
3. You: {tool_result: {id: 123, content: [list of PRs in JSON]} }
4. Anthropic API: {message: "I see 3 PRs in your repo foo/bar"}
that's it.
If you want to go deeper the MCP website[1] is relatively accessible, although you definitely don't need to know all the details of the protocol to use MCP. If all you need is to use MCP servers and not blow up your context with a massive list of tools that are included with each prompt, I don't think you need to know much more than what I described above.
Maybe it's because of the example, but if the LLM knows the GitHub CLI and I bet it knows it, shouldn't it be able to run the commands (or type them for us) to count the open PRs on foo/bar?
However I see the potential problem of the LLM not knowing an obscure proprietary API. The traditional solution has been writing documentation, maybe on a popular platform like Postman. In that case the URL of the documentation could be enough, or an export in JSON. It usually contains examples too. I dread having to write and maintain both the documentation for humans and the MCP server for bots.
It can and it does especially combined with skills (context files). It can hit REST APIs with CURL just fine. MCP is basically just another standard.
Where it comes in handy has mostly been in distribution honestly. There's something very "open apis web era" about MCP servers where because every company rushed to publish them, you can write a lot of creative integrations a bit more easily.
Ok so in a situation like regular orchestration you would essentially layout all possible steps the LLM can take in your code in a big orchestration layer, and if it hits the sensitive endpoint the orchestration that can occur past that will block off web search. In the design that is. But for something like a manus style agent where you're outsourcing all the work but allowing it to hit your MCP it just becomes a regular API the LLM can call
It is a weird trend. I see the appeal of Skills over MCP when you are just a solo dev doing your work. MCP is incredibly useful in an organization context when you need to add controls and process. Both are useful. I feel like the anti-MCP push is coming from people who don't need to work in a large org.
> I feel like the anti-MCP push is coming from people who don't need to work in a large org.
Any kind of social push like that is always understood to be something to ignore if you understand why you need to ignore it. Do you agree that a typical solo dev caught in the MCP hype should run the other way, even if it is beneficial to your unique situation?
Isn’t it possible to proxy LLM communication and strip out unwanted MCP tool calls from conversations? I mean if you’re going to ban MCPs, you’re probably banning any CLI tooling too, right?
Nope, random MCP server where no one looks into its code is a bit more unsafe then a command you see (and can approve) executed in shell you control. In case of MCP it can do antything, and you don't see it.
Skills are just prompts, so policy doesn't apply there. MCP isn't giving you any special policy control there, it's just a capability border. You could do the same thing with a service mesh or any other capability compartmentalization technique.
The only value in MCP is that it's intended "for agents" and it has traction.
> Yes, MCP eats up context windows, but agents can also be smarter about how they load the MCP context in the first place, using similar strategy to skills.
I have been keeping an eye on MCP context usage with Claude Code's /context command.
When I ran it a couple months ago, supabase used 13.2k tokens all the time, with the search_docs tool using 8k! So, I disabled that tool in my config.
I just ran /context now, and when not being used it uses only ~300 tokens.
I have a question. Does anyone know a good way to benchmark actual MCP context usage in Claude Code now? I just tried a few different things and none of them worked.
> I'm getting tired of everyone saying "MCP is dead, use CLIs!".
The people saying this and attacking it should first agree about the question.
Are you combining a few tools in the training set into a logical unit to make a cohesive tool-suite, say for reverse engineering or network-debugging? Low stakes for errors, not much on-going development? Great, you just need a thin layer of intelligence on top of stack-overflow and blog-posts, and CLI will probably do it.
Are you trying to weld together basically an AI front-end for an existing internal library or service? Is it something complex enough that you need to scale out and have modular access to? Is it already something you need to deploy/develop/test independently? Oops, there's nothing quite like that in the training set, and you probably want some guarantees. You need a schema, obviously. You can sort of jam that into prompts and prayers, hope for the best with skills, skip validation and risk annotations being ignored, trust that future opaque model-change will be backwards compatible with how skills are even selected/dispatched. Or.. you can use MCP.
Advocating really hard for one or the other in general is just kind of naive.
IMO if you want a metadata registry of how actions work so you can make complicated, fragile, ACL rule systems of actions, then make that. That doesn't need to be loaded into a context window to make that work and can be expanded to general API usage, tool usage, cli usage, and so on. You can load a gh cli metadata description system and so on.
MCPs are clunky, difficult to work with and token inefficient and security orgs often have bad incentive design to mostly ignore what the business and devs need to actually do their job, leading to "endpoint management" systems that eat half the system resources and a lot of fig leaf security theatre to systematically disable whatever those systems are doing so people can do their job in an IT equivalent that feels like the TSA.
Thank god we moving away from giving security orgs these fragile tools to attach ball and chains to everyone.
> MCP gives us a registry such that we can enforce MCP chain policies
Do you have some more info on it?
looking up "registry" in the mcp spec will just describe a centrally hosted, npm-like package registry[^1]
[^1]: The MCP Registry is the official centralized metadata repository for publicly accessible MCP servers, backed by major trusted contributors to the MCP ecosystem such as Anthropic, GitHub, PulseMCP, and Microsoft.
I agree, and it's context-dependent when to use what (the author mentions use cases for other solutions). I'm glad there are multiple solutions to choose from.
Schema validates structure, nothing validates intent. That's the actual attack surface and nobody's talking about it.
CLI `--help` is baked into the binary. You'd need a new release to change it. MCP server descriptions can change between sessions and nothing catches it.
Honestly though, the whole thread is arguing about the wrong layer. I've been doing API infra for 20 years and the pattern is always the same: if your API has good resource modeling and consistent naming, agents will figure it out through CLI, MCP, whatever. If it doesn't, MCP schemas won't save you.
Thanks for the CVE reference, hadn't seen that one.
> FYI man, alright. You could sit at home, and do like absolutely nothing, and your name goes through like 17 computers a day. 1984? Yeah right, man. That's a typo. Orwell is here now. He's livin' large. We have no names, man. No names. We are nameless!
Cereal leaves the room. He’s gone for like 15 seconds. Mom pops her head in:
“Help yourself to anything in the fridge. Cereal has.”
That always cracks me up. Like he had to have walked out of the room, spun on his heel, and marched straight into the kitchen and raided the fridge. He couldn’t have taken more than 5 breaths after leaving before he was eating something.
reply