Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?
It's expensive now, I expect once it is with inference providers it will be really dirt cheap. Then it would be truly be a "bicycle for the mind", which Fable promised to be except it proved to be too capricious for that.
They are not. Pretraining just dumps every piece of content in the mix, and only has one objective - next token prediction. And you can get pretty good results even with base models, you just have to manage context differently. Later stages (mid, post training) involve RL that "surfaces" the right "traces" out of the pre-training. But they are not trained individually, as we used to do.
Have you looked at the what data companies (e.g. Scale, Mercor) hire for? Why do you think Meta records their employees every keystroke/mousestroke/eye-movement?
EDIT: just re-read your comment. I don't think you have a good understanding here, no offence.
None taken, but it would be odd, since I've been training GOFAI models since 2010s and have had LMs in production since before chatgpt came out (before RLHF), so I think I have a pretty good understanding :) But I'm always open to learning.
I think the misunderstanding comes from "individually". You are thinking about diverse datasets, but that's not what individually means in this context. In ML individually trained means that for each task you prepare an architecture, dataset and eval and train that model on that data. And each model has its own objective that you train for. In LMs the objective is singular, for every data point - next token prediction. And, importantly, you train on every datapoint, the more diverse the better, but not independently. The cool thing is that training on diverse datasets improves scores on other downstream tasks, while the training objective is the same.
So you can have a training run on common crawl + programming that improves scores on logic puzzles, or common crawl + novels that improves scores on planning tasks. But the important thing is that it's all trained together, not independently.
I think we're saying the same thing? I said they are explicitly trained on these tasks, not that they are some separate models during programming RL, or business tasks RL.
> The cool thing is that training on diverse datasets improves scores on other downstream tasks, while the training objective is the same.
Maybe for some language modelling tasks, i.e. it learns some internal representation that is transferable. However I would find it quite odd if a model becomes good at Bio while not explicitly going through Bio training.
> I said they are explicitly trained on these tasks, not that they are some separate models during programming RL, or business tasks RL.
You said:
> > nor are they trained for each task individually.
> They are explicitly trained for each task individually.
And that's the main misunderstanding.
Collins says:
individually
in American English
(ˌɪndəˈvɪdʒuəli, ˌɪndəˈvɪdʒəli)
adverb
1. as an individual or individuals rather than as a group; one at a time; separately; singly
Which is precisely what LMs don't do (in contrast to previous "AI" models, which did do that). They are trained on every datapoint at the same time. So long as we agree on that, I think we are saying the same thing :)
Who is their right minds would be wedded to an identity of saying "No"? Code quality puritans are annoying but if they do their job right they actually speed-up the development process because they don't let technical debt accumulate. Ultimately saying "No" is protecting your codebase. In the era of LLMs, saying "No" is much easier because you don't have to worry about the author feeling bad.
Did you intentionally miss the point of my comment? Substitute Opus for GPT-5.5 if you will. I use both as well as locally hosted models using some of your branches, even.
Fair enough. I agree with you - although DS4 Pro is a GPT 5 class model which scores 46% on ARC-AGI-2[^1]. It's behind by maybe 9 months, I think it's still good enough for a lot of complex tasks as well. They definitely need to work on a "just fucking works" harness like CC/Codex. Also thanks!
Not a maintainer, but I've fixed some of the really jarring issues on desktop (mobile needs a complete overhaul though). IMO It's not that bad, and it gets the job done.
Any feedback on how to make it less "shitty"? I feel like doing some vibe coding tonight.
People doing economics with the cloud GPUs, of course cloud GPUs are going cheaper. But also, is generating tokens all you do with your computer? I can play games on DGX spark and also do LLM inference, so sometimes the economics work out, apart from having fun with it.
