It is inevitable. We (coders) already did it to other engineering disciplines where machines do most of the work, no human could possibly review or understand the actual details, and only verification ensures it is correct.
You don't think that there is competitive pressure between Anthropic, OpenAI, Google, the various Chinese model companies, and others, to advance the capabilities of their AIs?
Or you don't think that sufficiently advanced AI can cause (mass) harms?
I am saying that if they lost alignment and their new model started injecting cryptolockers, dropping all tables of productions databases into the software or if it stated writing poisonous cooking recipes there would be backlash, lawsuits, and more towards such a lab. Consumers don't want to trust such a dangerous model so they won't buy the tokens and the lab will not want to spend money on lawsuits.
>Or you don't think that sufficiently advanced AI can cause (mass) harms?
Even a feather can cause mass harm if it's used to sign a declaration of war. Something merely being capable of causing mass harm is not an issue and doesn't mean that the existence of feathers are an issue.
But this assumes an evil superintelligence. We don't even need that for terrible things to happen; we just need people doing people things and AI doing AI things at scale, and that scale is rapidly exploding as we scramble to deploy AI in the real world.
Case in point, that hallucination (which, if it was an evil superintelligence, could have been "strategic") that almost led to US boarding a Chinese ship over suspicions of nuclear weapons: https://www.msn.com/en-gb/news/other/us-military-ai-failure-...
It doesn't have SkyNet, it just has to be WOPR. And it doesn't have to be people who are motivated to cause harm, it just has to be people making consequential decisions who are careless, distracted, paranoid or anxious -- which everybody is at some point or another.
And I almost kill people when stopping at a cross walk. That doesn't mean I harmed a pedestrian. Society is set up to be very robust. Humans themselves make mistakes and do bad things and society has had to learn how to live with that truth.
>it just has to be WOPR
Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
> And I almost kill people when stopping at a cross walk. That doesn't mean I harmed a pedestrian. Society is set up to be very robust. Humans themselves make mistakes and do bad things and society has had to learn how to live with that truth.
Maybe not you personally, but many, many other people have killed many, many pedestrians, and when they exhibit a pattern of bad driving -- or other deviant behavior -- we take them off the streets. That is an example of society being robust.
Except, over here, we're rushing to make AI, which we know has many deviant behaviors and we know caused harms in many circumstances (starting with AI-assisted suicides), even more powerful AND deploy it in more and more real world systems!
As history and, literally, current events show us again and again, society has failure modes that lead to widespread harm and destruction. We've had world wars and then literally had multiple close calls with nuclear war right after. And then we have all that's going on out there. (In related news, the Pentagon threw a hissy fit because Anthropic would not let them use Claude for autonomous killing machines. Do I have to even mention what they've been up to these days?)
Most of these failures are caused by misaligned incentives and socioeconomic forces. The incentives and forces around AI have hints of many brand new failure modes that we can't even foresee because things are moving so fast.
> Then why argue for setting a pace for the frontier labs if we've already surpassed WOPR level integration / intelligence.
"We're already going down this mountain pass at 200 miles an hour, why slow down now?"
I think we should not just slow down frontier AI development, we should also slow down where and how that AI gets deployed.
You're saying this as if it is some kind of gotcha?
Don't you hope, not so secretly, that a peace deal can be brokered and we don't have to send our young men to war? I know I do, and I hope most people do. It would be weird if you don't.
The reality is, despite Trump being a gigantic arsehole, the US is still, by a huge margin, the worlds premier military. Realistically, they have the best chance of exerting pressure for a peace deal. I just wish Trump would live up to his promises for once.
You're saying this as if it is some kind of gotcha?
Don't you hope, not so secretly, that a peace deal can be brokered and we don't have to send our young men to war? I know I do, and I hope most people do. It would be weird if you don't.
The reality is, despite Trump being a gigantic arsehole, the US is still, by a huge margin, the worlds premier military. Realistically, they have the best chance of exerting pressure for a peace deal. I just wish Trump would live up to his promises for once.
I honestly have no expert knowledge about this stuff. What I say is based only on intuition.
- China is heavily, heavily incentivised to enhance their own chip making
- Looking at the rate Chinas has expanded into just about every single other
space, and from quantity to quality, I just think it is impossible that they
don't compete on equal grounds pretty soon.
- I don't buy the insurmountable moat of TSMC
Software engineering as we knew it is not coming back, except maybe for very niche fields and your own crafting.
Many other engineering fields have experienced this change long ago, where the majority of the "craft" is done by algorithms. Think chip design, aerospace, automotive, structural engineering, material design.
Their engineering is now focused on higher level design (and even that, less in some cases), and verification.
Programmers are not a special species, immune from this trend. Heck, we are the ones who did it to the other fields, and so it is quite amusing that we're not crying when it is our turn.
That doesn't sound completely correct to me. How many people out of a million US citizens would intentionally order a strike on a school filled with children? I like to think the answer is at or near enough to zero.
I don't really find it plausible that there are enough samples in the powerful people to uncover a chain of command that all agree to intentionally murder school children.
The most likely explanation is this was a mistake. But that doesn't absolve anyone from punishment in my view. It should be almost impossible to make a mistake of this magnitude when you're dealing with weapons of mass death, and you should be culpable for the ones you do make.
Doesn't make it less of a war crime. It just makes my point; they're willing to do this sort of thing, nobody is punished for it, so why wouldn't they be willing to murder a bunch of the enemy's children?
Our population is so emotionally stunted that we mature on things like 24 and Game of Thrones, so killing the children of the enemy isn't just a normalized thing, it is a good idea.
I can read the room. Coding is going to go the way of some other disciplines where machines do most of the detail work and and we know stuff works by verification. There are other fields like this.
Do I like it or not? That doesn't really matter. I need a job, so I'm going to get good at the new way to ensure I continue to have a job. I consider this to be a smart thing to do for myself and my family.
I agree with the work changing more towards verification, but if the verification is one LLM reviewing another one, then that isn't really verification. At best you "assume", but you don't "verify".
Unfortunately I see a lot of (senior as well) engineers who think that just a vanilla LLM reviewing another LLM is sufficient, and my comment was directed towards those. If however you see the LLM era as needing more test support and systems than ever before in the form E2E tests and so forth, where "code review" as such becomes mostly irrelevant as you have such a strong test system in place that if that passes you can be sure it doesn't break anything for users, then yes, that's good.
What I mean is verification through testing, and perhaps other more formal means. This has happened to other engineering domains, yet us programmers have so much hubris that we think it can't happen to us. And everyone is throwing their toys out the cot because it looks like it is actually going to happen.
Code reviews may not even happen, or if they do, it'll be all automated, and the verification will be the key.
Ask anyone in the semiconductor industry when last they understood the design of those things.
Unfortunately, a lot of previous PR reviews already were just gatekeeping, or "presenteeism". People would leave comments about class names or method names, like they couldn't understand what an `apply` method, the only method, meant on a class with that was already named appropriately and did one thing only (to give an oop example). No, the method needed to be renamed `PriceChecks.applyPriceChecksWithTimeConstraints`.
Lots of review comments about various conditions that wouldn't feasibly happen (same shit with claude now).
But then I'd see these same reviewers approving PRs where the bigger design was just fundamentally broken. Oh, we're adding a blocking call on our hot path, but at least the method name makes it very clear that it is blocking.
In general I agree that the current AI reviews are creating too much noise and it is masking these bigger design issues.
>Then we can look at Europe, you have this demilitarized continent that is doing things like seizing Russian ships because they are confident that the US will step and fight a war with Russia on their behalf. Well, they are going to learn that reality has changed and the US no longer has the power projection capability it once did. And what will they do? My guess is that at least one European state will nuclearize.
I was nodding along to your post until this point. Where are you from? Russia is not an innocent party in all of this, which is what your post is implying. Secondly, there are already two European states (who are not Russia) with nuclear weapons.
That's interesting I actually didn't read it this way. I could be wrong of course but I didn't really think op was implying guilt or innocence on Russias part. Just more a statement of what's happening. EU is seizing ships, and they probably do think US will back them, maybe not so much recently haha. Don't think op was making a moral judgement on whether that's right or wrong.
Why do you think coding is immune to this?
reply