One guess is that their "primary" target audience/market is the large corporations that get their employees unlimited tokens, and not the individual developer who may worry about spending and token accounting.
It's the opposite. The enterprises have all the tooling to monitor token usage of employees, and to limit access. For example, we have a $300 month limit, and then need to file exception tickets when we need more to justify the cost. Pretty similar at other non-silicon valley company process. I don't know any enterprise who'se on unlimitaged token budget for their employees. that's not how enterprises sign contracts.
> Someone should file the multi-hundred-billion-dollar class actions
Incidentally, last week Amazon changed their TOS [0] and now all disputes with them are to be resolved via Arbitration; class action waiver when you use their services, etc.
What was most remarkable to me from this transcript, was how strong of an equal the AI agent comes across compared to the user (Tao). And Tao is one of the top mathematicians of modern times.
Yes, Tao is guiding it to where he wants to go. But also, Tao is actively learning from it and relying on its explaining, analysis, and inference abilities. You can easily imagine this conversation having taken place between Tao and a PhD thesis student, or even another professor, explaining their results.
What can we imagine and predict about the future anymore? Maybe a year - or two model releases - from now, the AI assistant will be undeniably stronger than Tao, and not an equal anymore.
>Maybe a year - or two model releases - from now, the AI assistant will be undeniably stronger than Tao, and not an equal anymore.
we're kind of well past that (in my opinion), if you consider that this is the same ai assistant that can help you with a recipe, diagnose a weird sound in your car, help with biology homework, translate languages, and so on.
even in math alone, i think its indisputably already stronger than Tao, considering it has approximately this much depth in ~all of the math subfields.
Agreed, it's stronger "horizontally". But I also think that we're not far away from it being stronger vertically; i.e. superior to Tao, in that such turn-by-turn guidance by him in solving sophisticated and difficult problems will not be necessary for long.
I strongly suspect that the horizontal and the vertical strength are not perpendicular. I suspect having a very wide knowledge gives you access to patterns that reproduce across fields that an expert would lack.
It only has that depth because you prompt it towards the experts that understand that depth.
An example that happened 10 minutes ago: contracts in racket, it kept arguing that you can't use -> in a contract of a function with a rest argument. I had to mention ... explicitly that it wrote the code correctly.
Yeah...I know I should be massively impressed, and I am to a degree, but isn't this what we should expect? LLMs can pick up on patterns that no human can see. In that sense, they really are a type of "search engine", but I use that term loosely. Tao is using them as a way to sift and sort theories and information that the model has a plethora of training data on. It's like being able to converse with the codex of humanity, and extract data via an algorithm that is highly adept at sorting through it.
Also I feel like people are not noticing in Tao's chat that chatGPT fairly often is doing web searches. So it literally is acting as search engine sometimes, and that too is incredibly effective when done right. So in addition to Tao using his prompts to "search/sift through vector space" the LLM is also literally just searching the web to augment its "knowledge" at inference time too.
Which, to me, again demystifies these tools. They are incredible tools, but still "just" such.
> It's like being able to converse with the codex of humanity, and extract data via an algorithm that is highly adept at sorting through it.
Bingo
Thanks for laying it out so clearly.
This is what I feel LLMs are really good at yea. Almost to the point that I'd say this is what they are. But I'm not quite convinced that it is that, but it seems to point that way
Interestingly enough, ChatGPT started his answer like this only once:
"This is exactly the question I would ask next. My impression is:
Most standard invariants are...."
And this was a response to this prompt:
"Is there a chance of an indirect argument of X ~ A^3 coming from computing some invariant of X that forces it to be A^3? (I am not all that expert in algebraic geometry but I'm thinking like degree or Betti numbers or something.)"
I find it helpful to think of LLMs as reflections. If you can talk like an expert mathematician at the model it will respond like one. While Terrance's first prompt looks trivial I expect a first year Uni student would be hard pressed to provide something that good.
I guess it is kind of the inverse of the "you are an expert mathematician" prompt engineering of gpt3.5. Since no one ever says that to an expert mathematician when they are doing expert math the model immediately reflects that it is not an expert mathematician.
Isn't LLM doing something like argmax{x} P(x|context) where x is the next word, and context is the words before including your instructions? Thus, the LLM output is heavily depending on your instructions, and if you input an expert context it would produce an expert result. Consequently, domain expertise has a significant bearing on what you can achieve from LLM.
I dunno. To me, this seems like kind of a counterexample (pun intended?) to that thesis. Would a conversation with another AI have been as fruitful as this conversation with Tao? Certainly not! Will that change in a year? I dunno, but it seems like those goal posts keep moving, and I'm a bit skeptical.
I think you're mostly right, and there is still a lot of "cope" in this thread about how much he needed to guide it.
But I would say that in some ways it's already obviously superhuman. The reason I think it's lacking in some jagged ways still at the expert level is because although it's highly optimized to be incredibly capable in many domains, it still doesn't have quite the same raw capacity for complexity in understanding one problem that humans do.
I believe that LLMs (really should be called VLMs for most of them) can still get much larger, and that will push the absolute complexity level and general IQ way over human level.
They are maxing out at like 5 or 10 trillion parameters right now. I believe we will see 50 and 100 trillion parameter models and models with large portions of that active. It will round out the jaggedness and probably more than double the raw intelligence that a human can achieve. It's not a linear scaling but who knows what the limit is and can go significantly higher with the same architecture I think given continued improvements in training and hardware scale.
Of note: the author had AI help synthesizing the actual 10-page prompt that resulted in the proof! A powerful tool when you know what you're doing.
> Did you have first an AI help designing the prompt?
>> yes, I did! I basically had 5.6 Sol synthesize existing closely related work and their approaches, the past ideas I had, with OpenAI's prompt that had a lot of the presumably important mechanisms for how exactly the agent should act. Especially from the "results that do not count" section onwards is a lot of input from Sol.
Hmm I see. I only use "old" reddit and it does require login there to resolve to a real address. In any case, it is a special link that enables tracking (unnecessary, to say the least).
It's becoming much harder to determine on a daily basis what content is original, thought-out by a person, and trustworthy. Ironically, verifiably-old content is easier to trust now. Examples from recent personal experience:
1) Some time ago I was searching for growing information about a specific and uncommonly-grown plant, and was led to a top-ranked website with long pages containing everything about it, including other plants. Surprised at how prolific the writing was, I spent more than an hour on the website, taking notes, etc. Every few paragraphs it would include an amazon affiliate link to something topical, which I thought was fair. Until I realized that the links near the bottom of the page were looking more random. Then it hit me, the website is all AI-generated, and the affiliate links themselves are also AI-chosen. And everything new I "learned" from that site was now useless because I had no way to know what was grounded in actual agricultural experience and what was hallucinated.
2) Recently I did a youtube search for a book I had just finished reading, looking for some reviews. Came across a channel that was reading the book as new audio (i.e. not the original published audiobook). I thought it was a fan making it. The voice was beautiful, soothing, and natural with all kinds of relevant emotions correctly included. I started listening to the book again, until I noticed a consistent error in word ordering being made every few lines. Then it hit me! The channel even included one upload with a video recording of a seemingly-real person reading with that voice. Both the audio and video are AI-generated, but very hard to tell.
3) Next to those videos, YT recommended many strange/new channels. One had the photo and the exact voice of a famous (and now very old) physicist, with tens of clickbaity titles about controversial topics in the domain. The only tell was that the voice was too vigorous and consistently energetic, while if you've listened to that physicist before, you know his cadence is slower. At first I thought maybe the channel is reading one of his books; no, the content itself was AI-generated, maybe based on his books. There was a lot of engagement, with many comments like "mind blown" and "learned so much today".
Both #1 and #3 are harmful, because you think you're learning from a reliable source but you end up learning hallucinated nothings. #2 I didn't mind much, still enjoyed the new voice, and even preferred it over my original audible version.
Something I've recently started seeing, maybe even an emerging #4 is AI generated translations. You could have someone very intelligent, making well written subject matter expertise. Or just someone who has valid thoughts they wish to express to the world in a language more of a common tongue than their own.
Or on the other end you could have someone who wrote a sentance or two in their language and had some combination of AI generation and translation algorithm bloat it out.
In both cases you will get something that can look right and well thought out or explained, but probably will have at least some of the AI slop signs present. I don't know what the solution is for this type given claims Google Translate has started to do this kind of translation for people. An AI translation is probably just as prone to hallucinations as any other AI, but it probably will look more natural to readers than a direct translation.
You're making the classic mistake of looking for a trustworthy information source and then trusting it, instead of focusing on whether the information itself is trustworthy regardless of source. It's literally the same as my grandma saying "they said so on TV, therefore it must be true" while completely dismissing anything I've read on the internet because reasons.
If you develop the skill of judging information by its merit rather than source, you won't mind AI-generated content as long as it's helpful.
I talk to LLMs a lot. It's fucking great. Do I take everything they say at face value? No. But neither do I take at face value things that biological intelligence outputs.
Information itself cannot be trustworthy. It can be right, it can be wrong, or it can be somewhere in between. Only a source can have trustworthiness, as it's a mixed measure of reputation and provable accuracy.
You filter out known untrustworthy sources to not waste your time verifying false information 100x more than you need to. I know The Onion is a satire publication. I do not need to verify its claims. It's an intentionally untrustworthy source. I know that LLMs can hallucinate information, so I verify with a more trustworthy source. I cross-reference things random people say on the internet, because random people on the internet are not, individually, trustworthy sources of information.
If a rocket engineer explains to me why Rocket A isn't flight ready, I'm more inclined to believe them than if a random commenter on the internet explains it to me. Because the one source is more trustworthy than another, and if I wanted to verify the claim myself I'd have to spend a lot of time studying rocket science.
No it's not the same as your grandma. The point is that it's now more expensive to find the correct information to learn from. You don't know it's an LLM ahead of time, and you may spend hours until you figure out something is off. Hence why reputable sources will become more valuable.
> If you develop the skill of judging information by its merit rather than source..
Did you read example #1? I'm not talking about some piece of code from an LLM that you can verify or some political opinion that you can take with a grain of salt, but information that you can only gain and/or judge through expertise:
If you're not a physicist yourself, you can't judge "information by its merit" on specific physics topics, because you don't have a solid baseline.
Similarly, in growing plants, each plant has its own peculiarities, and only people experienced in growing it can tell you anything useful - it's knowledge accumulated by trial and error. Not knowledge that your "great discerning mind" can assess on its own. Even a botanist can't tell you the ideal growing conditions of a plant that they've never studied before.
What if your physics book is wrong because knowledge has advanced since it was released - you can still find lots of publications and people with degrees blissfully unaware of Hawking Radiation. What if your botanical book is wrong because facts have changed since then - climate is changing and so does flora. What if your book is wrong because it's state-funded propaganda mixed with petty fights of a bunch of people with suits and strong opinions disguised as academia - a huge chunk of linguistics is dealing with exactly this issue.
Again, you seem to miss the point that the idea of questioning new information, which was already useful to navigate life before LLMs, before television, before newspaper, before print, before clay tablets, even before speech itself, is equally applicable to LLMs as to any other form of communication. You just need to upgrade your strategies a little and that's it. Don't blow this out of proportion "somebody gasp lied to me on the internet!".
There’s a lot of things where this just doesn’t work. I was wrong about a lot of business strategy things when I was younger, to the point where I rejected what I now see were correct arguments against my view of things. How could I have gotten out of that trap without the ability to find trustworthy sources?
Well, if not disclosed you could assume that somebody did due diligence for you, and could include sources. I don't even trust LLM even if all the information is included in the context window if I need reliable information. Trying to make money on slop is really bad manners. It's a scam, you can't call it otherwise.
Btw, I like AI, it did a ton of value for me. We just need to find a way to live with it, without getting doomed in misinformation.
You do ultimately need to trust some sources to some degree. You can try to cross-correlate multiple sources (and this is in general a good habit!), but that depends on some level of trustworthiness in the sources you are looking at, you're not at all immune to misinformation by doing this (especially if multiple sources are, undisclosed, being generated from the same LLM. You can also get citenogenesis even pre-LLMs). And of course for some things it's possible to try to verify directly yourself, but this is infeasible to do for everything you depend on.
I feel for you. I was looking for some wildlife events on Youtube, only to find that all of them were AI generated, trying to get views. I can only find content somehow reliable if I put filter for content before of AI era.
Humans are also unreliable, we are competing for scarce attention, platforms decide what gets visibility and we cater to their algorithms. You could say humans are prompted by feed ranking AI - what and how to publish.
Sure! Who knows what's on that concrete curb! I'm clean if nothing else with my kitchenware. Just pass it over my natural gas stove's burner. Treat it the same as you do your pocket knife before/after you remove a bullet or shot from a wound. Sterile is best!
Why do we keep hearing about researchers and investigators - who all appear to be unaffiliated with OpenAI - discovering these new sites one by one?
All site URLs that the agent swarms used would appear in the network logs and be preserved by the first forensics team that looked into it.
Subpoena the logs.
reply