This is fun, but historically people have gone a bit overboard with saying that models like this, or n-gram language models, are anywhere close to large neural network models. There is certainly a connection though.
Yes, but it is a useful insight that both methods try to solve the same mathematical problem. It's better than thinking of LLMs as magic.
When you say "cross-entropy loss" people without stats background go to Wikipedia, take a glance, and adjust their mental model to "inscrutable magic".
Thinking of the main difference as the trade-off in how much CPU, memory and storage is allowed is not really wrong.
The part that is wrong is to think of gzip as a method that might reach similar complexity or generalization. And more importantly, to ignore the advanced way how training data gets curated or generated for (instructed, chain-of-thought) LLMs. But even then. The mental model that the LLM's goal is text compression is not wrong. The question to ask next is what kind of text it is expecting to compress.
Yep agree, good details, and this doesn't contradict my point above, about people going "overboard" with the comparison.
I do think when making these comparisons, it is worth emphasising that neural nets are really different. E.g. I used to see people equating LLMs to n-gram models, etc. which is overly simplistic, (especially in the early days when the models weren't as good).
More fragmented sleep, with people talking in the night, etc. I guess that's different from the low, unnatural, rumbles of cars and aeroplanes the majority of people have to put up with.
Really cool work. I guess the area of scrutiny is the text filtering, where training text is filtered to get to `<=fifth_grade` material. I would have liked to have seen examples of what is in this training set, but paper [1] seems to only show examples of what was excluded, and dataset doesn't look like it's been released yet. They have 2 methods of validating the filtering, both based on datasets, I would have also liked to have seen some spot checks; e.g. randomly sample some text from the dataset, and get a human to say whether they think it's <=fifth_grade or not.
(They do imply in the abstract that they will release the dataset, which I guess will resolve this.)
worth noting you don't usually read _that_ many books in your lifetime. e.g. 20 a books a year for 40 years is just 800 books
side note: i've noticed people writing with no caps, unusual grammar, splling mistakes, etc. which does differentiate their output from typical llm output - although ofc llms can probably imitate that
It's true and so it makes sense to be a bit picky about what you read. A great many non-fiction books should probably be blog posts or articles for example.
The article is heavily leaning on the paper "The Illusion of Thinking" [1].
It could be boiled down to: in 2025 this paper showed that "thought traces" in the models of the time could sometimes be inaccurate or misleading. Today they still might be, although OpenAI says actually they are accurate for their modern models, (based on internal research, rather than published research).
I was wondering whether this might have an explanation for why I don't really like major music, preferring minor / modal / non-functional harmony... it does:
> Throughout this derivation of different chords, you will note that a gradual progression or degradation from high-theme/low-complexity (Major Triad, Harmonic Series chords) to low-theme/high-complexity (Minor and Ambiguous chords). This progression is suggested by our measure of interestingness from Section 2.4 "Interestingness: Just Enough Complexity". That is, as we progress, more and more of the theme of the Harmonic Series is lost and more and more complexity is introduced. Notice that this progression seems to mirror that of musical sophistication as well: musically untrained listeners like Major chords while more musically trained listeners are more tolerant to loss of theme and more interested in complexity. (I met a signal processing engineer who had played piano for something like 18 years and who simply did not like Major chords at all.) Other fields seem to progress similarly: white wine is preferred by new wine drinkers, whereas more "complex" red wines are an acquired taste.
reply