It's great that people are starting to take continual learning seriously, and it seems like Jessy has been thinking about LLMs and continual learning longer than almost anyone.
I especially like this taxonomy
> I think of continual learning as two subproblems:
> Generalization: given a piece of data (user feedback, a piece of experience, etc.), what update should we do to learn the “important bits” from that data?
> Forgetting/Integration: given a piece of data, how do we integrate it with what we already know?
My personal feeling is that generalization is a data issue: given a datapoint x, what are all the examples in the distribution of things that can be inferred from x? Maybe we can solve this with synthetic datagen. And forgetting might be solvable architecturally, e.g. with Cartridges (https://arxiv.org/abs/2506.06266) or something of that nature.
Yeah, the standard SPLADE model trained from BERT typically already has a vocabulary/vector size of 30,552. If the SPLADE model is based on a multilingual version of BERT, such as mBERT or XLM-R, the vocabulary size could inherently expand to approximately 100,000, as does the vector size.
If you consider the actual latent space the full higher dimensional representation, and you take the first principle component, the other vectors are zero. Pretty sparse. No it's not a linked list sparse matrix. Don't be a pedant.
When you truncate Matryoshka embeddings, you get the storage benefits of low-dimensional vectors with the limited expressiveness of low-dimensional vectors. Usually, what people look for in sparse vectors is to combine the storage benefits of low-dimensional vectors with the expressiveness of high-dimensional vectors. For that, you need the non-zero dimensions to be different for different vectors.
It seems to be an error with the classifier. Sorry everyone. I probably shouldn't have posted that graph; I knew it was buggy, I just thought that the Perl part might be interesting to people.
Hi again. I had already written about this later in my blog post (which is unrelated to this thread), but the point was that RLHF hadn't been applied to language models at scale until InstructGPT. I edited the post just now to clarify this. Thanks for the feedback!
Its a bit like saying algorithms don't matter for solving computational problems. Two different algorithms might produce equivalent results but if you have to wait years for an answer, when seconds matter, the slow algorithm isn't helpful.
I believe the current approach of using mostly a feed-forward in the inference stage, with well-filtered training data and backpropagation for discrete "training cycles" has limitations. I know this has been tried in the past, but something modelling how animal brains actually function, with continuous feedback, no explicit "training" (we're always being trained), might be the key.
Unfortunately our knowledge of "whats really going on" in the brains is still limited, investigative methods are crude as the brain is difficult to image at the resolution we need, and in real time. Last I checked no one's quite figured out how memory works, for example. Whether its "stored in the network" somehow through feedback (like a SR-latch or flip-flop in electronics) or whether there's some underlying chemical process within the neuron itself (we know that chemicals definitely regulate brain function, don't know how much it goes the other way and it can be used to encode state)
> I don't think architecture matters. It seems to be more a function of the data somehow.
of course it matters
if I supply the ants in my garden with instructions on how to build tanks and stealth bombers they're still not going to be able to conquer my front room
I recently wrote a post about scaling RL that has some similar ideas:
> How to Scale RL to 10^26 FLOPs (blog.jxmo.io/p/how-to-scale-rl-to-1026-flops)
The basic premise behind both essays is that for AI to make another big jump in capabilities, we need to find new data to train on.
My proposal was reusing text from the Internet and doing RL on next-token prediction. The linked post here instead suggests doing 'replication training', which they define as "tasking AIs with duplicating existing software products, or specific features within them".
reply