Sagawa was mistaken in this article; he failed to appreciate the role of mutual information in computing, which is the proper basis for understanding Landauer's principle. I discussed this in https://www.mdpi.com/1099-4300/23/6/701.
There are a few folks working on this in neuroscience, e.g. training transformers to "decode" neural activity (https://arxiv.org/abs/2310.16046). It's still pretty new and a bit unclear what the most promising path forward is, but will be interesting to see where things go. One challenge that gets brought up a lot is that neuroscience data is often high-dimensional and with limited samples (since it's traditionally been quite expensive to record neurons for extended periods), which is a fairly different regime from the very large data sets typically used to train LLMs, etc.
This reminds me of a project I worked on during my PhD, where you create a network of scientific documents and notes/threads via Markdown, with a similarly structured "rabbit-hole" linking system: https://github.com/rkp8000/hypothesize . I'm not really a software engineer so never made it ready for general use, but I'm very happy to see a similar idea turned into something real! Kudos to the authors.
MSR has a very clear and accessible tutorial on quantum computing for anyone interested in getting up to speed with the fundamentals: https://www.youtube.com/watch?v=F_Riqjdh2oM .
Yep! This relationship is well known in statistical mechanics. I was just surprised that in many years of intersecting with information theory in other fields (computational neuroscience in particular) I'd never come across it before, even though IMO it provides an insightful perspective.
Connectome-adjacent neuroscientist here. Definitely not a dead end! But also definitely not the whole picture.
One of the main open questions in neuroscience right now is how network structure, dynamics, and function are related in the brain. Connectomes provide tremendous insight into structure, but as mentioned this does not generically solve either the dynamics or function problem. For example, for many of these neurons we don't have a good understanding of their input-output relationship, and the nature of this relationship can strongly affect the dynamics that emerge in a highly connected network. Individual variability across connectomes, and how connectomes change over development are also a significant issue, but at least for the fly it's thought that many of the basic structures are pretty conserved across adult animals, even if many of the details could differ.
Modulo these caveats, knowing the physical network structure of the brain does still impose huge constraints on what kinds of models we should be using for gaining insight into dynamics and function. For example, there are well known areas (the "mushroom bodies") with specific feed-forward connectivity patterns that are very different from a random recurrent network. Further, there are at least some areas in the fly brain where we think there are indeed quite clean structure-function relationships, e.g. in the central complex of the fly brain, which contains a physical ring of neurons and is thought to support a "bump" of activity that acts as a sort of compass that helps flies navigate via a ring-attractor-like dynamical system. Thus, even though it has many missing pieces, a wiring diagram like this can be tremendously useful for generating hypotheses to guide more targeted experiments and theoretical studies.
How's Open Worm coming along? The connectome of C. Elegans has been known for years, and Open Worm tries to simulate it. [1] Not with enormous success.
You know you would have thought all the years and years of "donations" to "cancer research" there would be constant news stories about how we accidentally cured a bunch of ancillary medical problems, and wow its all free to everyone because it was from donations!
Even without that I'm not sure why you think that's a good point — it's very easy to find serendipitous examples in medicine in general, e.g. viragra which was initially a heart treatment, or even thalidomide whose anti-cancer uses were suggested by the very birth defects that made it infamous.
Specifically cancer research finding other things by accident:
Was there any "productionization" of the "cure" for baldness & gray hair, after it was discovered 7 years ago? I reckon, there's a huge market for that cure.
I mean I get this is a joke, but it's still homophobic at its core, imo. Why riff on the same on argument against homosexuality when it would've been much more humorous to "turn it around" and joke about finding a cure for heterosexuality and research indicating that homosexuality should be the de facto state.
Tho tbf when I've joked about that with str8 friends they get really upsetti spaghetti, yet somehow still can't link their spaghettiness to why I was offended when they said "just don't act gay" when I said I would love to see Egypt but didn't want to travel there.
Very Nice. --from a Connectome-Centric neuroimager :)
One technique that I am pursuing right now is information decomposition of timeseries to separate the mutual information of two timeseries into redundant and synergistic informational atoms (synnergystic here means the degree to which knowing both timeseries gives you more information than the individual parts give (more than sum of parts). The big limitation of the method is the geometric explosion in complexity of the decomposition as the number of time series grow, with most analyses being limited to two or three times series at a time. However, the scale of the data on which it is applied is not requisite, meaning the approach can equally be used on the mutual information between two regions of interest in rsfMRI , or two spiking timeseries from individual neurons.
https://en.wikipedia.org/wiki/Partial_information_decomposit...
Thanks for your insight! How repeatable are these structures between individual animals? Are they very similar or is it more like “here’s a feed forward kinda bit, here’s a toroidal bit, and over here it’s just a mess”?
preprint coming out soon about this specifically :)
in the meantime, here's a simple tool paper we wrote explaining how you can treat this like a cool graph database challenge [1] and a preprint showing how you could approach that question when your number of samples per animal is close to N=1 [2]. basically..... it's hard! but also.... it's cool!
I love that Ridge Regression is introduced in the context of multicollinearity. It seems almost everyone these days learns about it as a regularization technique to prevent overfitting, but one of its fundamental use cases (and indeed its origin I believe) is in balancing weights among highly correlated (or nearly linearly dependent) predictors, which can cause huge problems even if you plenty of data.
I had the same thought. If a were constant I believe the solution would just be a convolution of b_t with an exponential filter. I think this is different because a_t depends on time, so it's equivalent to convolving b_t with a filter whose timescale changes at every timestep, which doesn't seem like something that's obviously parallelizable with FT.
reply