There isn't an obvious unsupervised problem to train medical imaging with.

What's the medical imaging equivalent to "predict the next word"?

It's the same thing. Predict the next pixel, or the next token (same way you handle regular images), or infill missing tokens (MAE is particularly cool lately). Those induce the abstractions and understanding which get tapped into.

There is none. But if the multimodal model is exposed to enough medical knowledge, it may be able to interpret images without specific training

Labelling data is easier, I think. It will just take a while...

Predict next entry in medical chart?

Presumably all these images would be connected with what ended up happening with the patient months or years later

If you has this level of data, wouldn’t it be trivial to label the images?

It's incredibly hard to disambiguate and accurately label images using the reports (area of my research).

Reports are also not analogous to ground truth labels, and you don't always have histopathologic/clinical outcomes.

You also have drift in knowledge and patient trends, people are on immunotherapy now and we are seeing complications/patterns we didn't see 5 years ago. A renal cyst that would have been follow-up to exclude malignancy before 2018 is now definitively benign, so those reports are not directly usable.

You would have to non-trivially connect this to a knowledge base of some form to disambiguate, one that doesn't currently exist.

And then there's hallucination.

Currently if you could even extract actionable findings, accurately summarize reports and integrate this with workflow you could have a billion dollar company.

Nuance (now owned by Microsoft) can't even autofill my dictation template accurately using free-text to subject headings.

I'm curious as to what your take on all this recent progress is Gwern. I checked your site to see if you had written something, but didn't see anything recent other than your very good essay "It Looks Like You’re Trying To Take Over The World."

It seems to me that we're basically already "there" in terms of AGI, in the sense that it seems clear all we need to do is scale up, increase the amount and diversity of data, and bolt on some additional "modules" (like allowing it to take action on it's own). Combine that with a better training process that might help the model do things like build a more accurate semantic map of the world (sort of the LLM equivalent of getting the fingers right in image generation) and we're basically there.[1]

Before the most recent developments over the last few months, I was optimistic on whether we would get AGI quickly, but even I thought it was hard to know when it would happen since we didn't know (a) the number of steps or (b) how hard each of them would be. What makes me both nervous and excited is that it seems like we can sort of see the finish line from here and everybody is racing to get there.

So I think we might get there by accident pretty soon (think months and not years) since every major government and tech company are likely racing to build bigger and better models (or will be soon). It sounds weird to say this but I feel like even as over-hyped as this is, it's still under-hyped in some ways.

Would love your input if you'd like to share any thoughts.

[1] I guess I'm agreeing with Nando de Freitas (from DeepMind) who tweeted back in May 2022 that "The Game is Over!" and that now all we had to do was scale things up and tweak: https://twitter.com/NandoDF/status/1525397036325019649?s=20

Perhaps, I'm admittedly not an expert in identifying use cases of Unsupervised Learning yet. My hunch would be that the lack of the labels would require orders of magnitude more data and training to produce an equivalent model, which itself will be a sticky point for health tech. companies.