I’ve played with Slint and it’s nice but depending on how this plays out it might be a concern. QML is of course much more mature than Slint. It would be interesting to read an in depth blog post by Slint about why fundamentally Slint is better than QML since one of the main advantages (multi-language) is potentially being nullified here.
That’s absurd, that’s like saying we should only use C++ for backend code because my CRUD business app might one day scale to infinity. Better be safe than sorry and sling pointers and CMake just in case I need that extra juice!
imo, even if the only "interactivity" a web app has is just a login page, then even that alone is enough to warrant using a framework rather than doing direct DOM manipulation (or even worse, full page refreshes after a form submit).
It's not about using the most powerful tool always, it's about knowing how to leverage modern standards rather than reinventing and solving problems that are already solved.
> ...a web app has is just a login page, then even that alone is enough to warrant using a framework rather than doing direct DOM manipulation (or even worse, full page refreshes
Maybe, but it doesn't necessarily need to be a SPA framework though. There are simpler libraries/frameworks like htmx that considerably reduce complexity and also let you avoid direct DOM manipulation.
Yeah, the debate about which framework is best is like debating which programming language is best, in that they all have pros and cons, that can be debated endlessly.
However, in general, whatever's most popular is a strong signal (like a crowd-sourced signal) that it's probably indeed "the best" all around, which are two words that can also, of course, be debated endlessly based on one's personal definition of "best".
Strange comment, there’s plenty of videos of both locations on YouTube to make the comparison and I think it’s quite apt. Chinese (and other SEA) major cities definitely feel much more modern than most American cities these days. Most American metropolitan areas are quite bland/bleak outside the “beautified” green areas.
> A trip to one of the major cities in China made it clear to me that they are ahead of the world right now
Sorry, I should have been more clear, this is what I was referencing. I have been to SF recently and would agree it's not hard to make a lot of cities look better in comparison.
I also married in my early twenties (21 actually) and am still happily married over a decade later. No regrets for me; I was lucky to find someone compatible early on and we were able to grow together. Yes undoubtedly you sacrifice some experiences choosing this path but I cherish that bond and antidote to loneliness. There's something comforting about having someone so close for important parts of your life spanning a long time. A real partner to the challenges so you don't have to do everything alone.
I was hesitant to share but I thought it would be better in the end. We often hear of negative experiences (divorce, abuse, etc.) but I regularly hear about couples quietly enjoying their long relationships together. Feels right to offer a counter-point to those stories to say it's not always a mistake.
How about being a dick makes you less inclined to have children? I’m inclined to believe selfish people are less inclined to have children (there will be ‘some’ that see them as a trophy, but fewer than the ones that don’t want them at all).
I have a kid, and I'm very sad about the state of the world my daughter going to live. I was relatively lucky, she is not. If I didn't have her, I'm sure I would care much much less about stuff I failed to change for the better, it would be much easier to say "f.k it, that's natural order of things, history repeats itself", give up and live my life fast and lose.
Surprisingly the Russian army is mostly volunteer since the average pay right now is quite high by Russian standards. There are also standard conscripted soldiers as part of the country’s required service. Allegedly the conscripts are kept away from front lines since the death of conscripted young men in Afghanistan was part of the political death knell that led to the fall of the Soviet Union.
Unfortunately Ukraine is having to rely more on involuntary conscription to fill the ranks as volunteer numbers have dwindled. There are many documented cases of TASS “kidnapping” military age men.
I don’t see the parent poster blaming anybody. Maybe you can say they provided a one-sided view but what they wrote was factual.
You are right, building a good UI for the desktop has become exceedingly difficult. In my experience new UI toolkits try to mimic the web experience (kivy, QML, Slint, Flutter, etc.) and end up being threadbare with the simplest widgets available. In my opinion, every new UI toolkit since QtWidgets/WinForms/Delphi/WPF/Win32/Gtkmm era has missed the point. The desktop is a power tool and requires powerful widgets. Virtualized lists, data grid with complex interactions, drag and drop, OS integration, modals, background tasks, docking windows, etc. A toolkit that puts a slider, some text and buttons on the screen doesn’t solve the harder problems and the web browser will run circles around that workflow. Easier UX can be built in anything really. Until there’s a desktop app framework that solves the hard problems in desktop UI dev then desktop will languish and hard stuff will continue to be hard.
I'm developer of Slint and I'm glad to see Slint mentioned.
I want to clarify that we're not trying to mimic the web. We have a great vision for the desktop integration, but we unfortunately have limited resources. Our team is fairly small and we are mainly working with paying customers to pay the bills, which are mostly in embedded space at the moment.
I agree with your points, and while there’s still a lot to do, I hope Slint can become a strong option for building powerful complex desktop applications.
I like Slint, the technical work is extremely impressive. I even contributed to the project! I hope you can realize those goals, you would fill a massive vacuum in the wider ecosystem.
Linus's reply is perfect in tone and hopefully will settle this issue.
He is forceful in making his points, but respectful in the way he addressed Christoph's concerns.
This gives me great hope that the Linux maintainer community and contributers using Rust will be able to continue working together, find more common ground, and have more success.
The response addressed Christoph's concerns _in word_.
According to the policy, rust folks should fix the rust binding when C changes breaks the binding. The C maintainer don't need to care rust, at all.
In practice, though, I would expect this needs lots of coordination. A PR with C only changes that breaks the whole building (because rust binding is broken) is unlikely to be merged to mainline.
Linus can reiterate his policy, but the issue can't be resolved without some rust developers keep on their persistent work and builds up their reputation.
> rust folks should fix the rust binding when C changes breaks the binding
I have never understood how that could work long-time. How do you release a kernel, where some parts are broken? Either you wait for Rust people to fix their side or you drop the C changes. Or your users suddenly find their driver doesn‘t work anymore after a kernel update.
As a preliminary measure when there isn‘t a substantial amount of Rust code, yet, sure. But the fears of some maintainers that the policy will change to "you either learn Rust and fix things or your code can be held up until someone else helps you out" are well-founded, IMO.
Are you familiar with Linux kernel development process? Features can be merged only in two weeks long merge window. After the merge window closes, only fixes are merged for eight weeks. Rust binding can be fixed in that time. I don't see any problems.
That's a gross simplificaftion of the development process. Yes, new features are mostly merged in that two-weeks window -- but you're now talking about the Linux release management process more than its development.
Before features are merged to Linus' release branch, pretty much all changes are published and merged to linux-next first. It is exactly here that build issues and conflicts are first detected and worked out, giving maintainers early visibility into changes that are happening outside their subsystem. Problems with the rust bindings will probably show up here, and the Rust developers will have ample time to fix/realign their code before the merge window even starts. And it's not uncommon for larger features (e.g. when they require coordination across subsystems) to remain in linux-next for more than one cycle.
And if no Rust developer has time or interest in those eight weeks? I don‘t claim that it can never work (or it cannot work in the common case), but as a hard rule it seems untenable.
> And if no Rust developer has time or interest in those eight weeks?
What if Linus decided to go on a two month long vacation in the middle of the merge window?
> I don‘t claim that it can never work (or it cannot work in the common case), but as a hard rule it seems untenable.
There are quite a few rust developers already involved, if they cannot coordinate that at least some are available during a release critical two month period then none of them should be part of any professional project.
I'm not familiar with kernel development, but what's the difference anyway with C code? If you change the interface of some part, any users of it will be broken Rust or not. It will require coordination anyway.
It is customary for maintainers to fix _all_ usage of their code themselves? That doesn't seem scalable.
Yes, that is the custom and is a key advantage of getting drivers in tree. I believe often the changes are applied automatically with a tool like coccinelle,
Keep in mind that actual breaking changes are by design incredibly rare in a project like the linux kernel. If you have a decade's-worth of device drivers depending on your kernel subsystem's API, you don't get to break them, you have to introduce a new version instead.
I think it's more a degree of how much effort it is to adjust to the new interface. If it's just 'added a new parameter to a function and there's an obvious default for existing code', then it'll (potentially mechanically) be applied to all the users. If it's 'completely changed around the abstraction and you need to think carefully about how to port your driver to the new interface', then that's something where there needs to be at least some longer-term migration plan, if only because there's not likely one person who can actually understand all the driver code and make the change.
(I do have experience with this causing regressions: someone updates a set of drivers to a new API, and because of the differences and lack of a good way to test, breaks some detail of the driver)
This isn't true; internal API's change all the time (e.g. adding extra arguments) Try running out of tree drivers on bleeding edge kernels to see for yourself.
Of course, for trivial mechanical changes like adding an argument the Rust binding changes are also trivial. If you've just spent half an hour walking through driver code for hardware you've never heard of changing stuff like
Then it's not beyond the wits of a C programmer to realise that the Rust binding
quaff(var1, n, maybe_doop) ... can be ... quaff(var1, n, 0, Q_DEFAULT | (maybe_doop << 4))
Probably the Rust maintainer will be horrified and emit a patch to do something more idiomatic for binding your new API but there's an excellent chance that meanwhile your minimal patch builds and works since now it has the right number and type of arguments.
I have never used Coccinelle but yes, sort of. However, you're on the hook for the patch you submit, Coccinelle isn't a person so if you blindly send out a patch Coccinelle generated, without even eyeballing it, you should expect some risk of thrown tomatoes if (unknown to you) this utterly broke some clever code using your previous API in a way you hadn't anticipated in a driver you don't run.
Because if in a few years I have a device whose driver is written in Rust, a new kernel version might have simply dropped or broken my device driver, and I cannot use my device anymore. But sure, if R4L wants to stay a second-class citizen forever, it can still be acceptable.
> Because if in a few years I have a device whose driver is written in Rust, a new kernel version might have simply dropped or broken my device driver, and I cannot use my device anymore.
At least for Debian, all you need to do if you hit such a case is to simply go and choose the old kernel in the Grub screen. You don't even need to deal with installing an older package and dealing with version conflicts or other pains of downgrading.
For my server or laptop at home, sure. Why not. For servers in commercial fleets you should have staged rollouts as a policy anyway so if you do it right you shouldn't get hit.
I think the way you do this is set things up so that no bits that are written in Rust are built by default, and make sure that the build system is set up such that Rust bindings for C code are only built when there's Rust code that's enabled that requires them.
Then sure, some people who download a kernel release might enable a Rust driver, and that makes the build fail. But until Rust is considered a first-class, fully-supported language in the kernel, that's fine.
In practice, though, I would expect that the Rust maintainers would fix those sorts of things up before an actual release is cut, after the 2-week merge window, during the period when only fixes are accepted. Maybe not every single time, but most of the time; if no one is available to fix a particular bit of breakage, then it's broken for that release. And that's fine too, even if it might be annoying to some users.
> I think the way you do this is set things up so that no bits that are written in Rust are built by default, and make sure that the build system is set up such that Rust bindings for C code are only built when there's Rust code that's enabled that requires them.
Which is currently the only way possible and it will stay that way for a long time because remember that clang support less targets than gcc and gcc cannot compile Rust.
Once gcc can /reliably/ compile Rust, then and only then Rust could be "upgraded" to a first class citizen in Linux. The "C-maintainers don't want to learn Rust" issue, will still be here of course, but there will already be many years of having a mixed code base..
Because nothing is forcing a distro to adopt a kernel that has items that are broken. Not a lot of folks out there are manually compiling and deploying standalone kernels to production systems.
C can break rust, and Debian/Ubuntu/Redhat/suse/etc can wait for it to be fixed before pushing a new kernel to end users.
You can merge it into your branch as e.g. the DMA maintainer, then the rust folk can pull your changes and fix the bindings. Maybe you as maintainer could give them a heads up and a quick consideration of the error.
Yes, Rust as something optional doesn't really make sense long term. Either it will continue to only be used in nieche drivers (in which case why bother?) or eventually you need to build Rust code to have a usable kernel for common hardware. Any promises to the contrary need to be backed up with more than "trus me bro".
That's the theory. However, isn't likely that as things like the new Nova Nvidia driver is written in Rust, the things that depend on Rust are suddenly so important, that shipping with it disabled is unrealistic, even without a policy change. (I don't think this is bad)
Rust for Linux is currently experiment. If a larger number of widely-used drivers get written in Rust and developers prefer writing them in Rust over C, then I guess it's time to declare the experiment a success and flip the switch?
Linus is one of the few people who can forcefully argue the case for moderation, and I've recognized some of the lines I've used to shift really contentious meetings back into place. There's the "shot-and-chaser" technique (a) this is what needs to happen now for the conversation...
"I respect you technically, and I like working with you[...] there needs to be people who just stand up to me and tell me I'm full of shit[...] But now I'm calling you out on YOURS."
...and (b) this is me recognizing that me taking charge of a conversation is a different thing than me taking control of your decisions:
"And no, I don't actually think it needs to be all that
black-and-white."
(Of course Linus has changed over time for the better, he's recognized that, and I've learned a lot with him and have made amends with old colleagues.)
I really liked this reply from Torvalds. I've seen a lot of his older rants, and while I respect his technical achievements, it really turned me off on the guy himself. I was skeptical of his come-to-jesus moment back in 2018 (or whenever it was), but these days it's great to read his measured responses when there's controversy.
It's really cool to see someone temper their language and tone, but still keep their tell-it-like-it-is attitude. I probably wouldn't feel good if I were Christoph Hellwig reading that reply, but I also wouldn't feel like someone had personally attacked me and screamed at me far out of proportion to what I'd done.
The whole "I respect you technically, and I like working with you." in the middle of being firm and typing in caps is such a vibe shift from the Linus of a decade ago. We love to see it!
My impression is that this has always been part of the core of his character, but he had to learn to put it into writing.
Contrast this to people who are good at producing the appearance of an upstanding character when it suits them, but being quite vindictive and poisonous behind closed doors when it doesn't.
Yeah, from my limited view point it really looks like Linus is a genuine person. He says what he thinks and there are no hidden agendas. That is very refreshing in current times.
In my eyes, this sentence by Linus is as straightforward as it gets. But if you think it's corporate lie-speak, I wonder which words you'd interpret as genuine - because there has to be a way to get something like that across if you really mean it, or we're all doomed. :)
I have always thought Linus may not like Rust or at least not Pro-Rust, and the only reason Rust is marching inside the Kernel is most of his close lieutenant are extremely pro rust. So there is this Rust experiment.
But looking at all the recent responses it seems Rusted Linux is inevitable. He is Pro Rust.
There was no reason to ever think otherwise. The experiment wouldn't have happened if he didn't want to try it and give it a certain amount of backing. He's been pretty vocal about his motivations.
> C++ isn't going to give us any of that any decade soon, and the C++ language committee issues seem to be pointing out that everyone better be abandoning that language as soon as possible if they wish to have any codebase that can be maintained for any length of time.
I'm pretty sure there is significant pressure from corporate sponsors in the Linux foundation to make Rust happen. That includes Google, Microsoft, AWS, ...
I think the pressures are from everywhere all the time. There's always a risk that something picks up enough steam to replace an OS even if it is a derivative or fork of the same OS. For C++ adoption, Linus gauged correctly that it's steam is limited and even directly taunting its supporters had low risk in a project that could challenge Linux because its set of choices over time has had many foot guns, where many developers would intentionally use what any user would consider a foot gun. As long as Linus didn't step in to define strong rules for how C++ would be used there was no likely challenger over any reasonable amount of time (for Linux/C). This is not true for rust.
Linus was pretty clear about his views on working with vendors, etc, in his Autobiography so I'm not sure if/why there would be surprise.
Solaris, Java and C/C++ conpilers were all owned by Sun. I feel that is a situation without anyone even trying to maintain whatever separation some might expect from Linus.
"Rusted Linux is inevitable" for a good reason because Rust is objectively good language or better compared to C (Rust is designed what to fix flaws many language has)
- Systems not supported by Rust can use older kernels. They can also -- at least for a while -- probably still use current kernel versions without enabling any Rust code. (And presumably no one is going to be writing any platform-specific code or drivers for a platform that doesn't have a Rust toolchain.)
- It will be a long time before building the kernel will actually require Rust. In that time, GCC's Rust frontend may become a viable alternative for building Linux+Rust. And any arch supported by GCC should be more-or-less easily targetable by that frontend.
- The final bit is just "tough shit". Keep up, or get left behind. That could be considered a shame, but that's life. Linux has dropped arch support in the past, and I'm sure it will do so in the future. But again, they can still use old kernels.
As for new architectures in the future, if they're popular enough to become a first-class citizen of the Linux kernel, they'll likely be popular enough for someone to write a LLVM backend for it and the glue in rustc to enable it. And if not, well... "tough shit".
Thinking pragmatically, the legacy systems where there is no current rust toolchain most likely do not need the drivers and components that are being written in rust.
Unless you somehow want to run Apple M1 GPU drivers on a device that has no rust toolchain ... erm...
or you want to run a new experimental filesystem on a device that has no rust toolchain support?
The answer to the "new and emerging platforms" question is pretty much the same as before: sponsor someone to write the toolchain support. We've seen new platforms before and why shouldn't it follow the same pathway? Usually the c compiler is donated by the company or community that is investing into the new platform (for example the risc-v compiler support for gcc and llvm are both getting into maturity status, and the work is sponsored by the developer community, various non-profit[1][2] and for-profit members of the ecosystem as well as from the academic community.)
realistically speaking, it's very hard to come up with examples of the hypothetical.
I suspect gcc-rs will be in good working order for a few years before any kernel subsystems require a Rust compiler to build; if the legacy system can't run a recent GCC, why does it need a much-newer kernel? (e.g., how would it cope with the kernel requiring an additional GCC extension, bumping the minimum standard version of C, etc.)
I honestly suspect new architectures will be supported in LLVM before GCC nowadays; most companies are far more comfortable working with a non-GPL toolchain, and IMHO LLVM's internals are better-documented (though I've never added a new target).
> What about legacy systems, which may not have a rust toolchain?
Linux's attitude has always been either you keep up or you get dropped - see the lack of any stable driver API and the ruthless pruning of unmaintained drivers.
> What about new architectures that may come up in the future?
Who's to say they won't have a Rust compiler? Who's to say they will have a C one?
Linux also cant be built by any minimal c compiler for obscure arch, it requires many gcc extensions. Its only because llvm added them that its also can be compiled with llvm
> Linux's attitude has always been either you keep up or you get dropped
Gonna need a citation on that one. Drivers are removed when they don't have users anymore, and a user piping up is enough to keep the driver in the tree:
For example:
> As suggested by both Greg and Jakub, let's remove the ones that look
> are most likely to have no users left and also get in the way of the
> wext cleanup. If anyone is still using any of these, we can revert the
> driver removal individually.
Or the x32 platform removal proposal, which didn't happen against after some users showed up:
> > > I'm seriously considering sending a patch to remove x32 support from
> > > upstream Linux. Here are some problems with it:
> >
> > Apparently the main real use case is for extreme benchmarking. It's
> > the only use-case where the complexity of maintaining a whole
> > development environment and distro is worth it, it seems. Apparently a
> > number of Spec submissions have been done with the x32 model.
> >
> > I'm not opposed to trying to sunset the support, but let's see who complains..
>
> I'm just a single user. I do rely on it though, FWIW.
> […snipped further discussion]
Curious: what widely-used (Linux) legacy systems do not have a Rust toolchain?
In the end the question is whether you want to hold back progress for 99.9% of the users because there are still 200 people running Linux on an Amiga with m68k. I am pretty sure that the number of Linux on Apple Silicon users outnumbers m68k and some other legacy systems by at least an order of magnitude (if not more). (There are currently close to 50000 counted installs. [1])
I think that'll become a question if/when rust starts to move closer to core parts of the kernel, as opposed to platform-specific driver code. It's already been considered for filesystems which could in theory run on those systems, and the project seems to be OK with the idea that it's just not supported on those platforms. But that's likely a long way off, after there's a significant body of optional rust code in the kernel, and the landscape may already be quite different at that point (both in terms of if those systems are still maintained, and in terms of the kind of targets rust can support, especially if the gcc backend matures)
Those are the tradeoffs, and it seems to me that Linux doesn't have to run in everything under the Sun as Doom ports do, and there might be other kernels that are better suited to such cases.
"But why though? What about legacy systems" their called legacy for a reason right
I'm sorry you cant hinder kernel development just because some random guy/corpo cant use your shit in obscure system, like how can that logic is apply to everything
There are some platforms which linux supports that LLVM does not (and GCC does). There is quite a lot of effort in making a decent LLVM backend, and these older systems tend to have relatively few maintainers, so there may not be the resources to make it happen.
> There is quite a lot of effort in making a decent LLVM backend, and these older systems tend to have relatively few maintainers
Well, it also takes effort to be held back with outdated tools. Also, the LLVM backend doesn't have to be top-notch, just runnable. If they want to run legacy hardware they should be okay with running a legacy or taking the performance hit of a weaker LLVM back-end.
Realistically
At version 16[1], LLVM supports:
* IA-32
* x86-64
* ARM
* Qualcomm Hexagon
* LoongArch
* M68K
* MIPS
* PowerPC
* SPARC
* z/Architecture
* XCore
* others
in the past it had support for Cell and Alpha, but I'm sure that the old code could be revived if needed, so how many users are effected here? Lets not forget the Linux dropped Itanium support and I'm sure someone is still running that somewhere.
Looking through this list [2], what I see missing is Elbrus, PA-RISC, OpenRisc, and SuperH. So pretty niche stuff.
Boy that response would've been helpful like a week ago, before several key Rust maintainers resigned in protest due to Linus's radio silence on the matter.
Huh, thanks. Really good to know where Linus stands here. Seems to me like Linus is completely okay with introduction of Rust to the kernel and will not allow maintainers blocking its adoption.
Really good sign. Makes me hopeful about the future of this increasingly large kernel
This is indeed an excellent response and will hopefully settle the issues. Aside from the ones already settled by Linus's previous email, such as whether social media brigading campaigns are a valid part of the kernel development process.
Honestly I was waiting for a reply from Linux like this to put Hellwig in his place.
> The fact is, the pull request you objected to DID NOT TOUCH THE DMA
LAYER AT ALL.
> It was literally just another user of it, in a completely separate
subdirectory, that didn't change the code you maintain in _any_ way,
shape, or form.
> I find it distressing that you are complaining about new users of your
code, and then you keep bringing up these kinds of complete garbage
arguments.
Finally. If he had been sooner maybe we wouldn't have lost talented contributors to the kernel.
I've made that mistake, and the inverse, often enough that I try to make sure to check I've written the correct word... and I still mess it up. Between the words being similar and the 'x' being right next to the 's' on US keyboard, it's bound to happen.
ON the flip side - when I (and I suspect many others) read Linux where Linus should be written, I rarely even notice and never really care because I've been there.
All this is a long winded way of saying: don't sweat it :) .
> Finally. If he had been sooner maybe we wouldn't have lost talented contributors to the kernel.
I feel that departure of the lead R4L developer was a compromise deliberately made to not make Hellwig feel like a complete loser. This sounds bad of course.
no lead R4L left because of the current situation. Marcan was the lead of Asahi Linux, not R4L.
Wedson (which was one of the leads of R4L) left some time ago, before all of this, and his problem was not with Hellwig (or, at least it was not the one that brought the last drop).
Marcan quitting wasn't a compromise, the resignation of a maintainer would never be used that way. Dude was just burnt out. I don't blame him at all, hopefully some time away from the situation does him some good.
I work in fintech and we replaced an OCR vendor with Gemini at work for ingesting some PDFs. After trial and error with different models Gemini won because it was so darn easy to use and it worked with minimal effort. I think one shouldn't underestimate that multi-modal, large context window model in terms of ease-of-use. Ironically this vendor is the best known and most successful vendor for OCR'ing this specific type of PDF but many of our requests failed over to their human-in-the-loop process. Despite it not being their specialization switching to Gemini was a no-brainer after our testing. Processing time went from something like 12 minutes on average to 6s on average, accuracy was like 96% of that of the vendor and price was significantly cheaper. For the 4% inaccuracies a lot of them are things like the text "LLC" handwritten would get OCR'd as "IIC" which I would say is somewhat "fair". We probably could improve our prompt to clean up this data even further. Our prompt is currently very simple: "OCR this PDF into this format as specified by this json schema" and didn't require some fancy "prompt engineering" to contort out a result.
Gemini developer experience was stupidly easy. Easy to add a file "part" to a prompt. Easy to focus on the main problem with weirdly high context window. Multi-modal so it handles a lot of issues for you (PDF image vs. PDF with data), etc. I can recommend it for the use case presented in this blog (ignoring the bounding boxes part)!
This is spot on, any legacy vendor focusing on a specific type of PDF is going to get obliterated by LLMs. The problem with using an off-the-shelf provider like this is, you get stuck with their data schema. With an LLM, you have full control over the schema meaning you can parse and extract much more unique data.
The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?"
You could improve your accuracy further by adding some chain-of-thought to your prompt btw. e.g. Make each field in your json schema have a `reasoning` field beforehand so the model can CoT how it got to its answer. If you want to take it to the next level, `citations` in our experience also improves performance (and when combined with bounding boxes, is powerful for human-in-the-loop tooling).
Disclaimer: I started an LLM doc processing infra company (https://extend.app/)
> The problem then shifts from "can we extract this data from the PDF" to "how do we teach an LLM to extract the data we need, validate its performance, and deploy it with confidence into prod?"
A smart vendor will shift into that space - they'll use that LLM themselves, and figure out some combination of finetunes, multiple LLMs, classical methods and human verification of random samples, that lets them not only "validate its performance, and deploy it with confidence into prod", but also sell that confidence with an SLA on top of it.
That's what we did with our web scraping saas - with Extraction API¹ we shifted web scraped data parsing to support both predefined models for common objects like products, reviews etc. and direct LLM prompts that we further optimize for flexible extraction.
There's definitely space here to help the customer realize their extraction vision because it's still hard to scale this effectively on your own!
What's the value for a customer to pay a vendor that is only a wrapper around an LLM when they can leverage LLMs directly? I imagine tools being accessible for certain types of users, but for customers like those described here, you're better off replacing any OCR vendor with your own LLM integration
I wonder about these takes. Have you never worked in a complex system in a large org before?
OK, sure, we can parse a PDF reliably now, but now we need to act on that data. We need to store it, make sure it ends up with the right people who need to be notified that the data is available for their review. They then need to make decisions upon that data, possible requiring input from multiple stakeholders.
All that back and forth needs to be recorded and stored, along with the eventual decision and the all supporting documents and that whole bundle needs to be made available across multiple systems, which requires a bunch of ETLs and governance.
I don't know what light cones or dog and pony mean here but I'm interested in your take - would you care to expand a bit on how the future can reshape that very complicated set of steps and humans described in the parent?
Yesterday I read an /r/singularity post in awe cus of a screenshot of a lead management platform from OAI in a japan convention supposedly meant a direct threat to SalesForce. Like, yeah sure buddy.
I would say most acceleracionist/AI bulls/etc don't really understand the true essential complexity in software development. LLMs are being seen as a software development silver bullets, and we know what happens with silver bullets.
I assume this is a slap intended to imply that ai actually IS a silver bullet answer to the parent's described problem and in just 18 months they will look back and realize how wrong they are.
Is that what you mean and, if so, is there anything in particular you've seen that leads you to see these problems being solved well or on the 18 month timeline? That sounds interesting to look at to me and I'd love to know more.
It isn't a silver bullet in that it can just "make software" but it is changing the entire dynamic.
You can't do point sampling to figure out where things are going. We have to look at the slope. People see a paper come out, look at the results and say, "this fails for x, y and z. doesn't work", that is now how scientific research works. This is why two minute papers has the tag line, "hold on to your papers ... two papers down the line ..."
Copy and paste the whole thread into a SOTA model and have meta me explain it.
That's not why more experienced people are doubting you.
They're doubting you because the non-digital portions of processes change at people/org speed.
Which is to say that changing a core business process is a year political consensus, rearchitecture, and change management effort, because you also have to coordinate all the cascading and interfacing changes.
> changing a core business process is a year political consensus, rearchitecture, and change management effort
You are thinking within the existing structures, those structures will evaporate. All along the software supply chain, processes will get upended, not just because of how technical assets will be created, but also how organizations themselves are structured and react and in turn how software is created and consumed.
This is as big as the invention of the corporation, the printing press and the industrial revolution.
I am not here to tutor people on this viewpoint or defend it, I offer it and everyone can do with it what they will.
If I'm an autobody shop or some other well-served niche, how unhappy with them do I have to be to decide to find a replacement, either a competitor of theirs that used an LLM, or bring it in house and go off and find a developer to LLM-acceleratedly make me a better shopmonkey? And there are the integrations. I don't own a low hanging fruit SaaS company, but it seems very sticky, and since the established company already exists, they can just lower prices to meet their competitors.
B2B is different from B2C, so if one vendor has a handful of clients and they won't switch away, there's no obliterating happening.
What's opened up is even lower hanging fruit, on more trees. A SaaS company charging $3/month for the left-handed underwater basket weaver niche now becomes viable as a lifestyle business. The shovels in this could be supabase/similar, since clients can keep access to their data there even if they change frontends.
Which means that the current vc-software-ecosystem is the walking dead. The front end webdev is now going to do things that previously took a 10 person startup.
There's still time. The situation in which you can effectively replace your OCR vendor with hitting LLM APIs via a half-assed Python script ChatGPT wrote for you, has existed for maybe few months. People are only beginning to realize LLMs got good enough that this is an option. An OCR vendor that starts working on the shift today, should easily be able to develop, tune, test and productize an LLM-based OCR pipeline way before most of their customers realize what's been happening.
But it is a good opportunity for a fast-moving OCR service to steal some customers from their competition. If I were working in this space, I'd be worried about that, and also about the possibility some of the LLM companies realize they could actually break into this market themselves right now, and secure some additional income.
EDIT:
I get the feeling that the main LLM suppliers are purposefully sticking to general-purpose APIs and refraining from competing with anyone on specific services, and that this goes beyond just staying focused. Some of potential applications, like OCR, could turn into money printers if they moved on them now, and they all could use some more cash to offset what they burn on compute. Is it because they're trying to avoid starting an "us vs. them" war until after they made everyone else dependent on them?
To the point after your edit, I view it like the cloud shift from IaaS to PaaS / SaaS. Start with a neutral infrastructure platform that attracts lots of service providers. Then take your pick of which ones to replicate with a vertically integrated competitor or manager offering once you are too big for anyone to really complain.
Never underestimate the power of the second mover. Since the development is happening in the open, someone can quickly cobble up the information and cut directly to the 90% of the work.
Then your secret sauce will be your fine tunes, etc.
Like it or not AI/LLM will be a commodity, and this bubble will burst. Moats are hard to build when you have at least one open source copy of what you just did.
I have some out-of-print books that I want to convert into nice pdf's/epubs (like, reference-quality)
1) I don't mind destroying the binding to get the best quality. Any idea how I do so?
2) I have a multipage double-sided scanner (fujitsu scansnap). would this be sufficient to do the scan portion?
3) Is there anything that determines the font of the book text and reproduces that somehow? and that deals with things like bold and italic and applies that either as markdown output or what have you?
4) how do you de-paginate the raw text to reflow into (say) an epub or pdf format that will paginate based on the output device (page size/layout) specification?
In my open source tool http://docrouter.ai I run both OCR and LLM/Gemini, using litellm to support multiple LLMs. The user can configure extraction schema & prompts, and use tags to select which prompt/llm combination runs on which uploaded PDF.
LLM extractions are searched in OCR output, and if matched, the bounding box is displayed based on OCR output.
With the docrouter.ai, it can be installed on prem. If using the SAAS version, users can collaborate in separate workspaces, modeled on how Databricks supports workspaces. Back end DB is Mongo, which keeps things simple.
One level of privacy is the workspace level separation in Mongo. But, if there is customer interest, other setups are possible. E.g. the way Databricks handles privacy is by actually giving each account its own back end services - and scoping workspaces within an account.
If you're using LLM APIs there are SLAs from the vendors to make sure your inputs are not used as training data and other guarantees. Generally these endpoints cost more to use (the compliance fee essentially) but they solve the problem.
As a mere occasional customer I've been scanning 4 to 5 pages of the same document layout every week in gemini for half a year, and every single week the results were slightly different.
To note the docs are bilingual so it could affect the results, but what stroke me is the lack of consistency, and even with the same model, running it two or three times in a row gives different results.
That's fine for my usage, but that sounds like a nightmare if everytime Google tweaks their model, companies have to reajust their whole process to deal with the discrepancies.
And sticking with the same model for multiple years also sound like a captive situation where you'd have to pay premium for Google to keep it available for your use.
Inconsistency comes from scaling - if you are optimizing your infra to be cos effective you will arrive at same tradeoffs. Not saying it's not nice to be able to make some of those decisions on your own - but if you're picking LLMs for simplicity - we are years away from running your own being in the same league for most people.
At temperature zero, if you're using the same API/model, this really should not be the case. None of the big players update their APIs without some name / version change.
This isn't really true unfortunately -- mixture of experts routing seems to suffer from batch non-determinism. No one has stated publicly exactly why this is, but you can easily replicate the behavior yourself or find bug reports / discussion with a bit of searching. The outcome and observed behavior of the major closed-weight LLM APIs is that a temperature of zero no longer corresponds to deterministic greedy sampling.
Temperature changes the distribution that is sampled, not if a distribution is sampled.
Temperature changes the softmax equation[1], not weather or not you are sampling from the softmax result or choosing the highest probability. IBM's documentation corroborates this, saying you need to set do_sample to True in order for the temperature to have any effect, e.g., T changes how we sample, not if we sample [2].
A similar discussion on openai forum also claim that the RNG might be in a different state from run to run, although I am less sure about that [3]
I have dealt with traditional ML models in the past and things like tensorflow non-reproducibility. Managed to make them behave reproducibly. This is a very basic requirement. If we cannot even have that or people who deal with Gemini or similar models do not even know why they don't deliver reproducible results ... This seems very bad. It becomes outright unusable for anyone wanting to do research with reliable result. We already have a reproducibility crisis, because researchers often do not have the required knowledge to properly handle their tooling and would need a knowledgeable engineer to set it up. Only that most engineers don't know either and don't show enough attention to the detail to make reproducible software.
Your response is correct. However, you can choose to not sample from the distribution. You can have a rule to always choose the token with the highest probability generated by the softmax layer.
This approach should make the LLM deterministic regardless of the temperature chosen.
P.S. Choosing lower and lower temperatures will make the LLM more deterministic but it will never be totally deterministic because there will always be some probability in other tokens. Also it is not possible to use temperature as exactly 0 due to exp(1/T) blowup. Like I mentioned above, you could avoid fiddling with temperature and just decide to always choose token with highest probability for full determinism.
There are probably other more subtle things that might make the LLM non-deterministic from run to run though. It could be due to some non-deterministism in the GPU/CPU hardware. Floating point is very sensitive to ordering.
TL;DR for as much determinism as possible just choose token with highest probability (i.e. dont sample the distribution).
Here probably routing would be dominating, but in general, unless I missed all the vendors ditching GPUs and switching to ASICs optimized for fixed precision math, floating points are still non-commutative therefore results are non-deterministic wrt. randomness introduced by parallelising the calculations.
Of course which part of the calculations happens where should also be specifiable and be able to be made deterministicor should not have an effect on the result. A map reduce process' reduce step, merging results from various places also should be able to be made to give reproducible results, regardless of which results arrive first or from where.
Floating points are fundamentally too bad for this. We use them because they're fast, which usually more than compensates for inaccuracies FP math introduces.
(One, dealing with FP errors is mostly a fixed cost - there's a branch of CS/mathematics specializing in it, producing formally proven recipes for computing specific things in way that minimize or at least give specific bounds on errors. That's work that can be done once, and reused forever. Two, most programmers are oblivious to those issues anyway, and we've learned to live with the bugs :).)
When your parallel map-reduce is just doing matrix additions and multiplications, guaranteeing order of execution comes with serious overhead. For one, you need to have all partial results available together before reducing, so either the reduction step needs to have enough memory to store a copy of all the inputs, or it needs to block the units computing those inputs until all of them finish. Meanwhile, if you drop the order guarantee, then the reduction step just needs one fixed-size accumulator, and every parallel unit computing the inputs is free to go and do something else as soon as it's done.
So the price you pay for deterministic order is either a reduction of throughput or increase in on-chip memory, both of which end up translating to slower and more expensive hardware. The incentives strongly point towards not giving such guarantees if it can be avoided - keep in mind that GPUs have been designed for videogames (and graphics in general), and for this, floating point inaccuracies only matter when they become noticeable to the user.
Why would the same multithreaded software run on the same CPU (not just architecture - the same physical chip) have its instructions execute in different order from run to run? Performance. Want things deterministic? You have to explicitly keep them in sync yourself. GPUs sport tens of thousands of parallel processors these days, which are themselves complex, and are linked together with more complexity, both hardware and software. They're designed to calculate fast, not to ensure every subprocessor is always in lock step with every other one.
Model inference on GPU is mostly doing a lot of GPU equivalent of parallelized product on (X1, X2, X3, ... Xn), where each X is itself some matrix computed by a parallelized product of other matrices. Unless there's some explicit guarantee somewhere that the reduction step will pause until it gets all results so it can guarantee order, instead of reducing eagerly, each such step is a non-determinism transducer, turning undetermined execution order into floating point errors via commutation.
I'm not a GPU engineer so I don't know for sure, especially about the new cards designed for AI, but since reducing eagerly allows more memory-efficient design and improves throughput, and GPUs until recently were optimized for games (where FP accuracy doesn't matter that much), and I don't recall any vendor making determinism a marketing point recently, I don't believe GPUs suddenly started to guarantee determinism at expense of performance.
Why would the reduction step of a single neuron be split across multiple threads? That sounds slower and more complex than the naive method. And if you do decide to write code doing that, then just the code that reduces across multiple blocks needs to use integers, so pretty much no extra effort is needed.
Like, is there a nondeterministic-dot-product instruction baked into the GPU at a low level?
Not unless you control the underlying scheduler and force deterministic order; knowledge of all the code running isn't sufficient, as some factors affecting threading order are correlated with physical environment. For example, minute temperature gradient differences on the chip between two runs could affect how threads are allocated to CPU cores and order in which they finish.
> Why would the reduction step of a single neuron be split across multiple threads?
Doesn't have to, but can, depending on how many inputs it has. Being able to assume commutativity gives you a lot of flexibility in how you parallelize it, and allows you to minimize overhead (both in throughput and memory requirements).
> Like, is there a nondeterministic-dot-product instruction baked into the GPU at a low level?
No. There's just no dot-product instruction baked into GPU at low level that could handle vectors of arbitrary length. You need to write a loop, and that usually becomes some kind of parallel reduce.
> could affect how threads are allocated to CPU cores and order in which they finish
I'm very confused by how you're interpreting the word "each" here.
> Being able to assume commutativity gives you a lot of flexibility in how you parallelize it, and allows you to minimize overhead (both in throughput and memory requirements).
Splitting up a single neuron seems like something that would only increase overhead. Can you please explain how you get "a lot" of flexibility?
> You need to write a loop, and that usually becomes some kind of parallel reduce.
Processing a layer is a loop within a loop.
The outer loop is across neurons and needs to be parallel.
The inner loop processes every weight for a single neuron and making it parallel sounds like extra effort just to increase instruction count and mess up memory locality and make your numbers less consistent.
I feel like you're imagining a toy network with couple dozen neurons in few layers, done on a CPU. But consider a more typical case of dozens of layers with hundreds (or thousands) of neurons each. That's some thousand numbers to reduce per each neuron.
Then, remember that GPUs are built around thousands of tiny parallel processors, each able to process a bunch (e.g. 16) parallel threads, but then the threads have to run in larger batches (SIMD-like), and there's a complex memory management architecture built-in, over which you only have so much control. Specific numbers of cores, threads, buffer sizes, as well as access patterns, differ between GPU models, and for optimal performance, you have to break down your computation to maximize utilization. Or rather, have the runtime do it for you.
This ain't an an FPGA, you don't get to organize hardware to match your network. If you have a 1000 neurons per hidden layer, then individual neurons likely won't fit on a single CUDA core, so you will have to split them down the middle, at least if you're using full-float math. Speaking of, the precision of the numbers you use is another parameter that adds to the complexity.
On the one hand, you have a bunch of mostly-linear matrix algebra, where you can tune precision. On the other hand, you have a GPU-model-specific number of parallel processors (~thousands), that can fit only so much memory, can run some specific number of SIMD-like threads in parallel, and most of those numbers are powers of two (or a multiple of), so you have also alignment to take into account, on top of memory access patterns.
By default, your network will in no way align to any of that.
It shouldn't be hard to see that assuming commutativity gives you (or rather the CUDA compiler) much more flexibility to parallelize your calculations by splitting it whichever way it likes to maximize utilization.
I'm not imagining toy sizes. Quite the opposite. I'm saying that layers are so big that splitting per neuron already gives you a ton of individual calculations to schedule and that's plenty to get full usage out of the hardware.
You can do very wide calculations on a single neuron if you want; throwing an entire SM (64 or 128 CUDA cores) at a single neuron is trivial to do in a deterministic way. And if you have a calculation so big you benefit from splitting it across SMs, doing a deterministic sum at the end will use an unmeasurably small fraction of your runtime.
And I'll remind you that I wasn't even talking about determinism across architectures, just within an architecture, so go ahead and optimize your memory layouts and block sizes to your exact card.
I recently attended a STAC conference where they claimed the GPUs themselves are not deterministic. The hand-wavy speculation is they need to temperature control the cores and the flop ops may be reordered during that process. (By temperature I mean physical temperature, not some nn sampling parameter). On such large scale of computation these small differences can show up in the actually different tokens.
I can assure you this isn't true. Having worked with GPUs for many years in an application where consist results are important it's not only possible but actually quite easy to ensure consistent inputs produce consistent results. The temperature and clock speed do not affect the order of operations, only the speed, and this doesn't affect the results. This is the same as with any modern CPU which will also adjust clock for temperature.
The parent is suggesting that temperature only applies at the generation step, but the choice of backend “expert model” that a request is given to (and then performs the generation) is non-deterministic. Rather than being a single set of weights, there are a few different sets of weights that constitute the “expert” in MoE. I have no idea if that’s true, but that’s the assertion
I don't think it makes sense? Somewhere there has to be a RNG for that to be true. MOE itself doesn't introduce randomness, and the routing to experts is part of the model weights, not (I think) a separate model.
The samples your input is batched with on the provider's backend vary between calls and sparse mixture of experts routing when implemented for efficient utilization induces competition among tokens with either encouraged or enforced balance of expert usage among tokens in the same fixed-size group. I think it's unknown or at least undisclosed exactly why sequence non-determinism at zero temperature occurs in these proprietary implementations, but I think this is a good theory.
I thought the temperature only affects randomness at the end of the network (when turning embeddings back I to words using the softmax). It cannot influence routing, which is inherently influenced by which examples get batched together (ie, it might depend on other users of the system)
You don't need RNG since the whole transformer is an extremely large floating-point arithmetic unit. A wild guess - how about the source of non-determinism is coming from the fact that, on the HW level, tensor execution order is not guaranteed and therefore (T0 * T1) * T2 can produce slightly different results than T0 * (T1 * T2) due to rounding errors?
I have seen numbers come differently in JAX just depending on the batch size, simply because the compiler optimizes to a different sequence of operations on the hardware.
Wait isn't there atleast a two step process here one is semantic segmentation followed by a method like texttract for text - to avoid hallucinations?
One cannot possibly say that "Text extracted by a multimodal model cannot hallucinate"?
> accuracy was like 96% of that of the vendor and price was significantly cheaper.
I would like to know how this 96% was tested. If you use a human to do random sample based testing, well how do you adjust the random sample for variations in distribution of errors that vary like a small set of documents could have 90% of the errors and yet they are only 1% of the docs?
One thing people always forget about traditional OCR providers (azure, tesseract, aws textract, etc.) is that they're ~85% accurate.
They are all probabilistic. You literally get back characters + confidence intervals. So when textract gives you back incorrect characters, is that a hallucination?
The hallucination in LLM extraction is much more subtle as it will rewrite full sentences sometimes. It is much harder to spot when reading the document and sounds very plausible.
We're currently working on a version where we send the document to two different LLMs, and use a 3rd if they don't match to increase confidence. That way you have the option of trading compute and cost for accuracy.
>We're currently working on a version where we send the document to two different LLMs, and use a 3rd if they don't match to increase confidence.
I’m interested to hear more about the validation process here.
In my limited experience, I’ve sent the same “document” to multiple LLMs and gotten differing results. But sometimes the “right” answer was in the minority of responses. But over a large sample (same general intent of document, but very different possible formats of the information within), there was no definitive winner. We’re still working on this.
What if you use a different prompt to check the result, did this work? I am thinking to use this approach, but now I think maybe it is better to use two different LLM like you do.
It’s a question of scale. When a traditional OCR system makes an error, it’s confined to a relatively small part of the overall text. (Think of “Plastics” becoming “PIastics”.) When a LLM hallucinates, there is no limit to how much text can be made up. Entire sentences can be rewritten because the model thinks they’re more plausible than the sentences that were actually printed. And because the bias is always toward plausibility, it’s an especially insidious problem.
It's a bit of a pick your poison situation. You're right that traditional OCR mistakes are usually easy to catch (except when you get $30.28 vs $80.23). Compared to LLM hallucinations that are always plausibly correct.
But on the flip side, layout is often times the biggest determinant of accuracy, and that's something LLMs do a way better job on. It doesn't matter if you have 100% accurate text from a table, but all that text is balled into one big paragraph.
Also the "pick the most plausible" approach is a blessing and a curse. A good example is the handwritten form here [1]. GPT 4o gets the all the email addresses correct because it can reasonably guess these people are all from the same company. Whereas AWS treats them all independently and returns three different emails.
The difference is the kind of hallucinations you get.
Traditional OCR is more likely to skip characters, or replace them with similar -looking ones, so you often get AL or A1 instead of AI for example. In other words, traditional spelling mistakes.
LLMs can do anything from hallucinating new paragraphs to slightly changing the meaning of a sentence. The text is still grammatically correct, it makes sense in the context, except that it's not what the document actually said.
I once gave it a hand-written list of words and their definitions and asked it to turn that into flashcards (a json array with "word" and "definition"). Traditional OCR struggled with this text, the results were extremely low-quality, badly formatted but still somewhat understandable. The few LLMs I've tried either straight up refused to do it, or gave me the correct list of words, but entirely hallucinated the definitions.
> You literally get back characters + confidence intervals.
Oh god, I wish speech to text engines would colour code the whole thing like a heat map to focus your attention to review where it may have over-enthusiastically guessed at what was said.
We did this for a speech to text solution in healthcare. Doctors would always review everything that was transcribed manually (you don’t want hallucinations in your prescription), and using a heatmap it was trivial to identify e.g. drugs that were pretty much always misunderstood by STT
I know nothing about OCR providers. It seems like OCR failure would result in gibberish or awkward wording that might be easy to spot. Doesn't the LLM failure mode assert made up truths eloquently that are more difficult to spot?
Speaking from experience, you need to double check "I" and "l" and "1" "0" and "O" all the time, accuracy seems to depend on the font and some other factors.
have a util script I use locally to copy some token values out of screenshots from a VMWare client (long story) and I have to manually adjust 9/times.
How relevant that is or isn't depends on the use case.
For an OCR company I imagine it is unconscionable to do this because if you would say OCR for an Oral History project for a library and you made hallucination errors, well you've replaced facts with fiction. Rewriting history? What the actual F.
Probaly totally fine for a "fintech" (Crypto?) though. Most of them are just burning VC money anyway. Maybe a lucky customer gets a windfall because Gemini added some zeros.
I think you can just ask DeepSeek to create a coin for you at this point, and with the recent elimination of any oversight, you can automate your rug pulls...
Normal OCR (like Tesseract) can be wrong as well (and IMO this happens frequently). It won’t hallucinate/straight make shit up like an LLM, but a human needs to review OCR results if the workload requires accuracy. Even across multiple runs of the same image an OCR can give different results (in some scenarios). No OCR system is perfectly accurate, they all use some kind of machine learning/floating point/potentially nondeterministic tech.
Wouldn’t the temperature on something like OCR be very low. You want the same result every time. Isn’t some part of hallucination the randomness of temperature?
The LLM's are near perfect (maybe parsing I instead of 1) - if you're using the outputs in the context of RAG, your errors are likely much much higher in the other parts of your system. Spending a ton of time and money chasing 9's when 99% of your system's errors have totally different root causes seems like a bad use of time (unless they're not).
This sounds extremely like my old tax accounting job. OCR existed and "worked" but it was faster to just enter the numbers manually than fix all the errors.
Also, the real solution to the problem should have been for the IRS to just pre-fill tax returns with all the accounting data that they obviously already have. But that would require the government to care.
It'd be nicer if you wouldn't presume to know the reasons people might believe erroneous information.
In this case, the reason for the misinformation is do to the lack of communication from the DOGE entity regarding their actions. Mr. Musk wrote via Tweet that he had "deleted" the digital services agency "18F" that develops the IRS Free File program and also deleted their X account.
If Gemini can do semantic chunking at the same time as extraction, all for so cheap and with nearly perfect accuracy, and without brittle prompting incantation magic, this is huge.
I don't know exactly how or what it's doing behind the scenes, but I've been massively impressed with the results Gemini's Deep Research mode has generated, including both traditional LLM freeform & bulleted output, but also tabular data that had to come from somewhere. I haven't tried cross-checking for accuracy but the reports do come with linked sources; my current estimation is that they're at least as good as a typical analyst at a consulting firm would create as a first draft.
If I used Gemini 2.0 for extraction and chunking to feed into a RAG that I maintain on my local network, then what sort of locally-hosted LLM would I need to gain meaningful insights from my knowledge base? Would a 13B parameter model be sufficient?
Yes. For example, to create AI agent 'assistants' that can leverage a local RAG in order to assist with specialist content creation or operational activities.
It loads the entire PDF into context, but then it would be my job to chunk the output for RAG, and just doing arbitrary fixed-size blocks, or breaking on sentences or paragraphs is not ideal.
So I can ask Gemini to return chunks of variable size, where each chunk is a one complete idea or concept, without arbitrarily chopping a logical semantic segment into multiple chunks.
Fixed size chunks is holding back a bunch of RAG projects on my backlog. Will be extremely pleased if this semantic chunking solves the issue. Currently we're getting around an 78-82% success on fixed size chunked RAG which is far too low. Users assume zero results on a RAG search equates to zero results in the source data.
Agree, BM25 honestly does an amazing job on its own sometimes, especially if content is technical.
We use it in combination with semantic but sometimes turn off the semantic part to see what happens and are surprised with the robustness of the results.
This would work less well for cross-language or less technical content, however. It's great for acronyms, company or industry specific terms, project names, people, technical phrases, and so on.
I wish we had a local model for semantic chunking. I've been wanting one for ages, but haven't had the time to make a dataset and finetune that task =/.
This is great, I just want to highlight out how nuts it is that we have spun up whole industries around extracting text that was typically printed from a computer, back into a computer.
There should be laws that mandates that financial information be provided in a sensible format: even Office Open XML would be better than this insanity. Then we can redirect all this wasted effort into digging ditches and filling them back in again.
I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings.
>>I've been fighting trying to chunk SEC filings properly, specifically surrounding the strange and inconsistent tabular formats present in company filings.
For this specific use case you can also try edgartools[1] which is a library that was relatively recently released that ingests SEC submissions and filings. They don't use OCR but (from what I can tell) directly parse the XBRL documents submitted by companies and stored in EDGAR, if they exist.
XBRL is what I'm using currently, but it's still kind of a mess (maybe I'm just bad at it) for some of the non-standard information that isn't properly tagged.
How do today’s LLM’s like Gemini compare with the Document Understanding services google/aws/azure have offered for a few years, particularly when dealing with known forms? I think Google’s is Document AI.
I've found the highest accuracy solution is to OCR with one of the dedicated models then feed that text and the original image into an LLM with a prompt like:
How does it behave if the body of text is offensive or what if it is talking about a recipe to purify UF-6 gas at home? Will it stop doing what it is doing and enter lecturing mode?
I am asking not to be cynical but because of my limited experience with using LLMs for any task that may operate on offensive or unknown input seems to get triggered by all sorts of unpredictable moral judgements and dragged into generating not the output I wanted, at all.
If I am asking this black box to give me a JSON output containing keywords for a certain text, if it happens to be offensive, it refuses to do that.
We use the Azure models and there isn't an issue with safety filters as such for enterprise customers. The one time we had an issue microsoft changed the safety measures. Of course the safety measures we might meet are the sort of engineering which could be interpreted as weapons manufacturing, and not "political" as such. Basically the safety guard rails seem to be added on top of all these models, which means they can also be removed without impacting the model. I could be wrong on that, but it seems that way.
This is for anyone coming across this link later. In their latest SDKs, if you want to completely switch off their safety settings, the flag to use is 'OFF' and not 'BLOCK_NONE' as mentioned in the docs in the link above.
member of the gemini team here -- personally, i'd recommend directly using gemini vs the document understanding services for OCR & general docs understanding tasks. From our internal evals gemini is now stronger than these solutions and is only going to get much better (higher precision, lower hallucination rates) from here.
GCP's Document AI service is now literally just a UI layer specific to document parsing use-cases back by Gemini models. When we realized that we dumped it and just use Gemini directly.
They will, and they'll still have a solid product to sell, because their value proposition isn't accurate OCR per se, but putting an SLA on it.
Reaching reliability with LLM OCR might involve some combination of multiple LLMs (and keeping track of how they change), perhaps mixed with old-school algorithms, and random sample reviews by humans. They can tune this pipeline however they need at their leisure to eke out extra accuracy, and then put written guarantees on top, and still be cheaper for you long-term.
With “Next generation, extremely sophisticated AI” to be precise, I wait say. ;)
Marketing joke aside, maybe a hybrid approach could serve the vendor well. Best of both worlds if it reaps benefits or even have a look at hugging face for even more specialized aka better LLMs.
I'm curious to hear about your experience with this. Which solution were you using before (the one that took 12 minutes)? If it was a self-hosted solution, what hardware were you using? How does Gemini handle PDFs with an unknown schema, and how does it compare to other general PDF parsing tools like Amazon Textract or Azure Document Intelligence? In my initial test, tables and checkboxes weren't well recognized.
> For the 4% inaccuracies a lot of them are things like the text "LLC" handwritten would get OCR'd as "IIC" which I would say is somewhat "fair".
I'm actually somewhat surprised Gemini didn't guess from context that LLC is much more likely?
I guess the OCR subsystem is intentionally conservative? (Though I'm sure you could do a second step on your end, take the output from the conservative OCR pass, and sent it through Gemini and ask it to flag potential OCR problems? I bet that would flag most of them with very few false positives and false negatives.)
but were often written with typewriters long ago to get nice structured tabular output. Deals with text being split across lines and across pages just fine.
It is cheaper now, but I wonder if it will continue to be cheaper when companies like Google and OpenAI decide they want to make a profit off of AI, instead of pouring billions of dollars of investment funds into it. By the time that happens, many of the specialized service providers will be out of business and Google will be free to jack up the price.
I use Claude through OpenRouter (with Aider), and was pretty amazed to see that it routes the requests during the same session almost round-robin through Amazon Bedrock, sometimes through Google Vertex, sometimes through Anthropic themselves, all of course using the same underlying model.
Literally whoever has the cheapest compute.
With the speed that AI models are improving these days, it seems like the 'moat' of a better model is only a few months before it is commoditized and goes to the cheapest provider.
I’ve been wanting to build a system that ingests pdf reports that reference other types of data like images, csv, etc. that can also be ingested to ultimately build an analytics database from the stack of unsorted data AB’s meta data but I have not found any time to do anything like that yet. What kind of tooling do you use to build your data pipelines?
It's great to hear it's this good, and it makes sense since Google has had several years of experience creating document-type-specific OCR extractors as components of their Document AI product in Cloud. What most heartening is to hear that the legwork they did for that set of solutions has made it into Gemini for consumers (and businesses).
Successful document processing vendors to use LLMs already. I know this at least of klippa. They have (apparently) fine-tuned models, prompts etc. The biggest issue with using LLMs directly is error handling, validation and "parameter drift"/randomness. This is the typical "I'll build it myself but worse" thing
I'm interested to hear what your experience has been dealing with optional data. For example if the input pdf has fields which are sometimes not populated or nonexistent, is Gemini smart enough to leave those fields blank in the output schema? Usually the LLM tries to please you and makes up values here.
You could ingest them with AWS Textract and have predictability and formatting in the format of your choice. Using LLMs for this is lazy and generates unpredictable and non-deterministic results.
Did you try other vision models such as ChatGPT and Grok? I'm doing something similar but struggled to find good comparisons in between the vision models in terms OCR and document understanding.
If the documents have the same format, maybe you could include an example document in the prompt, so the boilerplate stuff (like LLC) gets handled properly.
> Our prompt is currently very simple: "OCR this PDF into this format as specified by this json schema" and didn't require some fancy "prompt engineering" to contort out a result.
reply