Very cool and well executed project. Love the sprinkle of Rust in all the other companion projects as well :)
The ROAPI(https://github.com/roapi/roapi) project I built also happened to support a similar feature set, i.e. to expose sqlite through a variety of remote query interfaces including pg wire protocols, rest apis and graphqls.
Datafusion out performs spark by a large margin. It is on par with photon based on my experiences, see benchmarks at https://github.com/blaze-init/blaze.
Ah nice, thank you for sharing that. I hadn’t seen it before, and congrats on beating out Spark that hard, I hope it continues to improve!
As an aside, maybe it would make sense to publish a new blog post somewhere so that the top hit on Google for “DataFusion benchmark” isn’t that post I linked.
Haha, yeah, we should definitely put a little bit more efforts into SEO :) Everyone is so focused on the hard-core engineering at the moment. I think Matthew from the community is actually working on a new comprehensive benchmark for us at the moment, which I hope will be published soon.
Yes, I designed the code base so that the core of the IO and query logic are abstracted into a Rust library called columnq. My plan is to wrap it with pyo3 so the full API can be accessed as a Python package! If you are interested in helping with this, please feel free to submit a PR. The core library is located at https://github.com/roapi/roapi/tree/main/columnq
We cache all the data in memory in Arrow format so queries don't need to go through google api, it will only hit google api when a data refresh is needed.
It's manual for now, ROAPI is designed for slowly moving datasets. You just hit the data update API to force a refresh.
I have plan to add automated streaming data update in the background, starting with delta lake tables. It should all be very straight forward to implement.
We are hiring new grads across the board if you have proofs for exceptional skills. Feel free to take a look at all our engineering postings at: https://neuralink.com/careers.
There are more companies noticing this now and have stopped asking these questions. For example, we at Neuralink[1] only give out practical programming challenges. If are you good at building practical systems, you should be able to ace our coding interviews without any preparation. No leetcode and no whiteboarding. In fact, I prefer to hire those who doesn't waste time practicing leetcode.
Over the past couple years, I have interviewed at a handful of other startups who also have similar coding interview philosophies.
Hey! I run a job board at NoWhiteboard.org for companies not using whiteboard/LeetCode interviews in their interview process. I can add Neuralink’s roles to the website for free!
If you’re open to it, could you share the high level overview of the interview process?
You can also save your drawings using the option that embeds the drawio diagram source into the rendered svg or png file, and check that single file into git.
Then (a) no need for an automation to generate the image; (b) any image or svg is directly editable.
This is possibly one of Diagrams's killer features, especially in the days of proprietary cloud document hosting that (poorly) attempts to hide the "file" abstraction. You can plunk your diagram directly into the doc without worrying if you'll be able to keep track of the source if you need to edit it later.
Do an 'Open from' > 'URL...' and paste in http://shagie.net/wp-content/uploads/2015/05/Untitled-Diagra... (an example diagram I made several years ago) - its a png itself and you can open that up as a png, but I also exported it with the drawing embedded.
So when you open that up in the app, you'll get the drawing - not the raster version.
Also consider that I created that png 7 years ago - other drawing document formats have been less stable over that timeframe. I'm quite impressed with it.
The ROAPI(https://github.com/roapi/roapi) project I built also happened to support a similar feature set, i.e. to expose sqlite through a variety of remote query interfaces including pg wire protocols, rest apis and graphqls.