Recently I worked on a contract doing some data ingestion that the resident team didn't want to deal with.
They had been provided a JSON api to get what should have been a batch file. The owner/api creator said "this is good enough we won't accommodate you." So much for sensible data transmission and batch processing.
Because I have had to deal with these sorts of things before I have a fairly robust tool chain to hammer api's with requests to get the data in a reasonable time frame. In this case my client went full tilt - and simply hammered the API till its owner gave in and started sending batch files.
Yes the end result in most cases is good old CSV -
repeated variable names are bloat when getting lots of data, extra formatting is the same... CSV's are great for getting these things down.
As another poster pointed out (S)FTP is the way to go when sending CSV's, and they are compressed in most cases to save storage and data transmission on both ends.
I am old enough to remember when mailing a hard disc or a tape might be a faster way to move a LOT Of data. The practice is still alive and well, but the option is now by the "truckload"
Probably means a single report with all the data sent at a regular interval. (All transactions between 00:00 hours and 24:00 hours the previous day, sent by 9:00 am to these email inboxes).
Instead the client was given an api to query and no batch processing of information was allowed.
I’ve encountered this idea somewhere, and fortunately have never had to deal with such a broken scenario. Getting piecemeal information is infuriating.
He probably means the output of a batch processing pipeline. Clients will calculate a set of data once a day and shove it into some kind of storage[0] for you to download.
[0] probably an FTP server with no TLS and with the same weak username and password for everyone who connects, a setup which they won't change no matter how many times you ask
Recently I worked on a contract doing some data ingestion that the resident team didn't want to deal with.
They had been provided a JSON api to get what should have been a batch file. The owner/api creator said "this is good enough we won't accommodate you." So much for sensible data transmission and batch processing.
Because I have had to deal with these sorts of things before I have a fairly robust tool chain to hammer api's with requests to get the data in a reasonable time frame. In this case my client went full tilt - and simply hammered the API till its owner gave in and started sending batch files.