From their pricing example:

> You schedule a job to run once an hour for 15 seconds. This job uses 50% of one CPU core and 256 MiB of memory. This job will cost $0.016/day for the CPU, and $0.001/day for the memory, adding up to $0.017/day, or $0.51/month.

While you could do FaaS (e.g. AWS Lambda) + another service for state (e.g. S3/RDS/EFS), that seems like overkill.

Even today I would still do it the old way: store state in a filesystem (maybe SQLite if you have more complex needs), and configure a plain old cronjob on any of the servers you have access to.

Maybe put it in docker/dockerhub to make it easier to run.

Surprisingly it is an interesting question. You have a very simple task, but ideally you want the solution to include: running environment (let's say a container), scheduler, persistent state, monitoring (you want to know when it is down). Not to mention deployment.

There are solutions for each of those, but given the simplicity it would be nice to have a lightweight solution that includes all of that with minimal configuration. I doubt it exists.

I.e. you can do all of that on AWS, but I can't help but wonder if the infrastructure setup is going to be more complicated than your actual scraper.

Edit: a good way to think about it is: imagine that a person just learned how to scrape websites, they wrote a script that works locally. How much do they need to learn to move it to the cloud?

If your answer is: ok, so you just need to create an AWS account, dockerize your script, put it in a Lambda, create RDS instance, trigger your Lambda through EventBridge and setup CloudWatch... then there's something wrong with you. There must be a better way to reliably run a fucking 10 line script in the cloud.

There's been many times over the years when I wanted to write a simple scraper that does exactly what author describes and what stopped me is the amount of infrastructure bullshit I would have to deal with.

I mean, you don’t have to dockerize it, and I would use S3 or Dynamo for state, but otherwise it’s a pretty simple solution and it can run entirely in the free tier on AWS. It’s way better than managing your own host, especially when you consider that you get monitoring, alerting, logging, etc for free. EventBridge is just the standard AWS thing for triggering stuff (in this case, a cron schedule)—don’t let the branding scare you.

I think the cheapest "cheaty" way to store data for free would be to pollute tracking sites. Open an account at some company, change your mailing address every hour, etc.

https://docs.aws.amazon.com/glue/latest/dg/monitor-data-ware...

https://fly.io/docs/machines/working-with-machines/#create-a...

See the `schedule` parameter.

This might be a non-answer, but if you just want "scheduled notifications of a website change", then you can use something completely purpose built for that. There is a Github action that does exactly this: https://github.com/yasoob/github-action-scraper-tutorial/blo...

This might also be a non-answer, but I've found myself using Integromat/Make or Zapier more and more for this type of work. It allows me to whip things up quickly and it can actually be approachable by non-technical people.

One thing I love about supabase is that they are focused. They have a ton of features, but they're cohesive in ways no one else has managed. You can manage your document storage, their built-in auth provider, and websocket updates ALL IN SQL.

And now their QETL and remote servers?? (https://supabase.com/blog/postgres-foreign-data-wrappers-rus...) And their CDN and Image processing???

It is shocking and almost painful how beautiful this all is. And how with those basics, you could almost do anything. But yeah, I think the only thing between Supabase and Web Scraping Screenshots is a Postgres Plugin, entirely independent of what Supabase provides.

If I were one of those beautiful people who write a blog about rasterizing on a e-pregnancy tests, I'd give it a shot haha

I do a lot of web scraping, and Lambdas are very useful for web scraping since their IPs can change for each request.

For deployment, you can just do `serverless deploy` or set up a GitHub action each time you push to main.

You could use lambda if your runtime is expected to always be less than 15 minutes Batch fargate jobs documentation - https://docs.aws.amazon.com/batch/latest/userguide/fargate.h... Cloud watch cron documentation

https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/S...

https://aws.amazon.com/about-aws/whats-new/2018/03/aws-batch...

Everything can be deployed as a simple cdk app once you have your code ready in a docker repository somewhere . There are no servers to maintain and no ongoing maintenance will be necessary

I use Zappa and it's great.

I remember seeing a couple projects shared before, using this technique to scrape sites with GHA

This works perfectly for my case (content syndication for https://potato.horse), and I’m pretty happy with GH actions for this kind of stuff, but if you need something more precise, you might want to look somewhere else.

Is the spread really that egregious? That's essentially a 50% failure to trigger at all, like I don't think you can call that 15 minutes so much as 15±10 minutes lmfao

IIRC AWS EventBridge is also not guaranteed to execute on the exact minute, but in my experience running a small job every 5 minutes only had about a 30-40 second delay at worst.

Simon Willison has a bunch of examples of scraping sites with GHA and storing the results in a repo. But you can use the same technique without the storing part if need be.

This seems to meet OP’s use case as they have to commit state anyway.

[0] https://docs.github.com/en/actions/managing-workflow-runs/di...

Can't you configure a GHA which commits nonsense every month?

https://github.com/patrickgalbraith/rageagain/blob/master/.g...

I use their cronjob functionality to ensure my docker images are built daily and therefore in theory secure.

I wrote a notifier for when a local business has open appointments. A quick fetch every day and connected it to SNS seamlessly. Now I can easily add family members if they want the notifications and they can easily unsubscribe right from the email. All within the free tier.

I tried to do Chromium/Puppeteer based scraping this way.

Building a Dockerfile took ages due to the low compute. (Rust was a non-starter).

I also had (foolishly) only bought the Pi with 2GB instead of 8GB so RAM was an issue.

Disk was super slow.

I'm not sure how viable this is, especially with how hard it is currently to source a Pi, let alone its computation/memory constraints.

Why compile anything on the raspberry pi? Cross-compile on a machine with more compute (like your laptop, desktop, phone, or ec2 instance) one time, and then transfer the compiled binaries or built docker image over.

> Pi with 2GB instead of 8GB

For headless chrome, that should be enough unless you're doing other stuff with it. Unless you mean for compiling stuff, which as before can be done elsewhere.

First of all the answer recommends to use either a Pi or a VPS. If a Pi doesn't cut it for you, just switch to a VPS with sufficient specs for your requirement. Problem solved, now it's viable.

Besides, a significant part of the web can be scraped without resorting to a heavyweight browser such as Chromium. It should always be the last resort. Even if you have to evaluate Javascript (in case of SPAs), there are much cheaper solutions than Puppeteer (JSDOM being an example) which can get the job done most of the time.

As to Docker, I fail to see why you would need Docker for this kind of job, unless you don't know how to do it without Docker.

When there is no portability requirement, the costs of Docker easily surpass its benefits.

...

So no, it's not that what's being recommended in the answer is not viable. You're doing it wrong. Either you're using a Pi when you need to use a VPS or you are introducing unnecessary layers.

I have a Raspberry Pi that is natively running a scraper using headless Chromium and cron. It works great, except....

I ended up needing a virtual framebuffer. I got it working on the Raspberry Pi, but I got a new workstation and wanted to edit my script and test it there. I got cryptic errors that I needed to debug to understand they were framebuffer issues, then attempt to recreate the setup that's running on my Pi, then debug that.....

My first mistake was not writing down what I did in my README, but a Docker image would have saved me a ton of time here.

Isn't proposing Docker as a solution to this going nuclear?

I think there are some many cheaper things you can do to solve this.

For me this is absolutely no justification for Docker use in this scenario.

A lot of websites can detect the IPs from this and block it, basically almostly like a CAPTCHA.

I had other needs like Postgres, etc.

Scraping data is one thing, actually doing anything with it is another. I quickly hit the limits of a $50 RaspberryPi 4 or whatever they're going for on Amazon these days with the gouging, etc.

- (free) set up google sheets (not sure if this is required but that's what I do)

- (free) run an scheduled appscript bound to that sheet with regular intervals

- make it scrape the page (it's just javascript, you can do get requests, etc)

- Have it scrape a webarchive of the page you want to compare (not need to persist state or run a database, etc) and have it check the differences with the current page

- Have it send you an email if there's changes

It's totally free, takes no time to set up, and is relatively effortless. It can even send you alert on failures etc. I do something like this for my reading tracker where I crawl various sites (like amazon, etc) and RSS feeds for new releases of books/series I read and collect them in a dynamic spreadsheet.

I think it's a great starting point for students because it's totally free, abstracts over the coplications of oauth (for small things I still prefer it because it's so effortless) and you can combine google apps in fun ways. Considering many students live out of gmail + google docs, it's great

And having a GUI version of the app makes interacting with other students easy bc they can always edit the google sheet (aka database), forms, gmail, etc

EDIT: after reading the other solutions in this thread I'm blown away by how much people are over thinking and over engineering this. KISS people.

Your solutions costs $240/year.

There are solutions out there that would probably bring that under $20/year, and be more reliable than your VM that falls over when it's full of logs (at some indeterminate future date).

10x+ savings with greater reliability isn't "overthinking it", it's a greatly improved solution.

If this is an issue, I think all other solutions have similar risks: account banned, company acquired/closed down, forced update to new API/system, to a point that I think I'd take the bet on a VM living longer in most cases

https://cloud.google.com/free

I have a VM from those lowendbox providers from a few years ago that costs $4 a YEAR - static IP, 256MB RAM/512MB virtual, 10GB storage, U18.04

I do not use it for anything serious, but it actually has had 99.9% uptime.

Again, with cheap providers there is a good chance they go belly up but for cronjob that should suffice.

For more serious cronjobs on the budget Hetzner is nice.

Many of the other options listed in this thread are free or within a rounding error of free. Some of them equally are not the "cheapest, easiest" because they're complex. But some of them pretty clearly meet both criteria (GitHub Actions comes to mind), and some of them might meet the criteria depending on what somebody has familiarity with (if you're already used to deploying code to any FaaS provider with a free tier, that'll likely be the cheapest/easiest way).

If you have a VPS, I agree, adding a cronjob to it is both cheap and easy. But it's silly to assume that anybody who is a geek has a VPS already.

However, to your point on KISS, it's not really about that. Leaving a server on 24/7 is pretty cost ineffective if all you want to do is run a short job once a day.

These are just the "reputable" places, there's tons of small players where you can get a tiny VM for $10 for a year.

And a few funsies "droplets" for that much that host my resume and other low volume, but open to internet, sites.

They all run cron.

Works great.

Send an email (for free): https://blog.cloudflare.com/sending-email-from-workers-with-...

oh!! have you also thought about the right queue semantics? Do you want your job to trigger exactly one, at least one or at most once? What happens if the previous job is still running when it is time to trigger it again? Is it 15 minutes from start to start or end of previous to start of next? How about metrics? Is your process idempotent?

So, the right answer is 42 unicorns.

It's the future (2015) https://circleci.com/blog/its-the-future/

You probably want to setup a self-hosted cloud-based remotely-managed splunk and ensure someone does the maintenance/upgrade for you (those can be challenging)

Have you properly setup your pager-duty ? I wouldn't want to miss one of the cron execution !!

Cloud Run will run any docker image, and give it a HTTPS URL. Scales to zero, allows running in the background for up to an hour.

The benefit over FAAS is that you can run anything you want in the container, including multiple processes.

Costs around 1 Euro a month, I had a similar setup for free in Heroku before.

Here's how a cron job implementation would look with Temporal -

` func Subscription(ctx [redacted].Context, userUUID string) error { free_trial_on := true

  for {
    if free_trial_on {
      // Sleep until trial period passes
      [redacted].Sleep(ctx, days(15))
      free_trial_on = False
    } else {
      // Charge subscription fee
      [redacted].ExecuteActivity(ctx, ChargeSubscriptionAndEmailReceipt, userUUID, frequency).Get(ctx, nil)
      // Sleep until next payment is due
      [redacted].Sleep(ctx, days(30))
    }
  }

They are two very different concepts which I see very often used interchangeably. Temporal is clearly about orchestration.

Alternatively, for a more generic solution to “easy free cronjobs”, you can write some php page on a free php hosting service and point visualping to that page. Cost: 0/year

https://www.netlify.com/blog/how-to-schedule-deploys-with-ne...

https://docs.netlify.com/cli/get-started/#get-started-with-n...

Might be more precise to call it "GitHub scraping" as you set up your script to run on a schedule on GitHub Actions CI and keep its state by committing into the git repo.

We have our scripts running on GH Actions end by hitting the Slack API to notify us with a message.

There's a hosted solution, or you can self-host it

Does exactly what it says and gets bonus points due to self-hosted or as-a-service.

Cheapest: Any CI/CD service provider. Takes 20 minutes to set up. Not very reliable. Free

Next cheapest, least easy: complex proprietary serverless cloud functions to automatically trigger a task on a schedule. Takes an hour to set up. Most reliable. $0.30-$1.5/month

  # Hosted Cron Jobs
  -------------------------------------------------------------
  | Solution          | Difficulty | Reliability | Cost       |
  -------------------------------------------------------------
  | Linux VM w/Cron   | Easiest    | Moderate    | $1-$5      |
  | CI/CD Provider    | Moderate   | Low         | $0         |
  | Serverless Cloud  | Hardest    | High        | $0.30-$1.5 |
  -------------------------------------------------------------

Cheap: low in price, especially in relation to similar items or services.

Easy: achieved without great effort; presenting few difficulties.

Because "easiest" AND "cheapest" are logically conflicting, my answer only applies to "easy" ways of hosting cron jobs. My answer also holds true for background tasks that are not always scheduled.

The two tools I rely on are:

1. Pipedream

2. Windmill (open-source pipedream)

Both of them let me define functions that need to be run on schedule. Also functions that can react to real world events (webhooks). I can code in Go/ Js/ Ts or Python. Everything is version controlled, stored in Git.

For Pipedream, you pay a premium for managed service. Windmill can be hosted on a $5 machine.

i'm sure the major cloud providers have cheap / free tiers for this kind of work, but quite frankly i've been burned by run away pricing tiers too many times to ever consider using cloud again for a personal project .. so unless this is getting funded by a client try to host your own.

[1] https://learn.microsoft.com/en-us/azure/azure-functions/func...

The UI is excellent for this. You can probably find cheaper on Fly but probably not easier.

Even easier, maybe more expensive than Render:

https://www.zeplo.io/docs/schedule

You just hit a URL like so and it's done. zeplo.to/your_url.com?_cron=5|14|||*

We have companies running hundreds of concurrent schedules on our platform. Send me a note at ravi@airplane.dev if you'd like to chat about it.

* free-tier vps on GCP or Oracle Cloud

* lambda job on AWS

I have a cheap VPS I use for other things and just run my cron jobs there.

Flexibility to write custom code and alerting logic, but no headache of managing your own infrastructure

Disclaimer: I previously worked at Retool :)

My use cases have been: - Web Scraping of some websites to check stock availability of a product (4 times a day). - Web Scraping of a website to get an apppointment at a specific citizen office (every 5 seconds). If success then send a Telegram Chat Bot message to me.

The second use case was a bit tricky, because my IP was kicked frequently from the server, I tried out what the sweet spot timing for the ping was and my setup (all jobs with different IPs) was the following:

1. setup on my Raspberry Pi in combination of the package PM2 (PM2 is a daemon process manager that will help you manage and keep your application online 24/7)

2. setup on Render (https://render.com/) Advantage here was the very convinient, quick and easy setup. I was able to link my Github Repository to it and the Cron Job would re-build the program automatically whenever a new master version on GitHub was available. I used an in-memory TypeScript job scheduler library in my project that repeatedly executes given tasks within specified intervals of time (e. g. "each 5 seconds") until 7 PM and I let the scron job service execute the script again at 7 AM in the morning. Debug Print Console inclusive.

3. setup on EvenNode (https://www.evennode.com/docs/cron-jobs) Here the upload of my node.js script was done via FileZilla to the FTP-Server of EvenNode. Linkage to GitHub Repository is also possible but since I am not that familiar with the private key setup at GitHub I chose the FTP-Server option. I used an in-memory TypeScript job scheduler library in my project that repeatedly executes given tasks within specified intervals of time (e. g. "each 5 seconds") until 7 PM and I let the scron job service execute the script again at 7 AM in the morning. Debug Print Console inclusive.

Both online Service were very cheap, paid cents only. The Raspberry Pi solution was for free (only energy consumption). I can highly recommend render.com and evennode.com for cron jobs.

https://uptimerobot.com/

GitHub actions are another option

Not free but basically you wrap your job in http endpoint and then we take care of the rest.

there it is an apt-get away

Thanks!

There are two ways to set up a cron job: the dashboard or via the API if you need the flexibility, e.g., dynamically creating/starting/stopping crons.

We use HTTP webhooks to make the outgoing cron requests, which makes it easy to integrate with a wide range of services and platforms, including Vercel, Lambda, etc. We're also working on adding local execution so you can have even more control over how your tasks are run without using webhooks, but that's still being built & tested.

Feel free to reply here or email me directly if you have any questions: james@mergent.co

https://developers.google.com/apps-script

It's free, so pretty cheap.

You can set up a schedule to run the scripts. Has easy access to Google APIs (Gmail).

Very powerful and simple solution I've used for years.

It's cheap as in free, thanks to the generous free tier.

How cheap can you run this cron job, the basics:

- 8,765 invocations a year

- Lambda: $0.20 per 1M requests

- Cloudwatch Events: $1.00 per 1M requests

$1.2 / 1,000,000 * 8,765 = About 1 cent

Is anybody going to host that as a service for this price? Absolutely not.

I want to offload this, but any time I search for an option, they're overcomplicated or overpriced. Maybe there just isn't a market for it?

...maybe I don't want to offload this. The Pi and SD card was like $35, the power is basically zero, and the Internet is basically zero.

Priced: $0.25/GB-hour + data transfer and storage

https://beew.io/

You can create cron jobs over HTTP, both recurring and one-time:

Node SDK: https://www.npmjs.com/package/beew API Reference: https://beew.io/api

You can also create directly trough the UI.

Let me know if you need any help

I'm actively using all of the above approaches.

Better, forget about running 24/7 and use rtcwake(8) to wake it up for an hour at night to perform the job and then go back to sleep. This will bring costs far into the negligible.

It's worth checking out Shipyard (https://www.shipyardapp.com). I'm the co-founder and designed it to be the easiest way to deploy scripts/workflows in the cloud. You can schedule any code to run our platform (native support for Python, Node.js, Bash) and build reusable templates.

Our free dev plan allows for 10 hours of free runtime per month which is plenty for most use cases. If you want more or need webhooks/API, that starts at $50/month.

Feel free to contact if you want to learn more. Email is in my profile.

https://chrome.google.com/webstore/detail/distill-web-monito...

It's free.

https://docs.netlify.com/functions/scheduled-functions/

It is definitely the easiest way for Python code. Just paste your code and thats it

Full disclosure: I made this product. We are a YC company

We assumed commercial use cases for that price so the benefits of not having to deal with servers complexity + access control + etc.. are quickly paid back.

For hobby projects, we are thinking of creating a free plan

You won't spend much at all, it's fractions of a penny per call and I think there's a free tier.

0 code on the cron side (cannot be simpler), and if you only monitor a single route per hour is free as well. AND you get a free UI on how the cron job went :)

More info:

https://azure.microsoft.com/en-us/pricing/details/functions/

https://www.serverless360.com/blog/azure-functions-triggers-...

We self-host, but they have a GA cloud offering now. [1]

As long as you use js/ts, go, python, php or java that is. For some jobs we wrap process execution - e.g running a powershell command.

[0] https://docs.temporal.io/concepts/what-is-a-schedule

[1] https://temporal.io/cloud

[1]: https://workers.cloudflare.com

One of the cheaper options and probably one of the older options.

It might be more than what you need, but your needs will increase. The above cost will not.

That said, there are probably different forces at play if this is a personal infrastructure question: while cheap is good, if you aren't fluent in AWS it may be a slog to set up. If you're not, easiest thing is probably a real cron job run on a cheap VPS.

NixOS-configured systemd timers work great. Code them in bash, Go, Haskell, whatever. If you like polyglot (mostly to leverage any community's open source apps), NixOS is the best.

It isn't the cheapest or easiest for a single job. It's not hard to deploy to a random $5 VPS though (I use a Digital Ocean droplet). But once you have a box to deploy to, the incremental cost is as cheap as it gets.

Free, the job timeout is 10s but can be extended if needed, and minimum resolution is run every minute.

https://docs.deta.sh/docs/tutorials/cron-guide https://docs.deta.sh/docs/micros/cron

You assume someone purchased hardware (preferably available anywhere on the planet), powered it up, set it up, connected it to the internet, then built the software to handle your very specific task, then put it online to do specifically what you want and for a close-to-free price? And you're shocked this doesn't exist?

I’m using it to issue a daily db back up command for my app.

https://learn.microsoft.com/en-us/azure/azure-functions/func...

Zero infrastructure to manage, only write your script (in Dark, for that matter).

How much local compute do you have?

I think people underestimate dynamic DNS or a static IP if you can get one for your home for cheap.

Tl;dr: Github Actions for low frequency, upstash for high.

https://docs.gitlab.com/ee/ci/pipelines/schedules.html

Easiest way I found to do that was Azure Functions. Costs me about 35p per month. Mostly for storage for logs as far as I can tell. I'd sort that, but it's literally not worth the time it would take

https://sdf.org/

A VPS at hosthatch is still 4.99 I believe.

https://buddy.works/blog/introducing-cron-jobs

Is it about how to run cron? This is a thing of the past. As you are asking for 2022, look at systemd timers.

Is it about cheap hosting? The are lots of answers here already.

Or is it about how to monitor a site for changes?