This isn't new - Thompson warned us 40 years ago (and I believe others before him) in his Reflections on Trusting Trust paper.
It's something I've been thinking about lately because I was diving into a lot of discussion from the early 90s regarding safe execution of (what was, at the time, called) "mobile code" - code that a possibly untrustworthy client would send to have executed on a remote server.
There's actually a lot of discussion still available from w3 thankfully, even though most of the papers are filled with references to dead links from various companies and universities.
It's weirdly something that a lot of smart people seemed to have thought about at the start of the World Wide Web which just fell off. Deno's permissions are the most interesting modern implementation of some of the ideas, but I think it still falls flat a bit. There's always the problem of "click yes to accept the terms" fatigue as well, especially when working in web development. It's quite reasonable for many packages one interacts with in web development to need network access, for example, so it's easy to imagine someone just saying "yup, makes sense" when a web-related package requests network access.
Also none of this even touches on the reality of so much code which exists to brutally impact a business need (or perceived need). Try telling your boss you need a week or two to audit every one of the thousands of packages for the report generator app.
Trusting Trust is not about this at all. It's about the compiler being compromised, and making it impossible to catch malicious code by inspecting the source code.
The problem here is that people don't even bother to check the source code and run it blindly.
It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub. I know they're auditable and you pin versions, but it's crazy to me that the recommended way to ssh to a server is to just give a random package from a random GitHub user your ssh keys, for example.
This is especially problematic with the rise of LLMs, I think. It's the kind of common task which is annoying enough, unique enough, and important enough that I'm sure there are a ton of GitHub actions that are generated from "I need to build and deploy this project from GitHub actions to production". I know, and do, know to manually run important things in actions related to ssh, keys, etc., but not everyone does.
Almost everyone will just copy paste this snippet and call it a day. Most people don't think twice that v4 is a movable target that can be compromised.
In case of npm/yarn deps, one would often do the same, and copy paste `yarn install foobar`, but then when installing, npm/yarn would create a lockfile and pin the version. Whereas there's no "installer" CLI for GH actions that would pin the version for you, you just copy-paste and git push.
To make things better, ideally, the owners of actions would update the workflows which release a new version of the GH action, to make it update README snippet with the sha256 of the most recent release, so that it looks like
I mean, I think there's a difference between trusting GitHub and trusting third parties. If I can't trust GitHub, then there's absolutely no point in hosting on GitHub or trusting anything in GitHub Actions to begin with.
But yes I do think using tags is problematic. I think for one, GitHub should ban re-tagging. I can't think of a good reason for a maintainer to re-publish a tag to another commit without malicious intent. Otherwise they should provide a syntax to pin to both a tag and a commit, something like this:
`uses: actions/checkout@v4.5.6@abcdef9876543210`
The action should only work if both conditions are satisfied. This way you can still gain semantics version info (so things like dependabot can work to notify an update) but the commit is still pinned.
---
I do have to say though, these are all just band-aids on top of the actual issue. If you are actually using a dependency that is compromised, someone is going to get screwed. Are you really going to read through the commit and the source code to scan for suspicious stuff? I guess if someone else got screwed before you did they may report it, but it's still fundamentally an issue here. The simple answer is "don't use untrustworthy repositories" but that is hard to guarantee. Only real solution is to use as few dependencies as possible.
after this incident, I started pinning all my github workflows with hashes, like other folks here I guess :D
But I quickly got tired of doing it manually so I put together this [0] quick and dirty script to handle it for me. It just updates all workflow files in a repo and can be also used as a pre-commit hook to catch any unpinned steps in the future. It’s nothing fancy (leveraging ls-remote), but it’s saved me some time, so I figured I’d share in case it helps someone else :)
You would still be exposed if you had renovate or dependabot make a PR where they update the hash for you, though. Here's a PR we got automatically created the other day:
I don't think you should ever allow dependabot to make direct commits to the repository. The only sane setting (IMO) is that dependabot should just make PRs, and a human needs to verify that and hit merge. My personal opinion is for any serious repositories, allowing a robot to have commit access is often a bad time and ticking time bomb (security-wise).
Now, of course, if there are literally hundreds of dependencies to update every week, then a human isn't really going to go through each and make sure they look good, so that person just becomes a rubber-stamper, which doesn't help the situation. At that point the team should probably seriously evaluate if their tech stack is just utterly broken if they have that many dependencies.
Even if you don't automerge, the bots will often have elevated rights (it needs to be able to see your private repository, for instance), so it making a PR will run your build jobs, possibly with the updated version, and just by doing that expose your secrets even without committing to main.
Aren't GitHub action "packages" designate by a single major version? Something like checkout@v4, for example. I thought that that designated a single release as v4 which will not be updated?
I'm quite possibly wrong, since I try to avoid them as much as I can, but I mean.. wow I hope I'm not.
The crazier part is, people typically don't even pin versions! It's possible to list a commit hash, but usually people just use a tag or branch name, and those can easily be changed (and often are, e.g. `v3` being updated from `v3.5.1` to `v3.5.2`).
Fuck. Insecure defaults again. I argue that a version specifier should be only a hash. Nothing else is acceptable. Forget semantic versions. (Have some other method for determining upgrade compatibility you do out of band. You need to security audit every upgrade anyway). Process: old hash, new hash, diff code, security audit, compatibility audit (semver can be metadata), run tests, upgrade to new hash.
You and someone else pointed this out. I only use GitHub-org actions, and I just thought that surely there would be a "one version to rule them all" type rule.. how else can you audit things?
I've never seen anything recommending specifying a specific commit hash or anything for GitHub actions. It's always just v1, v2, etc.
Having to use actions for ssh/rsync always rubbed me the wrong way. I’ve recently taken the time to remove those in favor of using the commands directly (which is fairly straightforward, but a bit awkward).
I think it’s a failure of GitHub Actions that these third party actions are so widespread. If you search “GitHub actions how to ssh” the first result should be a page in the official documentation, instead you’ll find tens of examples using third party actions.
So much this. I recently looked into using GitHub Actions but ended up using GitLab instead since it had official tools and good docs for my needs. My needs are simple. Even just little scripting would be better than having to use and audit some 3rd party repo with a lot more code and deps.
And if you're new, and the repo aptly named, you may not realize that the action is just some random repo
> It's always been shocking to me that the way people run CI/CD is just listing a random repository on GitHub.
Right? My mental model of CI has always been "an automated sequence of commands in a well-defined environment". More or less an orchestrated series of bash scripts with extra sugar for reproducibility and parallelism.
Turning these into a web of vendor-locked, black-box "Actions" that someone else controls ... I dunno, it feels like a very pale imitation of the actual value of CI
Location: Portland, Oregon
Remote: yes, or local in Portland
Willing to relocate: no
Technologies: Elixir, Erlang, JavaScript, Python, C, Lisp, Lua, VBA, Java, C#, ...
Résumé/CV: contact me
Email: ian@hedoesit.com
I've been programming for two decades and have a lot of experience in various technologies. Lately I've been doing various things related to networking, automation, and databases. Things like parsing SQLite databases to perform ETL into Erlang ETS tables, setting up secure networks for some Elixir projects, and some aggregation of data for easier exploration. I really enjoy all things computers and programming, and have a vast set of skills that are perfect for any task. Looking forward to hearing about interesting tech!
Location: Portland, Oregon
Remote: yes, or local in Portland
Willing to relocate: no
Technologies: Elixir, Erlang, JavaScript, Python, C, Lisp, Lua, VBA, Java, C#, ...
Résumé/CV: contact me
Email: ian@hedoesit.com
I've been programming for two decades and have a lot of experience in various technologies. Lately I've been doing various things related to networking, automation, and databases. Things like parsing SQLite databases to perform ETL into Erlang ETS tables, setting up secure networks for some Elixir projects, and some aggregation of data for easier exploration. I really enjoy all things computers and programming, and have a vast set of skills that are perfect for any task. Looking forward to hearing about interesting tech!
- Understanding how it expands the surface area of both the API and discussion of a struct (and thus also libraries and dependencies of the librar{y,ies}). Now you don't need to ask someone if they're using v1.0 of some package, but also which revision of a struct they're using, which they may not even know. This compounds when a library releases a new breaking major version, presumably - `%SomeStruct{}` for v2 may be different from `%SomeStruct{}` v1r2.
- Documentation seems like it would be more complex, as well. If `%SomeStruct{}` in v1.0 has some shape, and you publish `v1.1`, then realize you want to update the revision of `%SomeStruct{}` with a modified (or added) field, would there be docs for `v1.1` and `v1.1r2`? or would `v1.1` docs be retroactively modified to include `%SomeStruct{}v1.1r2`?
- The first example is a common situation, where an author realizes after it's too late that the representation of some data isn't ideal, or maybe even isn't sufficient. Typically, this is a solved with a single change, or rarely in a couple or few changes. I'm not sure if the complexity is worth it. I understand the desire to not fragment users due to breaking changes, but I'm not sure if this is the appropriate solution.
- How does this interact with (de-)serialization?
I happen to be working with the SQLite file format currently, and I generally really enjoy data formats. It's not exactly the same as this, since runtime data structures are ephemeral technically, but in reality they are not. The typical strategy for any blob format is to hoard a reasonable amount of extra space for when you realize you made a mistake. This is usually fine, since previous readers and writers ought to ignore any extra data in reserved space.
One of the things one quickly realizes when working with various blob formats is the header usually has (as a guess), 25% of it allocated to reserved space. However, looking at many data formats over the last several decades, it's extraordinarily rare to ever see an update to them that uses it. Maybe once or twice, but it's not common. One solution is to have more variable-length data, but this has its own problems.
So, in general, I'm very interested in this problem (which is a real problem!), and I'm also skeptical of this solution. I am very willing to explore these ideas though, because they're an interesting approach that don't have prior art to look at, as far as I know.
EDIT: Also, thanks for all the work to everyone on the type system! I'm a huge fan of it!
> This compounds when a library releases a new breaking major version, presumably - `%SomeStruct{}` for v2 may be different from `%SomeStruct{}` v1r2.
You should never reuse the revisions. If you launch a major version, then it means you only support r5 onwards. Do not reset it back to r1.
I am also not sure if the user needs to know the version. Remember that if I am in r5, because of subtyping, the code naturally supports r1, r2, r3, and r4, and those already have to be pretty printed. All of the work here is to "generate automatic proofs" via type signatures, no new capability or requirement is being introduced on the type representation of things.
> Documentation seems like it would be more complex, as well. If `%SomeStruct{}` in v1.0 has some shape, and you publish `v1.1`, then realize you want to update the revision of `%SomeStruct{}` with a modified (or added) field
I don't think documentation is more complex because that's something already factored in today. Functions add new options in new releases. Structs add new fields. And today we already annotate the version they were added. Perhaps this approach can automate some of it instead.
---
Other than that, we should be skeptical indeed. :)
As a preamble, the discussion of versioning is difficult since it's abstract; this is difficult to discuss in general.
> You should never reuse the revisions. If you launch a major version, then it means you only support r5 onwards. Do not reset it back to r1.
Hm, so hypothetically a v1.0.0 of a library could begin its life with structs at a greater-than-1 revision number. This seems odd, since a major version bump is able to be breaking, but revisions don't have the same semantic meaning - it just means "it's different from before".
This kind of breaks my mental model of major version bumps. I consider v1 of something to be distinct from v2, where it just happens to be that v2 typically is mostly compatible with v1. However, with revisions, it maintains that connection explicitly. I suppose the answer to that would be to rename the structs and have them start at revision 1, but now we're back at square one, where I could've just done that without revisions to maintain backwards compatibility by introducing a new struct.
I'm not sure if I'm conveying what's odd about that sufficiently. Let me know if I should think on that more and try explaining it again.
> I don't think documentation is more complex because that's something already factored in today. Functions add new options in new releases. Structs add new fields. And today we already annotate the version they were added. Perhaps this approach can automate some of it instead.
The expectation here would be that a struct revision would be a minor version bump, correct? That seems like it kind of breaks semantic versioning, since a user of v1.0.0 wouldn't necessarily have any behavioral changes in v1.1.0, since the structs and users of the structs would be identical, right? It kind of pushes the minor version bump onto the consumer code at their discretion, but not in a typical way. Now if I bump a dependency's minor, I don't necessarily want to bump my own minor - but maybe I do? Especially if I'm a middleman library, where I pass behavior from client code to one of my dependencies.
This is getting a bit confusing to consider the hypothetical situations. The main gut feeling I have about this is the same about versioning in general - it's fine until it's not, which is usually some unknown point in the future when things get complex. Kind of the gist of my hesitation is just that versioning is already a complicated problem, and introducing more variation in that seems like it really needs to be a long experiment with the expectation that it may not work. Unfortunately, the situations in which versioning rears its ugly head is typically only in long-used, complex projects. So I don't really know how a test bed for this could exist to give realistic results.
As a bit of a disclaimer to everything I just said, semver is not the holy grail, in my opinion, and I think it's perfectly reasonable to experiment with it and try alternatives. Maybe part of my issue is just that I haven't entertained other versioning schemes that can sufficiently handle the difference between behavioral and representational changes.
> This kind of breaks my mental model of major version bumps.
If you want, you can reset the revisions in a major version, as you could rename all modules and change the behavior of all functions, but that's hardly a good idea. Revisions are not any different. If you give a revision a new meaning in a major version, it is going to be as confusing as giving a new implementation to a function: users upgrading will have to figure out why something has subtly changed, instead of a clear error that something has gone away.
> The expectation here would be that a struct revision would be a minor version bump, correct?
We could make either minor and major versions (as described by semver) work. For example, if we want to allow new revisions to be minor versions, we could make it so every time you install a dependency, we automatically lock all structs it provides. This way, if a new version is out, you have to explicitly upgrade any new revision too.
Of course, you could also release a new major version of the library once you introduce a new revision and _that's fine_. The goal is not to avoid major versions of the library that defines a struct, the goal is to avoid forcing its dependents to release major versions, which has a cascading effect in the ecosystem. Putting in Elixir terms, I want to be able to release Phoenix v2 without forcing LiveView to immediately become v2. LiveView should be able to stay on v1.x and say it supports both Phoenix v1 and v2 at once. But in most typed languages, if you change a field definition, it is game over.
The guarantees you need to provide are not that many: as long as two revisions overlap at one point in time, you can offer a better upgrading experience to everyone, instead of assuming *all code must be updated at once*.
I forgot to mention, the typical situation for these things for added elements is to add `_ex` or the like, which is not a great solution. You can see this in various places in Erlang, especially around erl_interface and related aspects.
On the flip side, what if you realize a data structure needs to remove elements? Say revision 2 adds a field, but revision 3 removes it?
Location: Portland, Oregon
Remote: yes, or local in Portland
Willing to relocate: no
Technologies: Elixir, Erlang, JavaScript, Python, C, Lisp, Lua, VBA, many more
Résumé/CV: on request
Email: ian@hedoesit.com
I've been programming for nearly two decades and have a lot of experience in various technologies. Lately I've been doing some fun stuff with Elixir and Zig related to graphics programming. I love working on weird, unknown projects. My wide range of experience (both with and without computers) helps me figure out tasks that don't have much prior art. I'm also equally thrilled by getting into the nitty gritty of deeply technical things like writing parsers for obscure binary formats. If you have some weird ideas that you don't have someone on your team to figure out, let me know!
