> However, the service (gitlab.com) is constantly having issues, most of
> them not reported on their status page or on their twitter status account.
> For the last week it's been practically unusable, to the point where our
> whole dev team combined has wasted almost a hundred hours just re-trying
> builds and deployment jobs.
Do you happen to have some examples of these problems? I recall there are some
known problems with CI runners getting stuck, but I'm wondering if you were
running into other problems as well.
Those are exactly the problems we were running into. Our pipelines ran 3 consecutive stages (lint, test, deploy). Because every push and every merge triggers a pipeline, and because every stage runs a job, almost anytime we did anything on gitlab.com something (or everything) would get stuck. This slowed our dev/deploy process to a crawl, to the point where we spent all of Monday pretty much just checking, waiting for, or retrying jobs. Sometimes we would merge a branch and an hour later none of the jobs in the pipeline had even started (had status created or pending). Yesterday (Tuesday) things were still pretty much the same. Today we just moved on.
Jobs getting stuck might not seem like a big deal, but when every dev on the team needs to manually check every 5 minutes to make sure things are running and retrying everything, while clients are waiting on a hotfix, bugfix or a new feature that we promised we would ship by a certain deadline, then the issue becomes critical.
