GA Deployment Gotchas

Posted: 29 September, 2021 Category: devops Tagged: github-actionsci-cd-workflows

Ran into these. I dunno, I run into things, okay? Anyway. These are my beefs with Github Actions and Github pipelines in general. My therapeutic venting might save someone somewhere some grief. First off:

Turns out github doesn't care about your cute lil "draft" releases

The Github Actions webhooks or whatever that listen for workflow trigger events: yeah, they don't care about the 'draft' status of your release. To me this is madness, because a draft is precisely what I'd want to listen for, to kick off a deployment workflow which, let's face it, may or may not faceplant. So that if a faceplant ensues, we can wave the whole thing away with a "bah, it was just a draft anyway, see?". But no. GA will shout it from the hilltops that you have FAILED, and that it can't and WON'T help you roll anything back. emoji-unamused (Look, I get it. A release isn't something you're supposed to be able to take back. But can we have something for clearly non-mass-consumed repos? Given that local dev cycles are increasingly tugged skywards and experimentation is happening more and more in the cloud?)

Also, release tags don't die

Further underscoring GA's penchant for seeing folks twist in the wind:

  • publish a release with:
    • pre-release checkbox checked
    • a useful tag, e.g. release/1.2.3
  • once deploy is successful, uncheck pre-release.

Simple. But say it wasn't successful. Say you want to get back to how things were, and forget unfortunate recent events. Ha! Chance would be a fine thing. To back out of the release entirely, github has made this difficult to do for non-enterprise users, so plebeian freeloaders have to do this instead:

  • locally, do a git push --delete origin <tagname>. This reverts the release to a plain 'draft' state.
  • go into the github ui, click Edit on the draft
  • you will now (finally!!) have an option to delete the draft entirely.

The release configuration is itself sticky

  • Even with pre-release checked, github caches the repo files which it zips and attaches to your release.
  • Thereafter, it will be deaf and blind to whatever changes you make to the repo. You will have to nuke the release entirely if you need it to reflect changes.

Workflows are sticky too!

  • Github actions will only list workflows that have been committed to the main branch. If they're hanging out on some other branch and not merged yet, you're not gonna see em.
  • Github actions also caches your workflow, based on the filename of the .yml and even the workflow name itself (like, what??!). If it suddenly stops paying attention to your workflow updates, you might have to:
    • rename your yml file
    • rename the workflow name as well, if the above fails.

Here's the worst part: after effectively being forced to introduce a new workflow, the old one does not die. It can't die, unless you manually go to each run and nuke the logs. That's not gonna be practical for a workflow that's been around for a while. So, all you can really do is manually disable the old workflow. Ughetty-ugh to the power UGH.