When Heroku Goes Rogue

I recently had an interesting experience where Heroku was running an old version of my codebase, so I had to go in and figure out how to start fresh. This was one of those production only problems, which helped isolate it as an environment issue. Note I’ve never had this happen before in years of using Heroku, so this is probably an edge case you won’t run into, but we’ll go through how I identified the problem and how it was solved.

First I checked the Heroku file system and poked around to see what I had.

heroku run bash --app <app-name

Inside the bash prompt, I was able to cat a few of my changed files. ¬†In my case, these were up to date. Makes sense, seeing as it’s spinning up a whole new server to run this one-off process.

My problem was in my Resque Workers, so I changed some of the capitalization in my error messages. Sure enough, I started having two different copies of my error messages showing up in Resque-Web. This confirmed to me that I had multiple versions of Resque workers reading from my queue.

I fixed this by completely wiping my heroku repo and putting a new one up. Heroku makes this easy via their Heroku-Repo plugin for the Heroku-Cli

heroku plugins:install https://github.com/heroku/heroku-repo.gite

Followed by purging, resetting the repo, and pushing the repo.

heroku repo:purge_cache -a appname
heroku repo:reset -a appname
git push <heroku remote> master

heroku restart -a appname

And that’s it! My application was up to date, no more errors, and everyone was happy. The best part-zero downtime! I hope you never have to follow these steps, but it’s worth knowing about the Heroku-Repo plugin if you ever need to confirm that yes-it’s your code’s fault, not Heroku’s.

Leave a Reply

Your email address will not be published. Required fields are marked *