Linode account compromised

Posted on October 5, 2020
Tags: , , ,

My Linode account got compromised!

5 days ago I got an email regarding events from Linode. Usually I tend to ignore these emails because from time to time they describe linode running some maintenance on my image.

Luckily, this time I didn’t ignore this email. To my surprise, I found this:

* ufljsc1r-eu-central - (215220139) System Shutdown - Completed Wed, 30 Sep 2020 20:00:58 GMT
 * ufljsc1r-eu-central - (215220141) Inactivate Linode - Completed Wed, 30 Sep 2020 20:00:59 GMT
 * linode2723253-us-west - (215220388) Linode Initial Configuration - Completed Wed, 30 Sep 2020 20:02:48 GMT
 * linode2723253-us-west - (215220389) Create Disk - linode2723253 - Debian 8 Disk - Completed Wed, 30 Sep 2020 20:02:48 GMT
 * linode2723253-us-west - (215220390) Thawing Disk - linode2723253 - Debian 8 Disk - Completed Wed, 30 Sep 2020 20:02:49 GMT
 * linode2723253-us-west - (215220391) Create Swap - Completed Wed, 30 Sep 2020 20:03:33 GMT
 * linode2723253-us-west - (215220392) System Boot - My linode2723253 - Debian 8 Disk Profile - Completed Wed, 30 Sep 2020 20:03:34 GMT
  * ufljsc1r-eu-central - (215193391) System Shutdown - Completed Wed, 30 Sep 2020 14:54:25 GMT
 * ufljsc1r-eu-central - (215193416) Change root password - [45311285] Ubuntu 18.04 LTS Disk - Completed Wed, 30 Sep 2020 14:54:50 GMT
 * ufljsc1r-eu-central - (215193438) System Boot - My Ubuntu 18.04 LTS Profile - Completed Wed, 30 Sep 2020 14:55:17 GMT
 * ufljsc1r-eu-central - (215193687) System Shutdown - Completed Wed, 30 Sep 2020 14:59:38 GMT

This didn’t seem right! I didn’t remember seeing this kind of message before. I logged into to my account( my password still worked) but then there were a bunch of instances I didn’t recognize the activity. As you can see from the text above a bunch of instances were created and deleted. And in that list the instance that was running this site also was deleted. So for a good few hours my site was down not that it matters :P.

I immediately panicked! And reached out to Linode support because I didn’t have backups turned on! Rather I felt like paying extra money to backup this site’s server wasn’t really money well spent. Some could also argue why pay to host a site in the first place at all but then I like having a server. That gives me freedom to tryout different things (hugo, wordpress) before I ended up making Hakyll the static site generator for this website but I digress!

I also tweeted @Linode account to understand if my account was a random account that got compromised or if Linode’s security was somehow breached and alot of accounts were compromised. Linode responded quickly to confirm that my account was freak show.

So I quickly changed my password on this account and also added 2FA on this account. The reason I didn’t setup 2FA sooner was because I didn’t really think my account/site was worth anyone’s effort! But lesson learned.

Linode was very prompt in getting me a snapshot of my instance that was running this site. I didn’t quite think that they would oblige since they had a feature that customers had to pay extra to ensure that my instance was constantly being backed up for a situation just like this. But kudos to Linode! They really won me over on this!

At this point I deleted all the unknown instances that was started by the person who gained access to my account. And thought that my account was now secured and damage reverted since I was easily able to boot backup from the saved snapshot given to me by the Customare Support from Linode.

As I was trying to check/analyze the damage of the account breach. A new instance got spun up. This spooked me up a bit. Upon investigating further, found out that the instances were being spun up from stack scripts.

My default instinct prompted me to delete those stack scripts and so I did. But as I deleted them I realized I should have opened those scripts to see what exactly they did. /me feels sad

If you ever find yourself in this situation here are things you should look do to secure yourself.

List of TODOs:

  1. Change password (should be a default thing to do)
  2. Set 2FA
  3. Make sure there aren’t any unauthorized apps or tokens that have access to your account.
  4. Check to make sure that there aren’t any stack scripts running. 4.1 Bonus point if you have curiosity to look at what the script does.
  5. Run a scan on all your online accounts on different service and make sure that you change the password if by faint chance they share same username/password.

One lesson that I learned from this incident is to enable 2FA. The flip side of this is if I ever decide to change my phone then I need to remember to move the accounts over to my new phone before I decommision the old one.

I remember couple of my online accounts for which I had 2FA setup on my older phones and now had to go through arduous process to get my account unlocked because I had totally forgotten that I had setup 2FA.

But as 2FA becomes more and more of a necessity I think this will start to become a part of the transfer protocol when I switch smartphones.

Earlier in the year my Etsy account was compromised, more on that in another post!