Security in Microservices

Introduction

Recently, there has been a large push away from monolithic application in favor of small, composable, purpose-built microservices. I won't go into the details of a microservice architecture, as Martin Fowler does a great job of that on his site. For the purpose of this post, I'll work under the assumption that you have evaluated microservices and determined that the pros outweigh the cons for your project (and, if you haven't...well, you did it wrong).

While I've really appreciated the increased development velocity that a microservice-based architecture has brought to my projects, it became evident quickly that I had overlooked security concerns in my initial designs. In fact, security is something commonly ignored in a microservices architecture. Security today is more important than ever before (and it will only continue to become more important over time). In a world where consumers continue to shift more of their lives online, it's on us as engineers to ensure that our systems protect our users as much as we possibly can. Of course, it also happens that protecting our users has the added benefit of protecting our business and keeping most of us employed. 

Scenario

Imagine we're engineers building an e-commerce API as a monolithic application. For brevity, the only endpoint we're considered with is the one that lets a customer create an order. We want to ensure that this endpoint can only be called by someone who has previously authenticated and has a valid API token. Depending on your framework, you'll add some middleware, filters, interceptors, (insert other tech buzzword here) which check for a valid token in the API request and rejects requests that don't include it. Great, we can assume we've exposed exactly one entry into our application, and it's properly secured.

However, luckily for us, our website gains popularity. So much popularity, that our monolithic API has become a bottleneck to our scaling and development. After a few long fiery debates, we have decided to break our application into microservices that we can build and deploy independently. Logically, we create a service that is responsible for accepting orders. Of course the API still needs authentication, so we'll leave that middleware we previously created. Our original monolithic API now becomes an API gateway that performs authentication and passes the request into the backend. Things are going well, our site is now "web-scale."

Problem

Unfortunately, there's a big flaw in this system. Previously, our application had one entry and it was authenticated. Now our application has two entries, one external (our API gateway), and one internal (the orders microservice). We've secured our external interface, but what about our internal interface? Anyone inside our network can call the create order endpoint without needing a token. In fact, by moving to microservices we've actually built a less secure system.

Before we make any more moves, let's look at the problems we're trying to solve:

  1. We want to ensure requests come from a trusted party
    Our orders microservice should be confident it is receiving traffic from a source that it can trust
     
  2. We want to convey context to our microservices
    Our orders service should know the user that is placing an order

It's worth noting here that microservices don't (and can't) cause any security issues. "Microservices" is simply a word for an architectural design pattern. The implementation of microservices at your organization determines their security - the pattern does not.

Assumptions

Before someone else calls it out, I want to acknowledge a couple of things about this post. Quite intentionally, I'm making the following assumptions

  1. It's possible for a malicious person to be in a privileged network position
    It's actually quite trivial to be in a "privileged network position" in a sufficiently complex network (and even in a simple one). Guest WiFi networks, SSH tunnels through dev machines that share keys with production, etc. Additionally, some malicious employees do exist.
     
  2. Communications with our database were authenticated over SSL to begin with
    If we're not connecting to the database with SSL and authenticating those connections, there's a larger issue we should fix before worrying about this post. However, given that all major database vendors provide SSL and authentication support, we should be using it by now.
     
  3. Your application has data that it wants to product or can be considered private
    It's possible that you simply may not have a need for rigid internal security (your data may simply not warrant it). However, even if you don't, I'd argue there are still reasons for requiring authentication to your microservices. Authentication conveys context, it tells you who is making the call, which you may not need for access control, but you may want for auditing, logging, etc. You can still take advantage of this information.

Alright, so we've set the baseline assumptions we're making. Let's tackle this problem.

Suboptimal Solutions

Being engineers, we're inherently lazy (and that's a good thing). So let's start with the solutions that seem easiest to implement:

  1. Strict network policies only allowing traffic from our API gateway
    This is the easiest solution to problem #1. We can identify that all requests to our orders microservice came from our trusted API gateway. We can safely secure our orders microservice this way. However, this doesn't inherently give us any context - it doesn't tell us who is placing the order. It also doesn't scale as well as we'd want it do. We either continue to add ACLs every time we add a host to our network, or we open up an entire subnet (which brings us back to the same problem).
     
  2. Rewrite all requests in the API gateway to include contextual information
    This solves problem #2 - we get information about who is making requests (at the expense of a lot of boilerplate code). Unfortunately, it still provides no guarantees that those requests came from a trusted system. In order for our API to feel secure, we want to solve both problems.
     
  3. Proxy the token through to all service calls
    We said at the beginning that we have some sort of API token that is used to authenticate requests. What if we just passed that through to all of our microservices? Well, this ensures we're getting traffic from a trusted source (or at least someone who knows the token). It also conveys context, because we can find out the user that owns that token. Unfortunately, it introduces a ton of latency - every microservice now needs to exchange that token for the associated information it relates to (the user, the time it was issued, it's expiration, etc).

Neither of these solutions are quite right. We want this security powered by data (after all, our web-scale e-commerce site is hot and if it's going to constantly scale, we want our security to scale with it).

JSON Web Tokens

What we want are something like JSON Web Tokens (JWT). These are base64-encoded claims (formed as a JSON object) that are digitally signed and compacted into a single string that can be passed between services. Because the claims are transferred as part of the string, each request to our microservice will include the appropriate context in the JWT claims. We don't need to exchange the token with the database to figure out the user that is associated with it. We just need access to the keys to be able to verify the signature of those claims, and then we can feel confident that a) the traffic came from a trusted source, and b) we're aware of the context of the request.

Proposal

My proposal for securing your microservices involves a combination of opaque access tokens, JSON Web Tokens, and an API gateway.

  • Build an API gateway that represents the externally-facing API contract
    Having an API gateway gives you a single, coherent interface that external consumers can interact with. Consumers of your service can't (don't want to, shouldn't, etc) be expected to interact with each service individually. An API gateway provides an excellent layer for you to perform common functionality such as rate limiting and authentication.
     
  • Provide your API consumers with opaque, meaningless API tokens
    Generate a random set of 64+ characters using a cryptographically secure random number generator. These opaque tokens are useful - they're virtually impossible to guess, but are also predictable in length and size. When your clients could be all over the world, bandwidth is important. It's much better to be able to transfer a meaningless 64-byte token than hundreds of bytes of JWT claims that mean nothing outside of your application.
     
  • Validate those tokens in your API gateway, and create a JSON Web Token
    At the API gateway layer, we can verify that the opaque token is still valid. Once we've confirmed the token is valid, we can form the claims to make up a JSON Web Token, and sign it with a shared secret that only our trusted services know about. This JWT can be short lived (a few seconds), since a new one will be created and signed on each request to our API gateway.
     
  • Provide the JSON Web Token to your microservice calls internally
    Your API gateway should then proxy requests to your services with the generated JWT. The digital signature solves problem #1, we know that the token had to originate from someone with knowledge of the secret key we use to sign our tokens. The claims embedded in the token solve problem #2, we know that, once we've validated the signature, the claims of the token provide the context of the request.

Conclusion

Hopefully this high level overview is helpful when designing your own APIs and service architectures. While I don't claim it to be perfect, it has provided me with a decent sense of security in my own application development.

If you're doing something different to secure your services, I'd love to hear about it. It's always interesting to see what others are doing.