Close the Gate: Why You Need Egress Controls in your Security Groups

Network engineers and security professionals spent decades securing on prem networks. Today many workloads run in the cloud. 10 years ago if you wanted build out infrastructure you needed purchase orders and months of lead time. Today you only need a credit card to build out the same thing in the cloud.

Agile delivery and DevOps means developers with little network or security knowledge are often put in charge of building and managing cloud environments. Many introductory tutorials omit a lot of the necessary security controls. In the case of AWS many of the default settings don’t implement security best practices or even encourage users to implement the recommendations from Amazon’s own Well Architected Framework.

One of the areas where this is most glaring is security groups and more specifically egress controls. The AWS console, CLI and SDKs default to allowing all traffic to exit security groups. Sure this makes it easy to connect to the internet from your application. It also makes it very easy for an attacker wanting to exfiltrate data once they have compromised resources in your stack.

Information security has the concept of “assume breach”. The idea behind this is that applications and infrastructure are architected in such a way that the impact of any breach is minimised. Make life difficult once an attacker has broken through your defences.

Not allowing open egress is an important component of your security controls. Generally developers know what inbound connections should be allowed for their environments. 443/tcp to the load balancer, 22/tcp to the bastion and so on. When it comes to outbound connections more often than not developers use the default option — open.

Teams should adopt the same allow listing approach to egress that has been a standard security control for ingress for over 2 decades. It takes a small amount of time upfront to identify all the legitimate connections required within an environment. Still it takes a lot longer to deal with a breach where the attacker was able to easily extract all of your data.

Developers need to know what network connections their application makes. If you’re building immutable infrastructure there’s no need to connect to operating system package repositories. Similarly your bastion should only allow connections to the database nodes. Generally a connection to an IRC server or third party DNS is an indicator of compromise.

Say your load balancer is in the load-balancer security group and your web servers are in the web security group; you need a rule allowing traffic out load-balancer for 443/tcp (or 80/tcp if you’re offloading TLS at the load balancer) with a target of the web security group. Then in the web security group, you need to allow 443/tcp (or 80/tcp) from the load balancer group. The web servers need to connect to the RDS instances, so that means two new rules - one to allow 3306/tcp the web group to connect to the db group and one to allow 3306/tcp from the web group into the db group. Then there’s the connection to the S3 endpoint, the resources used by the CodePipeline and so on.

Managing all of these two sided or “mutual” security group rules gets tedious.

For many teams, determining the appropriate rules is challenging. The effort involved in manually creating mutual rules kills any prospect of it happening in existing environments. If you’re using terraform it doesn’t need to be like that.

While trying to roll out mutual security group rules at scale, I realised a need for a module to make this easier. That module is now available on the Terraform Registry as skwashd/mutual-security-groups/aws. Drop this module into your configuration and specify the connections between your security groups.

Using the example above for our web servers we could use something like this to allow the connections needed for our application to function properly:

module "mutual-security-groups" {
  source  = "skwashd/mutual-security-groups/aws"
  version = "1.0.0"
  rules = [
     {
         source_sg_id = "sg-a1ba1"
         target_sg_id = "sg-80443"
         destination_port = "80"
         description = "Allow HTTP from the ALB to webs"
     },
     {
         source_sg_id = "sg-80443"
         target_sg_id = "sg-db3306"
         destination_port = "3306"
         description = "Allow webs to RDS"
     },
     {
         source_sg_id = "sg-80443"
         target_sg_id = "sg-f113s"
         destination_port = "443"
         description = "Allow webs to S3 endpoint"
     }
  ]
}

Internally the module uses the aws_security_group_rule resource to create the security group rules. This means in your other modules you can define the security groups with egress locked down and the module will setup the rules properly for you.