Most articles on AWS security rightfully spend a lot of time talking about the basics, such as setting up minimized IAM roles, encrypting data, and basic monitoring. It is more difficult to find guidance and specific implementation recommendations on advanced, automated security configurations.

In this series of articles, I will be outlining advanced security architecture for large AWS deployments. This is the kind of stuff I wished had been laid out for me when I started securing our cloud environments, and had to learn on my own.

It's also not enough to just show you the architectural layouts I recommend - I'll be diving deep to make sure the concepts, trade offs, and implementation details are clear.

To that end, for each component, I will look at the security goals, the architecture and concepts, and finally a specific implementation example with CloudFormation templates, where I will call out what should change for actual implementations (Everything here is designed for testing on a free personal account)

By the end of the series, you will have an application VPC to host business applications in, and a logically separate security VPC to aggregate data of interest such as logs, incident information, configurations, and an automated framework to analyze and respond to these items.

Since this is a large and complicated topic, I have broken it out into the following planned series of articles, building up the infrastructure and explaining it piece by piece.

1. A Multi-VPC security strategy (this article)
2. Using Session Manager to automate patching, secure SSH, and automated security agent installs.
3. Leveraging and centralizing AWS security tools with SecurityHub.
4. Aggregating and analyzing logs for security.
5. Building centralized compliance and security monitoring of infrastructure with Config, the Compliance Engine, and Rule Development Kit.

AWS Security Basics

I won't go in depth here about setting up basic security, as this is a well covered topic. If the team isn't solid on the basics, it would pay better dividends to develop practices and procedures to put them in place before moving on to building out the advanced security components.

Many cloud security incidents are the result of mis-configurations, where sensitive data is inadvertently exposed to the internet without controls. These controls are largely about preventing, and detecting those possibilities.

In brief, be sure to have:

• IAM practices that ensure minimal security groups, roles, and groups, along with a change management process for modifying these items.
• A solid patching strategy for any OS's, software, and platforms where Amazon doesn't take the lead.
• A NACL / network firewall strategy and review process, along with a solid understanding of what might make a service open to the internet vs. internal users.
• Similar to the above, a good understanding of bucket policies, and a review process to catch public buckets (We'll automate this later).
• Solid secure development practices with a focus on cloud native architectures.

Multi VPC Security Architecture

Many teams start their cloud journey with a single VPC where all applications, logs, and data are stored. This is sub-optimal from a security perspective because if the VPC itself should be compromised, so are many of the security controls we depend on as security professionals.

Therefore, I prefer to have a separate, dedicated VPC for security, which will constantly be monitoring the application VPC. As the business scales and adds additional VPC's, it is simple to roll out and centralize the same controls across all business environments, while keeping security analysts and data in a single place for all their work.

Eventually, we will have something like the below, with any number of application VPC's reporting back.

The key elements of this architecture is that all relevant information for security monitoring and incident analysis is stored in the security VPC. We can drain any number of other logs (sysmon, syslog, apache, etc) via the cloudwatch drain shown.

Note that this is a high level diagram, and does not show every service that will need to be leveraged to complete this setup - only the critical components to illustrate what we are trying to accomplish and the core AWS services we will be leveraging, When we implement each of these, I'll talk about each component in detail. The core services will be:

• Security Hub to centralize and view mis-configurations and potential incidents that Amazons internal tools have detected
• Config to track all infrastructure configuration changes, and validate them against policies, such as requiring S3 bucket encryption at rest.
• Session Manager to automate patching, configuration of common tooling, and SSH (instead of a bastion)
• CloudWatch to aggregate logs from many services and store them in our security VPC. We could easily add log analysis tools at this phase.

Setting up an application and security account

I have attempted to keep everything we build to minimal cost, expecting it to be built in a free tier personal account. Some of the stacks do carry charges (usually less than $1/hour for any examples shown in this series), so be sure to delete stacks as soon as you are done testing to minimize costs.

To follow along, you'll need a free AWS account, with an IAM user capable of managing organizations and accounts (I don't recommend using the root user for anything more than creating an admin user).

We will be creating separate accounts for each function. If you created a new AWS account, you first have to change it to an organization - go to the drop-down under your admin user name, and select My Organization.

Go through the steps of creating an application account, and a security account. The root account (the one that was initially created) can now be reserved only for management of the sub accounts. Because this master account has access to everything, it should be completely locked down (MFA on all logins, severely restricted list of who has any access, regular audits of activity, and perhaps more).

Securing the master account is a little outside the scope of this series, because you may want to include non-AWS protective measures to protect against internal threat actors, as well as the normal external actors. This account also is generally used to manage all billing for an organization, and may require other special considerations.

For each sub account, create an admin user with access keys who will be able to build infrastructure.

Building the VPC's within each account

We're finally ready to create the actual VPC's we'll be building in. I am basing the templates on this sample vpc creation template found in the AWS documentation with a few minor changes.

Both VPC's include an internet gateway and public subnets, to allow internal resources to reach out to the internet for things like patches, but no traffic is currently allowed from the internet to our VPC's.

In a production situation, you might use a VPN gateway or direct connect when setting up ingress routes to avoid any internet traffic for internal applications and especially for our security account.

Here is a sample cloud formation template for building out a basic VPC. You can find these templates, along with all the samples shown in this series, in my github:

AWSTemplateFormatVersion: "2010-09-09"
Description:  This template deploys an application VPC, with a pair of public and private subnets spread
  across two Availability Zones. It deploys an internet gateway, with a default
  route on the public subnets. It deploys a pair of NAT gateways (one in each AZ),
  and default routes for them in the private subnets.

Parameters:
  EnvironmentName:
    Description: An environment name that is prefixed to resource names
    Type: String
    Default: "lab"

  VPCName:
    Description: A friendly name to refer to this VPC
    Type: String
    Default: "Application"

  VpcCIDR:
    Description: IP range (CIDR notation) for the application VPC
    Type: String
    Default: 10.11.0.0/16

  PublicSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
    Type: String
    Default: 10.11.10.0/24

  PublicSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone
    Type: String
    Default: 10.11.11.0/24

  PrivateSubnet1CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
    Type: String
    Default: 10.11.20.0/24

  PrivateSubnet2CIDR:
    Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
    Type: String
    Default: 10.11.21.0/24


Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      CidrBlock: !Ref VpcCIDR
      EnableDnsSupport: true
      EnableDnsHostnames: true
      InstanceTenancy: "default" # I recommend setting to dedicated for sensitive workloads.
      Tags:
        - Key: Name
          Value: !Ref VPCName
        - Key: Environment
          Value: !Ref EnvironmentName

  InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
      Tags:
        - Key: Name
          Value: !Sub "${EnvironmentName}-${VPCName}-InternetGateway"
        - Key: Environment
          Value: !Ref EnvironmentName

  InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
      InternetGatewayId: !Ref InternetGateway
      VpcId: !Ref VPC

  PublicSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs '' ]
      CidrBlock: !Ref PublicSubnet1CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ1)

  PublicSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs  '' ]
      CidrBlock: !Ref PublicSubnet2CIDR
      MapPublicIpOnLaunch: true
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Subnet (AZ2)

  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 0, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet1CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ1)

  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId: !Ref VPC
      AvailabilityZone: !Select [ 1, !GetAZs  '' ]
      CidrBlock: !Ref PrivateSubnet2CIDR
      MapPublicIpOnLaunch: false
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Subnet (AZ2)

  NatGateway1EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway2EIP:
    Type: AWS::EC2::EIP
    DependsOn: InternetGatewayAttachment
    Properties:
      Domain: vpc

  NatGateway1:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway1EIP.AllocationId
      SubnetId: !Ref PublicSubnet1

  NatGateway2:
    Type: AWS::EC2::NatGateway
    Properties:
      AllocationId: !GetAtt NatGateway2EIP.AllocationId
      SubnetId: !Ref PublicSubnet2

  PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Public Routes

  DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId: !Ref PublicRouteTable
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId: !Ref InternetGateway

  PublicSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet1

  PublicSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PublicRouteTable
      SubnetId: !Ref PublicSubnet2


  PrivateRouteTable1:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Routes (AZ1)

  DefaultPrivateRoute1:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway1

  PrivateSubnet1RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable1
      SubnetId: !Ref PrivateSubnet1

  PrivateRouteTable2:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      Tags:
        - Key: Name
          Value: !Sub ${VPCName} Private Routes (AZ2)

  DefaultPrivateRoute2:
    Type: AWS::EC2::Route
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      DestinationCidrBlock: 0.0.0.0/0
      NatGatewayId: !Ref NatGateway2

  PrivateSubnet2RouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      RouteTableId: !Ref PrivateRouteTable2
      SubnetId: !Ref PrivateSubnet2

  # Here is where we might setup ingress for SSH or a bastion host. For now, no ingress allowed 
  NoIngressSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: "no-ingress-sg"
      GroupDescription: "Security group with no ingress rule"
      VpcId: !Ref VPC
      ```

We will build this twice - one VPC in our security account, one in our app account. To build this, you will need the access keys created earlier for both accounts. I assume you have stored one set of security keys during aws configure.

$ aws cloudformation create-stack --stack-name security-vpc --template-body file://security_vpc.yaml
$ export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAA
$ export AWS_SECRET_ACCESS_KEY=BBBBBBBBBBBBBBBBBBBBBBB
$ aws cloudformation create-stack --stack-name application-vpc --template-body file://application_vpc.yml

You should now be able to view VPC information in the console.

Conclusions

This seems like a good stopping point for this step. We have setup our initial account structure and admin users, and built a VPC in each account to host our information.

In the next section, we will setup SSH access to EC2, and automate control installation. From there, we will move into log centralization and automated controls

What else have you used to secure your AWS environments? Amazon is constantly coming out with new tools, so there are at least a few that I haven't touched on (yet!)