Thursday, November 19, 2020

AWS Network Firewall: More Than Just Layer 4


Up until very recently, network prevention has been quite limited in Amazon Web Services (AWS). Consumers were left with the following options:
  • Create Security Groups to limit various types of layer 3 and 4 traffic to/from Elastic Compute Cloud (EC2) instances
  • Create Network Access Control Lists (NACL) to limit layer 3 and 4 traffic to/from entire Virtual Private Cloud (VPC) subnets
  • Route traffic through a network appliance running as an EC2 instance (not as "cloud-friendly" as this is often less scalable and sized to handle peak traffic)
To add more network protection options, AWS just released an awesome new capability in select regions called AWS Network Firewall. Protections that are afforded here are:
  • Allow or deny based on source IP and/or port, destination IP and/or port, and protocol (also known as 5-tuple)
  • Allow or deny based upon domain names
  • Allow or deny based upon Suricata-compatible IPS rules
Wait... we can make forwarding decisions based on PAYLOADS?! Now we're talking!

Similar to physical or virtual firewalls, some thought about which assets to protect, which types of traffic to hone in on, and other major considerations must be determined prior to deployment. Likely, you may have already done this if deploying an EC2 instance-based firewall.

To deploy one or more of these, changes to the existing VPC architecture are required (more on this in a bit).

Some Assembly Required

A best practice outlined by AWS is to architect your VPC to support this VPC Firewall. It is not as simple as turning on the service and being on your merry way. To support the upcoming case study, I'll keep the re-architected VPC (formerly set up by default) to the following:
  • VPC: Create or use a default VPC
  • Subnets: Use two of the existing subnets created by the default VPC
    • Subnet 1: Rename to FirewallSubnet. This will be used to contain the Network Firewall endpoint.
    • Subnet 2 through X: Rename to ProtectedSubnetX. These will be used to house the instances being protected by the Network Firewall
  • EC2 instance(s): Deploy or re-home into the ProtectedSubnet(s)
  • Route Tables: The default Route Table will no longer work, but more on this after the Network Firewall is deployed

Network Firewall Rule Groups

The first component to build out the AWS Network Firewall (and last on their list in the VPC service... WHY AMAZON?!) is the Network firewall rule group. This is where you decide what to allow or deny based on the previous list (5-tuple, domain names, or IPS rules).

This is very simple to set up:
  • Determine if this rule group will be stateless (inspect packets within the context of the traffic flow) or stateful (inspect individual packets on their own)
  • Give the rule group a name and optional description
  • Assign a capacity (see for more detail)
  • If a stateful rule group, follow the rest of the wizard to define your rule(s) for the traffic you wish to allow or deny

Firewall Policies

The next piece of the puzzle is a Firewall policy. These policies simple consist of one or more Firewall rule groups so stepping through the wizard is very straight-forward (give it a name and description, add the rule groups, choose a default action, and tag the policy if you wish).


The Firewall configuration is slightly more complex, but not by much. Here is where the re-architecting decisions begin to make more sense. 

To begin configuring this Firewall, yet again, give it a name and description. Selecting the appropriate VPC is the next step (easy if you only have one... otherwise, ensure you are choosing the correct one!). After this, choose the FirewallSubnet (or whatever you named it) subnet as this is the subnet in which the Firewall endpoint will be placed.

The only other required step is to select the Firewall policy created earlier.

Some (More) Assembly Required

Your Firewall will deploy in a few minutes, but... it won't actually be doing anything until you force traffic through it. To do so, some VPC Route Table adjustments need to be made. Following guidance from, creating a few Route Tables is the easiest-to-understand option. Here is an example of establishing three Route Tables to force traffic through the Network Firewall:

  • ProtectedRouteTable: Used to send any traffic from any instances in the ProtectedSubnet to any destination outside the subnet to the Network Firewall
  • FirewallRouteTable:  Used to send any traffic to the Internet Gateway
  • IGWRouteTable: Used to send any traffic destined to the ProtectedSubnet from outside the subnet to the Network Firewall
Sounds complex (and it is!) but the upcoming case study will show how this is set up in more detail. But first...


To appease your security analysts (or to at least see the fruits of your labor), you should be logging Network Firewall alerts at the very least. This is achieved by navigating to the Firewall details tab to set up logging as this service does not log AT ALL unless you configure it to. The usual suspects are available as log destinations: S3 bucket, CloudWatch, Kinesis data firehose.

Case Study: Bring Your Own Suricata Rules (BYOSR?)

VPC Setup

And now a very detailed walkthrough. To explore this service, I began by using a "clean" and unused region in AWS to not affect some production workloads. I chose us-west-2 as it is one of two US regions that offer this service at the time of this writing (the other being us-east-1).

Since the default VPC is already there, time to make a few changes. I left the subnet configuration as it was (even through I'm only using two or the four subnets for this example) other than changing two names: ProtectedSubnet for one and FirewallSubnet for another.

EC2 Target Instance

Next up was to deploy an instance I'd like to protect. I simply chose an Amazon Linux 2 AMI, logged in, and added the following to support my upcoming "attack":

sudo yum update -y
sudo yum install httpd -y
echo "Hello Visiter" | sudo tee /var/www/html/Hello
sudo systemctl enable httpd
sudo systemctl start httpd

Network Firewall Rule Group

The major capability that really piqued my interest is the fact that you can not only block traffic based upon known bad IPs, ports, or domains, but also by ingesting Suricata IPS rules! To test this out, I created a stateful rule group that simply blocks any time it sees this very nasty traffic or... traffic that contains the text "Hello". Here is what this Suricata IPS-compliant rule looks like:

Network Policy

The policy, as stated earlier, is very simple. I just have the policy incorporate my lone rule.

Network Firewall

Again, my setup is very simple as it's ultimately a single rule in a rule group, a single rule group in a policy, and then the policy is used in the Network Firewall configuration.


To ensure that my rule is actually doing anything, I'll set up a CloudWatch log group called FirewallAlerts. I'll dig into this log group in a bit once some data is generated.

Back to the Network Firewall configuration under Firewall details, I chose the FirewallAlerts CloudWatch log group as my destination, but was faced with a few options on which types of data to send there:

Another opportunity to enable Flow logging! To save some money, through, and to keep my CloudWatch log group focused on alerts, I just chose Alerts here.

Route Tables

By far, the most complex part of this setup is the Route Table configuration, so I'll keep it as brief as possible. I followed the previous guidance of creating three Route Tables like so:


The Target of vpce-0698c483f02db5617 represents the Network Firewall. Also, the ProtectedSubnet was associated with this Route Table (for obvious reasons). With this, any traffic sent from the instance to a destination outside the subnet will be forwarded to the Network Firewall.


Here, the target of igw-0dd503989e7a965b9 represents the Internet Gateway which was already created when setting up the default VPC. The FirewallSubnet was associated with this Route Table and, combined with these rules, will force any traffic meant for the outside world to be routed to the Internet Gateway.


This Route Table is a bit different as it looks at the traffic as it is returning to the VPC. Since I wish to have the return traffic traverse the firewall, I have any traffic destined for the ProtectedSubnet IP space ( in this case) configured to be routed to the Network Firewall. Since this Route Table is to be assigned, not to a subnet, but to an Internet Gateway, an Edge association is configured.


And now to trigger the Firewall block. If you remember, the Suricata rule was simply looking for the text "Hello" in any TCP traffic:

deny tcp any any -> any any (msg: "No Hellos"; content: "Hello"; sid: 4000000; rev:1;)

To attempt to trigger this rule, I opened up a web-browser session and navigated to the IP address plus /Hello.

What caused the issue here? Was it the Network Firewall? Let's find out! Going back to the CloudWatch log group, I found an interesting log:

    "firewall_name": "NoHellos",
    "availability_zone": "us-west-2c",
    "event_timestamp": "1605798915",
    "event": {
        "timestamp": "2020-11-19T15:15:15.242314+0000",
        "flow_id": 967950538327967,
        "event_type": "alert",
        "src_ip": "XXX.XXX.XXX.XXX",
        "src_port": 49882,
        "dest_ip": "",
        "dest_port": 80,
        "proto": "TCP",
        "alert": {
            "action": "blocked",
            "signature_id": 4000000,
            "rev": 1,
            "signature": "No Hellos",
            "category": "",
            "severity": 3
        "app_proto": "http"

Looks like it worked!


Thus far, I'm really impressed with what can be done using this new service. Yes, it is quite the level of effort if it involves re-architecting an existing deployment and can be costly (well over $200/month if my implementation were to run 24/7), but I'm sure as the service really takes off more capabilities will be included to enhance your organization's defense in depth even further.

Monday, February 3, 2020

Impossible Traveler

Use Case

An impossible traveler, at least in the case of this article, is defined as the occurrence of a user interacting with the same resource from two different locations, but, given the time delta and distance between the sources, could not possibly have made that trip in a reasonable amount of time. 

For instance, let's assume that we have a user log in to a web resource from an IP address in San Francisco, CA at 20:00 UTC and then submit a POST to the site from an IP address located in New York, NY at 20:30 UTC. There is no method of transportation (at least until Elon Musk figures this out) that a human can take to travel over 4,100km in 30 minutes.


We are basing the location of the user on a source IP address. As you probably know, Virtual Private Network (VPN) connections can easily allow a user to log in to the resource from one location, connect their VPN client to a VPN endpoint in another country within seconds, and log into (or simply interact with) the resource again. 

This will give the appearance of an impossible traveler. This could still warrant investigation depending on your corporate security policy. For example, if VPNs are required, they broke policy by connecting to the resource without using the VPN. Or, if VPN clients are not allowed, this would help highlight that violation as well.

Existing Tools

There are plenty of tools, most specifically supporting cloud services, that can pinpoint impossible traveler situations. Some vendors may also categorize this activity as a part of user anomaly detection. Some vendors I'm familiar with (there are many more):

These services do work pretty well, but what if we are generating our own logs that need to be analyzed? Don't worry, I wrote a Python script for that...

Located on my GitHub, there is a script that I... mostly wrote myself (thanks Stack Overflow) that you give a few parameters:

$ python -h
usage: [-h] [-k KMPH] firstIP secondIP

positional arguments:

optional arguments:
  -h, --help            show this help message and exit

  -k KMPH, --kmph KMPH  Speed (in kmph)

What the script is essentially doing is taking your IP addresses, doing a GeoIP lookup to get their coordinates, then determining, based on the average speed you give it, how long it would take to get from the first IP to the second IP. By default, I chose the speed to default to 900kmph since, according to Wikipedia, the average cruising speed is between 880 and 926kmph.

To use this script, simply add the first IP in question, then the secondIP to get a rough time of how long it would take a commercial flight, at average cruising speed, to get from the first source to the second. If the time different between the two events is less than that, you have an anomaly. Let's take a look at the following log:

2020-02-03T20:02:17Z "Login successful for ryanic"
2020-02-03T21:13:11Z "Login successful for ryanic"

If we were to do a GeoIP lookup (which the script is essentially doing), we would find that is located in Norwalk, CT (IBM) and is located in Houston, TX (HP). We would also find that these two locations are roughly 1,821km apart*.

* Assuming the GeoIP records are correct

In this situation, the user logged in just 1 hour, 10 minutes, and 54 seconds apart from these two locations. Let's use our script to see if this is possible given today's air travel:

$ python
Time to travel 1821.0 kilometers (Hours:Minutes)
-> 1:57

There you have it: Looks like that's a no. If you've traveled via commercial flights, you know we did not account for taxing to/from gates as well as getting up to cruising speed. This flight more realistically would take 3+ hours gate-to-gate.

This script is also flexible in that you can give it a different speed for commuters. Let's say average speed in your local commuting area is 60kmph. You can give the -k (or . --kmph) flag to the script and indicate the speed of your choice.

So, if you find your vendor does not support all "impossible traveler" situations, feel free to use my script!