AWS ABCs – Network Building Blocks

Given that my technical background is largely in the networking space (exhibit A, exhibit B, exhibit C(CIE)), one of the first things I tried to wrap my head around when being introduced to AWS is how networking works in the AWS cloud.

What I attempted to do was build a mental model by relating cloud networking constructs such as Virtual Private Cloud (VPC), subnets, and routing tables to on-prem, physical networking constructs. This worked pretty well but I did get tripped up at times because some of these constructs don’t map exactly one-for-one.

This post will explain the mental model I used while also calling attention to the elements or behaviors that don’t map exactly between on-prem and AWS.

The basis for building the model will be a single VM on-prem and a single compute instance in AWS. I’m going to build all the networking constructs around both of these elements, starting from the outer-most layers and working closer and closer to the VM/instance.

Our VM and compute instance need them some networking!

A Note on Layer 2

On the AWS platform there is no explicit building blocks for Layer 2 connectivity. There’s no “elastic virtual switch” or some-such that you have to define and there’s no need to attach an instance to a virtual switch port. Nothing like that.

Layer 2 connectivity in AWS is abstracted away and handled by the platform’s network fabric itself.

All of the networking constructs in AWS focus on Layer 3; Layer 2 “just works”.

Virtual Routing and Forwarding ~ Virtual Private Cloud

The most abstract construct in the on-prem networking world is Virtual Routing and Forwarding (VRF). A VRF is a logical partitioning of the routing and ARP/ND (Address Resolution Protocol/Neighbor Discovery) tables that accomplishes two major things:

  1. Allows for network-layer separation of different security zones. Devices in one VRF cannot talk to devices outside of their VRF without explicit permission and configuration.
  2. Allows for reuse of IP address space across multiple VRFs. Networks in one VRF are free to reuse the very same IP address space as networks in another VRF.

The term “VRF” is used by some network vendors while other vendors use terms such as “routing instance”.

The comparable building block in AWS is the Virtual Private Cloud (VPC) which provides an isolated, private network in which customers can attach compute instances and other resources.

A VRF and an AWS VPC are same-ish

Similarly to a VRF, a VPC accomplishes two major things architecturally:

  1. Logical isolation and segmentation of instances and services within a VPC from any instances and services in another VPC. In other words, traffic cannot be exchange between VPCs unless explicit permission and configuration has been made. Granted, traffic could pass from say a compute instance in one VPC to a compute instance in another VPC via the Internet, but never directly between VPCs without explicit configuration.
  2. Allows for IP address reuse. The same Classless Inter-Domain Routing (CIDR) blocks can be used across multiple VPCs without causing a conflict.

While it’s possible to reuse the same CIDR blocks or subnets between multiple VPCs, it’s highly discouraged. Doing this will tie your hands in the future if you ever decide you need to interconnect two VPCs with overlapping address space. Plan your address space as if you will one day connect your VPCs together and you won’t have to worry about being in a tight spot in the future should that plan become reality.

Subnets == Subnets

An on-prem subnet is just like an AWS subnet. All good? Great. Next topic!

A subnet is a subnet. Unless it’s not.

Ok, ok, I’ll extrapolate before moving on.

Just like in a VRF, the next logical construct is a subnet and just like a VRF, a VPC can contain multiple of them. There are a couple of fun differences though:

  1. A subnet in an AWS VPC is always tied to a specific availability zone (AZ) within the region. This makes it incredibly easy to place instances across AZs when designing for high availability: configure a subnet (or multiple) for each AZ and then spread the instances across the subnets. The instances will be provisioned in the AZ of their respective subnet.

    This is my subnet. There are many like it, but this one is mine.
  2. Unlike in a VRF, there is no need to explicitly configuring routing between subnets within a VPC. No need to define routes in a routing table and no need to configure dynamic routing. The VPC defines an implicit router that will get packets between subnets within the VPC.
  3. Within the address space that is assigned to a subnet, some of the addresses are automatically reserved by the AWS platform:
    • The network address (“.0” if the subnet is a /24)
    • The network address + 1 (“.1” if the subnet is a /24): this is used by the implicit router function and serves as the default gateway for the subnet.
    • The network address + 2 (“.2” if /24): this is used by the AWS platform to provide a recursive DNS resolver to instances within the subnet.
    • The network address + 3 (“.3” if /24): this address is reserved for future use.
    • The broadcast address (“.255” if /24): since the AWS network fabric doesn’t support broadcast traffic, the broadcast address in each subnet is reserved and not used.

Routing Tables

This is where things got trippy for me.

I’m used to on-prem networks where the routing table–be it in a VRF or otherwise–reaches a state of convergence that causes traffic to a particular destination to follow the same path regardless what the source subnet of that traffic is.

Not so in AWS.

In AWS you bind a routing table to a subnet such that traffic sourced from that subnet is forwarded based on the entries in its associated routing table. By default a VPC is created with a single routing table that gets bound to every subnet, however additional routing tables can be created to modify the forwarding behavior for specific subnets. For example, this is how public and private subnets are defined (see my earlier post on that subject here).

I’m trying to show here that a subnet in an on-prem network is more tightly bound to a VRF and in AWS is tightly bound to a subnet

The trippy part here is that routing tables are decoupled from subnets; they’re defined and managed as  discreet resources and are then bound to a subnet.

Gateways == Gateways. Mostly.

Just like in an on-prem network, AWS networks have a concept of gateways. There are a couple subtle but important differences:

  • Gateways in AWS exist at the edge of a VPC; they’re building blocks used to get traffic in and out of a VPC
  • Routing within a VPC–i.e., between subnets–does not require a gateway be provisioned

In an on-prem network, each subnet has a concept of a (default) gateway; it’s the thingie that gets traffic in and out of that subnet and is an object that must be explicitly configured and managed.

In AWS networking, the default gateway attached to a subnet is implicitly instantiated when the subnet is created. You, as the cloud administrator, don’t have to explicitly configure anything. Repeating what I said in the subnets section above: there is an implicit routing function inside each VPC that takes care of routing traffic between subnets in the VPC.

What we’re left with then is the need to get packets in and out of the VPC to and from things like the Internet and an on-prem location via either Virtual Private Network (VPN) or Direct Connect (Dx). There are different types of gateways for each of those use cases: internet gateways, virtual private gateways, and direct connect gateways. I don’t want to repeat what I’ve already written, so refer to a previous post I wrote (AWS ABCs – EC2 Internet Connectivity) that covers these gateways in more detail.


Networking in AWS uses a lot of familiar building blocks but doesn’t adopt all of the same rules as on-prem networking. In this post, I’ve attempted to explain the most common building blocks and how they compare and contrast to their on-prem counterparts in an effort to help my fellow network engineers wrap their heads around how networking works in AWS.

If I’ve helped you avoid some of the confusion that I struggled with as I have learned the AWS platform, then this has been a successful post. 😬

If you have any tips, trips, or hints for folks who are learning networking in AWS, please comment below.



Disclaimer: The opinions and information expressed in this blog article are my own and not necessarily those of Amazon Web Services or Amazon, Inc.

Leave a Reply

Your email address will not be published. Required fields are marked *

Would you like to subscribe to email notification of new comments? You can also subscribe without commenting.