Lessons Learned in Cloud Networking – AWS vs Azure

I’ve been working a lot with cloud networking lately. I will share some of my findings as this is still quite new and documentation around some topics is poor. Especially on the Azure side. Let me just first start with two statements that I have seen made around cloud networking:

Cloud networking is easy! – Not necessarily so. I’ll explain more.

We don’t need networking in cloud! – Wrong. You do but in basic implementations it’s not visible to you.

This post will be divided into different areas describing the different components in cloud networking. You will see that there are many things in common between AWS and Azure.

System Routes

Within a VPC/VNET, there are system routes. If 10.0.0.0/22 was assigned to the VPC/VNET, there will be a system route saying along the lines of “10.0.0.0/22 local”. Subnets are then deployed in the VPC/VNET and there is full connectivity due to the system route. This route will point to a virtual router which is the responsibility of AWS/Azure. Normally this router will have a “leg” in each subnet, at the first IP address of the subnet, for example 10.0.0.1 for the 10.0.0.0/24 subnet.

In AWS, system routes can not be overridden. It doesn’t matter if you try to put a static route or a longer route, the system route always takes precedence.

In Azure, you CAN override system routes. These are called UDR, User Defined Routing. While very useful, it can be confusing, and a little dangerous, that Azure chooses a route according to the list below:

1. User-defined route
2. BGP route
3. System route

This means that a BGP route of equal length will be preferred to the system route. I learned this because I was advertising 0.0.0.0/0 over BGP and it made a host lose internet connectivity because it was following the BGP route instead of the system route.

Internet Connectivity

In AWS, to provide internet connectivity to a subnet, an Internet Gateway (IGW) or a NAT Gateway must be attached to the VPC, and the route table associated with the subnet, must have a route towards the IGW or NAT Gateway. A subnet with internet connectivity is often referred to as a public subnet.

In Azure, internet connectivity is provided by default. My only guess as to why is that they want to let people easily get started even if they don’t have any networking knowledge. I don’t like this default though. Subnets shouldn’t have internet access unless I decided that they should have. This also means that if you don’t want the subnet to have internet access, you have to play stupid tricks by writing ACLs and by doing so you risk blocking access to Azure services. I don’t think this is a sensible default and at the very least Azure should provide an easy mechanism to remove internet access for a subnet.

Virtual Router

As I described above, both in AWS and Azure there is a virtual router that lives in each and every subnet. Because broadcasts aren’t supported, there are some tricks behind the scene like answering ARP replies in order for virtual machines to have connectivity between each other.

One thing I’ve noticed in Azure, which is kind of annoying, is that the virtual router does not reply to ICMP. This makes troubleshooting more difficult as you can’t ping the virtual router or see it as a hop in a traceroute.

The other thing with Azure is, that the virtual router inserts itself into EVERY flow. I was very surprised by this as this is not really documented anywhere and certainly not in Azure’s official documentation. The way I discovered this was that I had two devices talking BGP to each other, all the expected routes were there, and still traffic was being dropped somewhere. I did extensive troubleshooting and then I found something weird. I had two devices in the same subnet, let’s say we had devices A and B and A was 10.0.0.4 and B was 10.0.0.5. Let’s say that A’s MAC was 0001.aaaa.aaaa and that B’s MAC was 0001.bbbb.bbbb. I noticed that A did not have a MAC address of 0001.bbbb.bbbb for B. This was confusing as they are in the same subnet. Instead I saw a MAC address of 0123.4567.89ab. What was this MAC address? It turns out that the Azure virtual router does this “man in the middle” where it replies to ARP requests with its own MAC. All traffic is then relayed through the virtual router. Even though I had two devices on the same subnet, they were not directly communicating with each other. This broke my forwarding of traffic because my traffic was now not only between these two devices, the traffic was hitting Azure’s routing tables as well. I hadn’t put any routes into those because I shouldn’t be needing them. I then had to update Azure’s routing tables with static routes in order to get traffic through. I don’t really like how the virtual router inserts itself into every flow.

VPN Gateways

Both AWS and Azure offers VPN gateways, whether they be called Virtual Network Gateway (Azure) or Virtual Private Gateway (AWS). AWS now also has the Transit Gateway. The AWS VGW/TGW is fairly straight forward. It’s fast to create, it supports BGP, it’s easy to configure. The only caveat was it didn’t do IKEv2 until just recently. IKEv2 is now supported. Make sure to enable route propagation to get your routes from the VGW to your route table. Keep in mind that it may not be possible to change some settings for the VGW after it has been created.

Azure though is the exact opposite. It does support IKEv2 and has done for a long time but that’s pretty much the only thing positive I can say about it. It takes a LONG time to create the VNG, often 30 minutes or so. Why, Azure, why?! When you create the VNG, you need to point out which VNET it belongs to. This can NOT be changed at a later stage. You also can’t change from policy based VNG to dynamic VNG. You also have to select a SKU to size your VNG. Excuse me, I thought this was cloud?!

The worst part of Azure’s VNG though, is that it’s very poorly documented how you do active/active BGP. I basically had to reverse engineer this to figure it out. First off, the VNG is deployed into something that is called a Gatewaysubnet. This is a subnet that you need so that the VNG can get an IP address that you will later peer to. This means that you don’t get a shared subnet between Azure’s VNG and your VPN device, which is in the case in AWS. Something like 169.254.0.0/30. Instead you have to run multi-hop eBGP to Azure but not only that, you also need to peer from a loopback because you can’t configure Azure to have two BGP peers on your side. This also means that you need to come up with a link network yourself, and then put a static route to Azure’s BGP peer IP over that link network. This was not documented anywhere to be found.

Azure’s VNG is a job poorly done and I can’t imagine why they’ve designed it this way. It feels like someone lacking enough networking knowledge designed the VNG. I know this is probably not the case but that just what it feels like.

Also keep in mind that you can’t start out with a VNG that is static and then move to dynamic. Considering how long it takes to create and delete VNGs, a change like that means you’ll likely lose connectivity to your on-premises environment for an hour or more.

Summary

I’ve tried to cover as many things as I could but I’m sure I’ve missed something. Networking in the cloud is different and not always as cloud like as you would expect. Especially in the case of Azure. While they do support you overriding system routes, that’s pretty much the only advantage they have over AWS. It’s quite obvious that they are a couple of years behind AWS feature-wise. I can only hope that they seriously reconsider the way they deploy and configure the VNG. I hope you find some good information in this post as it’s quite difficult to find the proper documentation, especially for Azure. Thanks for reading!