\nlook up information related to BGP on my computer, so I wrote a tweet asking for tools<\/a>.<\/p>\n
I got a bunch of useful replies as always, so this blog post shows some tools
\nyou can use to look up BGP information. There might be an above average number
\nof things wrong in this post because I don\u2019t understand BGP that well.<\/p>\n
I can\u2019t publish BGP routes<\/h3>\n
One of the reasons I\u2019ve never learned much about BGP is \u2013 as far as I know, I don\u2019t have access to publish BGP routes on the internet.<\/p>\n
With most networking protocols, you can pretty trivially get access to
\nimplement the protocol yourself if you want. For example you can:<\/p>\n
- \n
- issue your own TLS certificates<\/li>\n
- write your own HTTP server<\/li>\n
- write your own TCP implementation<\/li>\n
- write your own authoritative DNS server for your domain (I\u2019m trying to do that right now for a small project)<\/li>\n
- set up your own certificate authority<\/li>\n<\/ul>\n
But with BGP, I think that unless you own your own ASN, you can\u2019t publish routes yourself!
\n(you could<\/em> implement BGP on your home network, but that feels a bit boring to
\nme, when I experiment with things I like them to actually be on the real internet).<\/p>\nAnyway, even though I can\u2019t experiment with it, I still think it\u2019s super
\ninteresting because I love networking, so I\u2019m going to show you some tools I
\nfound to learn about BGP \ud83d\ude42<\/p>\nFirst let\u2019s talk through some BGP terminology though. I\u2019m going to go pretty
\nfast because I\u2019m more interested in the tools and there are a lot of high level
\nexplanations of BGP out there (like this cloudflare post<\/a>).<\/p>\nWhat\u2019s an AS (\u201cautonomous system\u201d)<\/h3>\n
The first thing we need to understand is an AS. Every AS:<\/p>\n
- \n
- is owned by an organization (usually a large organization like your ISP, a government, a university, Facebook, etc)<\/li>\n
- controls a specific set of IP addresses (for example my ISP\u2019s AS includes 247,808 IP addresses)<\/li>\n
- has a number (like 1403)<\/li>\n<\/ol>\n
Here are some observations I made about ASes just by doing some experimentation:<\/p>\n
- \n
- Some fairly big tech companies don\u2019t have their own AS. For example, I
\nlooked up Patreon on BGPView, and as far as I can tell they don\u2019t own as AS
\n\u2013 their main site (patreon.com,
\n104.16.6.49<\/a>) is in Cloudflare\u2019s AS.<\/li>\n- An AS can include IPs in many countries. Facebook\u2019s AS (AS32934<\/a>) definitely has IP addresses in Singapore, Canada, Nigeria, Kenya, the US, and more countries.<\/li>\n
- It seems like IP address can be in more than one AS. For example, if I look up 209.216.230.240<\/a>, it has 2 ASNs associated with it \u2013 AS6130 and AS21581. Apparently when this happens the more specific route takes priority \u2013 so packets to that IP would get routed to AS21581.<\/li>\n<\/ul>\n
what\u2019s a BGP route?<\/h3>\n
There are a lot of routers on the internet. For example, my ISP has routers.<\/p>\n
When I send my ISP a packet (for example by running
ping 129.134.30.0<\/code>), my ISP\u2019s routers
\nneeds to figure out how to actually get my packet to the IP address
\n129.134.30.0<\/code>.<\/p>\nThe way the router figures this out is that it has a route table<\/strong> \u2013 it has a list
\nof a bunch of IP ranges (like129.134.30.0\/23<\/code>), and routes it knows about to
\nget to that subnet.<\/p>\nHere\u2019s an example of a real route for
129.134.30.0\/23<\/code>: (one of Facebook\u2019s subnets). This one isn\u2019t from my ISP.<\/p>\n11670 32934 206.108.35.2 from 206.108.35.254 (206.108.35.254) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:33 2021\n<\/code><\/pre>\nI think that this is saying that one path to
129.134.30.0<\/code> is through the
\nmachine206.108.35.2<\/code>, which is on its local network. So the router might send
\nmy ping packet to206.108.35.2<\/code> next, and then206.108.35.2<\/code> will know how to
\nget it to Facebook. The two numbers at the beginning (11670 32934<\/code>) are ASNs.<\/p>\nwhat\u2019s BGP?<\/h3>\n
My understanding of BGP is very shaky, but it\u2019s a protocol that companies use to
\nadvertise BGP routes.<\/p>\nWhat happened yesterday with Facebook is that they basically made BGP
\nannouncements withdrawing all their BGP routes, so every router in the world
\ndeleted all of its routes related to Facebook, so no traffic could get there.<\/p>\nOkay, now that we\u2019ve covered some basic terminology, let\u2019s talk about tools you
\ncan use to look at autonomous systems and BGP!<\/p>\ntool 1: look at your ISP\u2019s AS with BGPView<\/h3>\n
To make this AS thing less abstract, let\u2019s use a tool called BGPView<\/a> to look at a real AS.<\/p>\n
My ISP (EBOX) owns AS 1403<\/a>. Here are the IP addresses my ISP owns<\/a>. If I look up my
\ncomputer\u2019s public IPv4 address, I can check that it\u2019s one of the IP addresses
\nmy ISP owns \u2013 it\u2019s in the104.163.128.0\/17<\/code> block.<\/p>\nBGPView also has this graph of how my ISP is connected to other ASes<\/p>\n
![](\"https:\/\/jvns.ca\/images\/ebox-graph.png\")
<\/p>\ntool 2:
traceroute -A<\/code> andmtr -z<\/code><\/h3>\nOkay, so we\u2019re interested in autonomous systems. Let\u2019s see which ASes I go through from<\/p>\n
traceroute<\/code> andmtr<\/code> both have options to tell you the ASN for every IP you go through. The flags aretraceroute -A<\/code> andmtr -z<\/code>, respectively.<\/p>\nLet\u2019s see which autonomous systems I go through on my way to facebook.com with
mtr<\/code>!<\/p>\n$ mtr -z facebook.com 1. AS??? LEDE.lan 2. AS1403 104-163-190-1.qc.cable.ebox.net 3. AS??? 10.170.192.58 4. AS1403 0.et-5-2-0.er1.mtl7.yul.ebox.ca 5. AS1403 0.ae17.er2.mtl3.yul.ebox.ca 6. AS1403 0.ae0.er1.151fw.yyz.ebox.ca 7. AS??? facebook-a.ip4.torontointernetxchange.net 8. AS32934 po103.psw01.yyz1.tfbnw.net 9. AS32934 157.240.38.75 10. AS32934 edge-star-mini-shv-01-yyz1.facebook.com <\/code><\/pre>\nThis is interesting \u2013 it looks like we go directly from my ISP\u2019s AS (1403) to
\nFacebook\u2019s AS (32934), with an \u201cinternet exchange\u201d in between.<\/p>\n
\nI\u2019m not sure what an internet
\nexchange<\/a> is but I know
\nthat it\u2019s an extremely important part of the internet. That\u2019s going to be for
\nanother day though. My best guess is that it\u2019s the part of the internet that
\nenables \u201cpeering\u201d \u2013 like an IX is a server room with a gigantic switch with
\ninfinite bandwith in it where a bunch of different companies put their
\ncomputers so they can send each other packets.
\n<\/small><\/p>\nmtr looks up ASNs with DNS<\/h3>\n
I got curious about how mtr looks up ASNs, so I used strace. I saw that it looked like it was using DNS, so I ran dnspeep<\/a>, and voila!<\/p>\n
$ sudo dnspeep\n...\nTXT 1.190.163.104.origin.asn.cymru.com 192.168.1.1 TXT: 1403 | 104.163.176.0\/20 | CA | arin | 2014-08-14, TXT: 1403 | 104.163.160.0\/19 | CA | arin | 2014-08-14, TXT: 1403 | 104.163.128.0\/17 | CA | arin | 2014-08-14\n...\n<\/code><\/pre>\nSo it looks like we can find the ASN for
104.163.190.1<\/code> by looking up thetxt<\/code> record on1.190.163.104.origin.asn.cymru.com<\/code>, like this:<\/p>\n$ dig txt 1.190.163.104.origin.asn.cymru.com\n1.190.163.104.origin.asn.cymru.com. 13911 IN TXT \"1403 | 104.163.160.0\/19 | CA | arin | 2014-08-14\"\n1.190.163.104.origin.asn.cymru.com. 13911 IN TXT \"1403 | 104.163.128.0\/17 | CA | arin | 2014-08-14\"\n1.190.163.104.origin.asn.cymru.com. 13911 IN TXT \"1403 | 104.163.176.0\/20 | CA | arin | 2014-08-14\"\n<\/code><\/pre>\nThat\u2019s cool! Let\u2019s keep moving though.<\/p>\n
tool 3: the packet clearing house looking glass<\/h3>\n
PCH (\u201cpacket clearing house\u201d) is the organization that runs a lot of internet
\nexchange points. A \u201clooking glass\u201d seems to be a generic term for a web form
\nthat lets you run network commands from another person\u2019s computer. There are
\nlooking glasses that don\u2019t support BGP, but I\u2019m just interested in ones that
\nshow you information about BGP routes.<\/p>\nHere\u2019s the PCH looking glass: https:\/\/www.pch.net\/tools\/looking_glass\/<\/a>.<\/p>\n
In the web form on that site, I picked the Toronto IX (\u201cTORIX\u201d), since that\u2019s what
mtr<\/code> said I was using to go to facebook.com.<\/p>\nthing 1: \u201cshow ip bgp summary\u201d<\/strong><\/p>\n
Here\u2019s the output. I\u2019ve redacted some of it:<\/p>\n
IPv4 Unicast Summary:\nBGP router identifier 74.80.118.4, local AS number 3856 vrf-id 0\nBGP table version 33061919\nRIB entries 513241, using 90 MiB of memory\nPeers 147, using 3003 KiB of memory\nPeer groups 8, using 512 bytes of memory Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up\/Down State\/PfxRcd\n...\n206.108.34.248 4 1403 484672 466938 0 0 0 05w3d03h 50\n...\n206.108.35.2 4 32934 482088 466714 0 0 0 01w6d07h 38\n206.108.35.3 4 32934 482019 466475 0 0 0 01w0d06h 38\n... Total number of neighbors 147\n<\/code><\/pre>\nMy understanding of what this is saying is that the Toronto Internet Exchange
\n(\u201cTORIX\u201d) is directly connected to both my ISP (EBOX, AS 1403) and Facebook (AS
\n32934).<\/p>\nthing 2: \u201cshow ip bgp 129.134.30.0\u201d<\/strong><\/p>\n
Here\u2019s the output of picking \u201cshow ip bgp\u201d for
129.134.30.0<\/code> (one of Facebook\u2019s IP addresses):<\/p>\nBGP routing table entry for 129.134.30.0\/23\nPaths: (4 available, best #4, table default) Advertised to non peer-group peers: 206.220.231.55 11670 32934 206.108.35.2 from 206.108.35.254 (206.108.35.254) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:33 2021 11670 32934 206.108.35.2 from 206.108.35.253 (206.108.35.253) Origin IGP, metric 0, valid, external Community: 3856:55000 Last update: Mon Oct 4 21:17:31 2021 32934 206.108.35.3 from 206.108.35.3 (157.240.58.225) Origin IGP, metric 0, valid, external, multipath Community: 3856:55000 Last update: Mon Oct 4 21:17:27 2021 32934 206.108.35.2 from 206.108.35.2 (157.240.58.182) Origin IGP, metric 0, valid, external, multipath, best (Older Path) Community: 3856:55000 Last update: Mon Oct 4 21:17:27 2021\n<\/code><\/pre>\nThis seems to be saying that there are 4 routes to Facebook from that internet exchange.<\/p>\n
the quebec internet exchange doesn\u2019t seem to know anything about Facebook<\/strong><\/p>\n
I also tried the same thing from the Quebec internet exchange QIX (which is
\npresumably closer to me, since I live in Montreal and not Toronto). But the QIX
\ndoesn\u2019t seem to know anything about Facebook \u2013 when I put in129.134.30.0<\/code> it
\njust says \u201c% Network not in table\u201d.<\/p>\nSo I guess that\u2019s why I was sent through the Toronto IX and not the Quebec one.<\/p>\n
more BGP looking glasses<\/h3>\n
Here are some more websites with looking glasses that will give you similar
\ninformation from other points of view. They all seem to support the sameshow ip bgp<\/code> syntax, maybe because they\u2019re running the same software? I\u2019m not sure.<\/p>\nThere seem to be a LOT of these looking glass services out there, way more than just those 3 lists.<\/p>\n
Here\u2019s an example session with one of the servers on this list:
\nroute-views.routeviews.org. This time I connected via telnet and not through a
\nweb form, but the output looks like it\u2019s in the same format.<\/p>\n$ telnet route-views.routeviews.org route-views>show ip bgp 31.13.80.36 BGP routing table entry for 31.13.80.0\/24, version 1053404087\nPaths: (23 available, best #2, table default) Not advertised to any peer Refresh Epoch 1 3267 1299 32934 194.85.40.15 from 194.85.40.15 (185.141.126.1) Origin IGP, metric 0, localpref 100, valid, external path 7FE0C3340190 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 6939 32934 64.71.137.241 from 64.71.137.241 (216.218.252.164) Origin IGP, localpref 100, valid, external, best path 7FE135DB6500 RPKI State valid rx pathid: 0, tx pathid: 0x0 Refresh Epoch 1 701 174 32934 137.39.3.55 from 137.39.3.55 (137.39.3.55) Origin IGP, localpref 100, valid, external path 7FE1604D3AF0 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 20912 3257 1299 32934 212.66.96.126 from 212.66.96.126 (212.66.96.126) Origin IGP, localpref 100, valid, external Community: 3257:8095 3257:30622 3257:50001 3257:53900 3257:53904 20912:65004 path 7FE1195AF140 RPKI State valid rx pathid: 0, tx pathid: 0 Refresh Epoch 1 7660 2516 1299 32934 203.181.248.168 from 203.181.248.168 (203.181.248.168) Origin IGP, localpref 100, valid, external Community: 2516:1030 7660:9001 path 7FE0D195E7D0 RPKI State valid rx pathid: 0, tx pathid: 0 <\/code><\/pre>\nHere there are a few options for routes:<\/p>\n
- \n
3267 1299 32934<\/code><\/li>\n6939 32934<\/code><\/li>\n701 174 32934<\/code><\/li>\n20912 3257 1299 32934<\/code><\/li>\n7660 2516 1299 32934<\/code><\/li>\n<\/ul>\nI think the reason there\u2019s more than one AS in all of these is that
31.13.80.36<\/code> is a Facebook IP address in
\nToronto, so this server (which is maybe on the US west coast, I\u2019m not sure) is
\nnot able to connect to it directly, it needs to go to another AS first. So all of the routes have one or more ASNs<\/p>\nThe shortest one is 6939 (\u201cHurricane Electric\u201d), which is a \u201cglobal internet backbone\u201d. They also have their own hurricane electric looking glass<\/a> page.<\/p>\n
tool 4: BGPlay<\/h3>\n
All the other tools so far have just shown us the current state of Facebook
\nrouting where everything is fine, but this 4th tool lets us see the history of
\nthis Facebook BGP internet disaster! It\u2019s a GUI tool so I\u2019m going to include a bunch of screenshots.<\/p>\nThe tool is at https:\/\/stat.ripe.net\/special\/bgplay<\/a>. I typed in the IP address 129.134.30.12 (one of Facebook\u2019s IPs), if you want to play along.<\/p>\n
First, let\u2019s look at the state of things before everything went wrong. I clicked in the timeline at 13:11:28 on Oct. 4, and got this:<\/p>\n
![](\"https:\/\/jvns.ca\/images\/bgplay-before.png\")
<\/p>\nI originally found this very overwhelming. What\u2019s happening? But then someone
\non Twitter pointed out that the next place to look is to click on the timeline
\nright after<\/em> the Facebook disaster happened (at 18:38 on Oct. 4).<\/p>\n![](\"https:\/\/jvns.ca\/images\/bgplay-after.png\")
<\/p>\nIt\u2019s pretty clear that something is wrong in this picture \u2013 all the BGP routes are gone! oh no!<\/p>\n
The text at the top shows the last Facebook BGP route disappearing:<\/p>\n
Type: W > withdrawal Involving: 129.134.30.0\/24\nShort description: The route 50869, 25091, 32934 has been withdrawn.\nDate and time: 2021-10-04 16:02:33 Collected by: 20-91.206.53.12\n<\/code><\/pre>\nIf I then click the \u201cfast forward\u201d button, we see the BGP routes start to come back:<\/p>\n
![](\"https:\/\/jvns.ca\/images\/bgplay-return.png\")
<\/p>\nThe first one announced is
137409 32934<\/code>. I don\u2019t think this is actually<\/em> the
\nfirst one announced though \u2013 there are a lot of route announcements inside the
\nsame second (at 2021-10-04 21:00:40), and I think the ordering inside BGPlay is arbitrary.<\/p>\nIf I click the \u201cfast forward\u201d button again, more and more routes start to come back and routing starts to go back to normal<\/p>\n
I found looking at this outage in BGPlay really fun, even though the interface is pretty confusing at first.<\/p>\n
maybe it is important to understand a little about BGP?<\/h3>\n
I started out this post by saying you can\u2019t change BGP routes
\nBGP, but then I remembered that in 2016 or 2017 there was a Telia routing issue<\/a> that caused us some minor
\nnetwork at work. And when that happens, it is actually useful to understand why
\nyour customers can\u2019t reach your site, even if it\u2019s totally out of your control.
\nI didn\u2019t know about any of these tools at that time but I would have liked to!<\/p>\nI think for most companies all you can do to respond to outages caused by
\nsomeone else\u2019s bad BGP routes is \u201cdo nothing and wait for it to get fixed\u201d, but
\nit\u2019s nice to be able to confidently<\/em> do nothing.<\/p>\nsome ways to publish BGP routes<\/h3>\n
If you want to (as a hobbyist) actually publish BGP routes, here are some links from the comments:<\/p>\n
- \n
- a guide to getting your own ASN<\/a><\/li>\n
- dn42<\/a> seems to have a playground for BGP (it\u2019s not on the public internet, but it does have other people on it which seems more fun than just doing BGP by yourself at home)<\/li>\n<\/ul>\n
that\u2019s all for now<\/h3>\n
I think there are a lot more BGP tools (like PCH has a bunch of daily snapshots of routing data<\/a> which look like fun), but
\nthis post is already pretty long and there are other things I need to do today.<\/p>\nI was surprised by how much information I could get about BGP just as a regular
\nperson, I always think of it as a \u201csecret network wizard\u201d thing but apparently there
\nare all kind of public machines anybody can just telnet to and use to look at the
\nroute tables! Who knew!<\/p>\n - dn42<\/a> seems to have a playground for BGP (it\u2019s not on the public internet, but it does have other people on it which seems more fun than just doing BGP by yourself at home)<\/li>\n<\/ul>\n
- An AS can include IPs in many countries. Facebook\u2019s AS (AS32934<\/a>) definitely has IP addresses in Singapore, Canada, Nigeria, Kenya, the US, and more countries.<\/li>\n
- Some fairly big tech companies don\u2019t have their own AS. For example, I