Overview

Border gateway protocol (BGP) is the routing protocol used to exchange network reachability information between autonomous systems (ASes), also known as the routing protocol of the Internet. These prefixes carry some attributes that are used to perform loop prevention (for example, AS PATH), to choose the best entry or exit into/from an AS or other functions.

For a junior network engineer, troubleshooting BGP may sound scary and this article will show different mistakes in configuration, debugging procedures used to identify them, and solutions and recommendations.

Throughout this article these case studies will be based on this diagram:

1. Inside the AS—iBGP

A BGP sessions built between peers that are part of the same AS is called an internal BGP (iBGP). In order to successfully identify and solve iBGP problems, let’s first summarize its characteristics:

  • it requires a full mesh of iBGP speakers (Route Reflector or Confederations are beyond the scope of this article).
  • iBGP sessions are usually formed between the loopbacks of peers.
  • iBGP does not require neighbors to be one hop away (directly connected).
  • Most attributes of the prefixes are not changed when advertised to an iBGP peer .

In the following sections, we will see how each of these characteristics can have a negative impact in the network if they are not followed nor understood.

1.1. TCP Connectivity/Routing

Let’s start our troubleshooting journey with the simplest scenario: lack of end-to-end connectivity between the BGP peers. In the example below, R1 is trying to establish a BGP session with R2, both of them using their loopback IPs.

R1#sh   ip   bgp   summary   |   b Neighbor
Neighbor                  V             AS   MsgRcvd  MsgSent    TblVer   InQ OutQ  Up/Down   State/PfxRcd
192.168.255.2          4            123          21           21         0         0      0     00:31:24   Active
R1#
R1#ping 192.168.255.2 source 192.168.255.1Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.255.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.255.1
…..
Success rate is 0 percent (0/5)
R1#

The simplest way of troubleshooting this problem is to check the routing on each side:

R1#sh ip route 192.168.255.2
Routing entry for 192.168.255.2/32
Known via “ospf 1”, distance 110, metric 4, type intra area
Last update from 192.168.13.3 on FastEthernet0/1, 00:33:49 ago

Routing Descriptor Blocks:

* 192.168.13.3, from 192.168.255.2, 00:33:49 ago, via FastEthernet0/1
Route metric is 4, traffic share count is 1
R1#
————————————————————————
R2#
R2#sh ip route 192.168.255.1
% Subnet not in table
R2#

As shown, R2 does not know how to reach R1’s loopback. This requires some investigation of the interior gateway protocol (IGP), OSPF in our case:

R1#sh run | s router ospf
router ospf 1
log-adjacency-changes
network 192.168.13.0 0.0.0.255 area 0
R2#sh run | s router ospf
router ospf 1
log-adjacency-changes
network 192.168.0.0 0.0.255.255 area 0

Looking at the OSPF configuration, we immediately see that R1’s loopback, 192.168.255.1, is not advertised into OSPF (since it’s not matching 192.168.13.0 0.0.0.255).

Upon fixing, the connectivity between R1’s and R2’s loopbacks works and the BGP session goes UP:

R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#router ospf 1
R1(config-router)# network 192.168.0.0 0.0.255.255 area 0
R1(config-router)#end
R1#*Mar 1 01:05:53.839: %SYS-5-CONFIG_I: Configured from console by console
R1#
R1#ping 192.168.255.2 source 192.168.255.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.255.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.255.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/48/64 ms
R1#
1.2. BGP—Source Address of the BGP Connection

In this scenario, the network administrator is trying to establish a BGP peering between the border routers, R1 and R2, using their loopback0 interfaces. He has completed this configuration:

R1#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 192.168.255.2 remote-as 123
no auto-summary
R2#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 192.168.255.1 remote-as 123
no auto-summary

… but the BGP session does not come up, although connectivity between the routers is okay:

R1#sh ip bgp summary | b Neighbor
Neighbor            V         AS  MsgRcvd MsgSent    TblVer   InQ OutQ   Up/Down   State/PfxRcd
192.168.255.2    4        123       0            0              0          0     0       never        Active
R1#
R1#ping 192.168.255.2 source 192.168.255.1Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.255.2, timeout is 2 seconds:
Packet sent with a source address of 192.168.255.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/50/80 ms
R1#

Troubleshooting this issue is relatively easy. Enabling the debugging for BGP will provide valuable information about the problem:

R1#debug ip bgpBGP debugging is on for address family: IPv4 Unicast
R1#
R1#
*Mar 1 01:13:40.371: BGP: 192.168.255.2 open active, local address 192.168.13.1
*Mar 1 01:13:40.447: BGP: 192.168.255.2 open failed: Connection refused by remote host,
open active delayed 31104ms (35000ms max, 28% jitter)
R1#

The debug logs tell us that R1 is trying to establish the BGP session with IP address 192.168.13.1, which is refused by R2. Since the configuration on both sides is using the loopback0 IP addresses (192.168.255.x), R2 expects the connection to come from that IP so it is normal behavior to reject the connection coming from the physical interface.

By default in Cisco IOS, all connections that are initiated by the router itself (no matter whether they are BGP, syslog, RADIUS, TACACS, etc.) will have the source IP address of the outgoing interface—fa0/1 in this case.

In order to solve this problem, we need to tell the router to use its loopback0 IP address as the source of the connection—this is done using the command “neighbor x.x.x.x update-source lo0“. Although doing this only on one router makes the BGP session to get established, the recommendation and practice is to configure on both sides:

R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#router bgp 123
R1(config-router)#neighbor 192.168.255.2 update-source lo0
R1(config-router)#end
*Mar 1 01:27:36.727: %SYS-5-CONFIG_I: Configured from console by console
R1#
*Mar 1 01:27:50.011: %BGP-5-ADJCHANGE: neighbor 192.168.255.2 UpR1#
R1#sh ip bgp summary | b Neighbor
Neighbor          V     AS    MsgRcvd MsgSent     TblVer   InQ OutQ   Up/Down   State/PfxRcd
192.168.255.2   4    123           6         6            237      0       0       00:01:45           275
R1#
1.3. Next-Hop Not Known Inside the AS

Let’s add another iBGP between border router, R1, and internal router, R3.

R1#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes

neighbor 1.1.1.1 remote-as 100

neighbor 1.1.1.1 password cisco123

neighbor 192.168.255.2 remote-as 123

neighbor 192.168.255.2 update-source Loopback0
neighbor 192.168.255.3 remote-as 123
neighbor 192.168.255.3 update-source Loopback0
no auto-summary
R3#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 192.168.255.1 remote-as 123
neighbor 192.168.255.1 update-source Loopback0

no auto-summary

Looking at R3, we can see that the BGP session is UP, it receives 235 prefixes from R1, but none of them is making it into the routing table:

R3#
R3#sh ip bgp summary | b Neighbor
Neighbor          V    AS     MsgRcvd MsgSent  TblVer   InQ OutQ   Up/Down   State/PfxRcd
192.168.255.1   4    123         7           6          236      0     0       00:02:38       235
R3#R3#sh ip route bgpR3#
R3#

The best way to identify the issue is to have a look at the routes in the BGP table:

R3#sh ip bgp
BGP table version is 236, local router ID is 192.168.255.3
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete
Network             Next Hop       Metric   LocPrf   Weight   Path
* i1.1.1.11/32     1.1.1.1            0         100        0        100 ?
* i1.1.1.12/32     1.1.1.1            0         100        0        100 ?* i1.1.1.13/32     1.1.1.1            0         100        0        100 ?

* i1.1.1.14/32     1.1.1.1            0         100        0        100 ?

R3#sh ip bgp 1.1.1.11

BGP routing table entry for 1.1.1.11/32, version 3
Paths: (1 available, no best path)
Not advertised to any peer
100
1.1.1.1 (inaccessible) from 192.168.255.1 (192.168.255.1)
Origin incomplete, metric 0, localpref 100, valid, internal

R3#
R3#sh ip route 1.1.1.1
% Network not in table
R3#

Since 1.1.1.1 is not known , R3 cannot accept any BGP routes that has this IP address as the BGP next-hop. The two most common solutions to this problem are:

  • On border router R1, redistribute the transit uplink (1.1.1.0/30) into the internal OSPF domain:
R1#sh run | s router ospf
router ospf 1
log-adjacency-changes
redistribute connected subnets route-map MATCH_FA0-0_ONLY
network 192.168.0.0 0.0.255.255 area 0
R1#
——————————————————
R3#sh ip route 1.1.1.1
Routing entry for 1.1.1.0/30 Known via “ospf 1“, distance 110, metric 20, type extern 2, forward metric 1
Last update from 192.168.13.1 on FastEthernet0/0, 00:00:55 ago
Routing Descriptor Blocks:
* 192.168.13.1, from 192.168.255.1, 00:00:55 ago, via FastEthernet0/0
Route metric is 20, traffic share count is 1

R3#
R3#sh ip bgp
BGP table version is 470, local router ID is 192.168.255.3
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete

Network              Next Hop   Metric   LocPrf    Weight    Path
*>i1.1.1.11/32    1.1.1.1          0       100           0       100 ?
*>i1.1.1.12/32    1.1.1.1          0       100           0       100 ?
*>i1.1.1.13/32    1.1.1.1          0       100           0       100 ?
*>i1.1.1.14/32    1.1.1.1          0       100           0       100 ?

R3#sh ip route bgp
   1.0.0.0/8 is variably subnetted, 235 subnets, 3 masks
B     1.2.11.0/24 [200/0] via 1.1.1.1, 00:01:06
B     1.2.10.0/24 [200/0] via 1.1.1.1, 00:01:06
B     1.1.10.0/24 [200/0] via 1.1.1.1, 00:01:06
B     1.1.1.11/32 [200/0] via 1.1.1.1, 00:01:06

  • On R1, change the next-hop to itself for all the prefixes advertised to iBGP peers (R3 in our case now):
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#router bgp 123
R1(config-router)#neigh 192.168.255.3 next-hop-self
R1(config-router)#end
R1#
*Mar 1 01:51:36.567: %SYS-5-CONFIG_I: Configured from console by console
R1#clear ip bgp * soft out
R1#——————————————————————————
R3#sh ip route 1.1.1.1
% Subnet not in table
R3#
R3#sh ip bgp
BGP table version is 938, local router ID is 192.168.255.3
Status codes: s suppressed, d damped, h history, * valid, > best, i – internal,
r RIB-failure, S Stale
Origin codes: i – IGP, e – EGP, ? – incomplete

Network               Next Hop      Metric LocPrf  Weight    Path
*>i1.1.1.11/32    192.168.255.1    0     100     0           100 ?
*>i1.1.1.12/32    192.168.255.1    0     100     0           100 ?

R3#sh ip route bgp
1.0.0.0/8 is variably subnetted, 234 subnets, 2 masks
B     1.2.11.0/24 [200/0] via 192.168.255.1, 00:01:03
B     1.2.10.0/24 [200/0] via 192.168.255.1, 00:01:03

Notice that the next hop has changed now to 192.168.255.1 and BGP prefixes are successfully installed into the routing table.

1.4. iBGP Full Mesh Requirement

At this moment, we have the built two iBGP sessions: R1  R2 and R1  R3, as shown below:

Let’s now build another iBGP session, between R3 and R4, in an attempt to make R4 learn the routes from R1 (via R3):

R3#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 192.168.255.4 remote-as 123
neighbor 192.168.255.1 remote-as 123
neighbor 192.168.255.1 update-source Loopback0
no auto-summary
R4#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 192.168.255.3 remote-as 123
no auto-summary

At this moment, R4 peers with R3 but it does not receive any prefixes:

R4#sh ip bgp summary
BGP router identifier 192.168.255.4, local AS number 123
BGP table version is 1, main routing table version 1

Neighbor            V   AS   MsgRcvd   MsgSent   TblVer   InQ OutQ   Up/Down   State/PfxRcd
192.168.255.3    4   123          20            21         1       0       0     00:04:31         0
R4#
R4#sh ip bgp
R4#
———————————– on R3———————————————–
R3#sh ip bgp summary

Neighbor           V   AS   MsgRcvd MsgSent   TblVer   InQ OutQ   Up/Down   State/PfxRc
192.168.255.4   4   123       24         23         235       0       0      00:07:33        0
192.168.255.1   4   123       20         17         235       0       0      00:13:56       234
R3#
R3#
R3#sh ip bgp neighbor 192.168.255.4 advertised-routes

Total number of prefixes 0
R3#

The rule of thumb for internal BGP is that iBGP prefixes are not advertised to other iBGP peers. In this case, R3 has only iBGP prefixes (all learned from R1) and it will not advertise them to R4. Here is some data collected from R3: Out of 234 path entries, all of them (234) are internal, so none of them will be sent to R4:

R3#sh ip bgp summary
BGP router identifier 192.168.255.3, local AS number 123
BGP table version is 235, main routing table version 235
234 network entries using 27378 bytes of memory
234 path entries using 12168 bytes of memory
2/1 BGP path/bestpath attribute entries using 248 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memoryBGP using 39818 total bytes of memory

BGP activity 234/0 prefixes, 234/0 paths, scan interval 60 secs

Neighbor          V   AS   MsgRcvd   MsgSent   TblVer   InQ OutQ   Up/Down    State/PfxRcd
192.168.255.4  4   123      26             25        235       0      0       00:09:58        0
192.168.255.1  4   123      23             20        235       0      0       00:16:22       234
R3#
R3#sh ip route summary
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths is 16
Route Source   Networks   Subnets   Overhead   Memory (bytes)
connected        3              1             288            544
static               0              0             0               0
ospf 1              1              3             504            544
Intra-area: 4 Inter-area: 0 External-1: 0 External-2: 0
NSSA External-1: 0 NSSA External-2: 0
bgp 123           0              234         16848         31824
External: 0 Internal: 234 Local: 0
internal           2                                               2312
Total              6                238         17640         35224
Removing Queue Size 0
R3#

To fix this problem, we have several options:

  • Use features such as route reflection or confederations. These methods are out of the scope of this article.
  • Configure full mesh of iBGP between all internal BGP speakers (R1, R2, R3, R4).

The fastest way of configuring full mesh is to use BGP groups, by applying this configuration to all routers:

router bgp 123
neighbor IBGP peer-group
neighbor IBGP remote-as 123

neighbor IBGP update-source Loopback0

neighbor 192.168.255.1 peer-group IBGP

neighbor 192.168.255.2 peer-group IBGP

neighbor 192.168.255.3 peer-group IBGP

neighbor 192.168.255.4 peer-group IBGP
end

Note: after you apply the above snippet on R1 and R2, you will have to solve again the next-hop issues described in Section 1.3.

1.5. Peering between Loopbacks

Internal BGP peers are not necessarily directly connected. As you saw in the previous scenario, R1 and R4 are more than one hop away.

Even for the peering between R3 and R4, although they are directly connected, since there are multiple links from one to the other, it’s again a good practice to peer using loopback interfaces, as this will ensure redundancy in case one of the physical links fails.

An important note here is the fact that, by default, iBGP packets are generated with a TTL of 255, which allows internal BGP peers to establish a session even when they are multiple hops away.

1.6. Become a Transit AS for Others

An interesting situation when customer is multi-homed to different providers is the danger of becoming a transit network. For example, traffic originating from behind ISP-1 and destined to addresses behind ISP-2 will go via your company’s network, R1 and R2 in our scenario. Of course, this causes slowness or, even worse, your external uplinks could melt down.

The snippet below shows that ISP-1 has prefixes originating in ISP-2 (AS 200) received via AS 123:

ISP-1#sh ip bgp s | b Neighbor
Neighbor   V   AS    MsgRcvd  MsgSent   TblVer   InQ OutQ   Up/Down   State/PfxRcd
1.1.1.2      4   123        13          14          371     0      0       00:09:46      136
ISP-1#
ISP-1#sh ip bgp | i Next|2.2.2.
Network              Next Hop      Metric LocPrf Weight Path
*> 2.2.2.11/32    1.1.1.2                             0 123 200 ?
*> 2.2.2.12/32    1.1.1.2                             0 123 200 ?
*> 2.2.2.13/32    1.1.1.2                             0 123 200 ?ISP-1#
ISP-1#ping 2.2.2.11 source lo1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.11, timeout is 2 seconds:
Packet sent with a source address of 1.1.1.11
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 64/92/120 ms
ISP-1#
ISP-1#traceroute 2.2.2.11 source lo1

Type escape sequence to abort.
Tracing the route to 2.2.2.11

1 1.1.1.2 32 msec 40 msec 20 msec
2 192.168.13.3 32 msec 28 msec 12 msec
3 192.168.134.4 60 msec 44 msec 52 msec
4 192.168.24.2 80 msec 44 msec 60 msec
5 2.2.2.1 116 msec * 112 msec
ISP-1#

At this moment , traffic between providers ISP-1 and ISP-2 goes via your company’s network. Of course, ISP-1 and ISP-2 have connectivity to each other over the Internet as well, but the path via your company may be shorter, from the AS PATH perspective, or it may be preferred for other reasons.

To prevent situations like this, Internet providers usually configure route filtering on their side, but it’s always a good practice to do the same on your side: The only prefixes that R1 & R2 should advertise to ISP-1 and ISP-2 are the ones belonging to the company. Here is an example:

R1#
ip prefix-list COMPANY_PREFIXES seq 5 permit x.x.x.x/m
ip prefix-list COMPANY_PREFIXES seq 10 permit y.y.y.y/m
ip prefix-list COMPANY_PREFIXES seq 15 permit z.z.z.z/m
!
router bgp 123
neighbor 1.1.1.1 prefix-list COMPANY_PREFIXES out
R1#
R1#R1#sh ip bgp neigh 1.1.1.1 advertised-routes

Total number of prefixes 0
R1#
R1#

1.7. BGP Not Running on Transit Routers inside the AS

This scenario is the opposite of the previous one: your company is an ISP itself and you want to ensure that connectivity through your AS is working. To illustrate it, let’s kill the BGP process on R4, so the only iBGP speakers are R1, R2, and R3, with a full mesh of sessions between them.

Suppose that the link to ISP-1 is down and R1 needs to reach IPs behind ISP-2. Here is how the control plane looks like on R1:

R1#sh ip bgp summary

Neighbor           V   AS   MsgRcvd   MsgSent   TblVer   InQ OutQ   Up/Down   State/PfxRcd
1.1.1.1              4   100      124         124            0       0      0      00:06:02        324
192.168.255.2   4    123     118         127       1077       0      0       01:04:23       275
192.168.255.3   4    123     110         127       1077       0      0       01:04:30         0
R1#
R1#sh ip bgp 2.2.2.11
BGP routing table entry for 2.2.2.11/32, version 330
Paths: (1 available, best #1, table Default-IP-Routing-Table)
Not advertised to any peer
200
192.168.255.2 (metric 4) from 192.168.255.2 (192.168.255.2)
Origin incomplete, metric 0, localpref 100, valid, internal, best
R1#
R1#sh ip cef 2.2.2.11
2.2.2.11/32, version 307, epoch 0, cached adjacency 192.168.13.3
0 packets, 0 bytes
via 192.168.255.2, 0 dependencies, recursive
next hop 192.168.13.3, FastEthernet0/1 via 192.168.255.2/32
valid cached adjacency
R1#

But there is a problem with the data plane:

R1#ping 2.2.2.11

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 2.2.2.11, timeout is 2 seconds:
U.U.U
Success rate is 0 percent (0/5)
R1#
R1#traceroute 2.2.2.11

Type escape sequence to abort.
Tracing the route to 2.2.2.11

1 192.168.13.3 [AS 200] 28 msec 40 msec 24 msec
2 192.168.134.4 [AS 200] 52 msec 36 msec 60 msec
3 192.168.134.4 [AS 200] !H * !H
R1#

This shows another important aspect of BGP: recursive routing is used for the BGP next-hop address to identify the outgoing interface. This is best seen above: R1 learns about destination 2.2.2.11 from R2 with BGP next-hop 192.158.255.2 → the path toward 192.168.255.2 is via R3, R4, R2 and hence the problem: All these internal routers in the forwarding path need to know how/where to route destination 2.2.2.11. Since we stopped BGP on R4, such traffic will be blackholed there.

The best solution is to ensure that transit routers run BGP. Of course, in our scenario, we would re-enable BGP on R4.

1.8. Synchronization

Synchronization is not something that you would face in today’s network, but for completeness of this article, I will include its definition here.

The synchronization rule states that iBGP learned routes will not be installed into the routing table if they are not “synchronized” with IGP (also learned via IGP). This was relevant in the past, when not all routers had enough resources to run BGP and (as shown in above section 1.6) traffic would have been blackholed inside the AS on some transit router that don’t have BGP knowledge.

2. Outside the AS—eBGP

Building external BGP sessions requires proper configuration of devices that are in different autonomous systems (ASes), so they are under different administration, so the network engineers that manage each end will have to exchange some information, such as:

  • First and foremost, the AS number
  • MD5 password and other security features, if any
  • Routes that each peer is expected to send and receive on

From experience, I find that this communication is not always successful and situations like these end up with multiple hours of troubleshooting and a lot of phone calls back and forth… just to find out, in the end, that the AS number was wrong or the MD5 password did not match!

2.1. Incorrect AS number

Situations when the configured AS numbers do not match happen, most of the time, because of typos in configuration, but sometimes also because of miscommunication between the engineers. Here is the configuration of R1 and ISP-1:

R1#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes
neighbor 1.1.1.1 remote-as 10
no auto-summary
ISP-1#sh run | s router bgp
router bgp 100
no synchronization

bgp log-neighbor-changes

neighbor 1.1.1.2 remote-as 123
no auto-summary

This mistake is easily spotted because of the continuous logs indicating the mistake:

R1#
*Mar 1 01:56:33.719: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 2/2 (peer in wrong AS) 2 bytes 0064
R1#
*Mar 1 01:57:07.835: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 2/2 (peer in wrong AS) 2 bytes 0064
R1#
*Mar 1 01:57:38.699: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 2/2 (peer in wrong AS) 2 bytes 0064
R1#
*Mar 1 01:58:08.123: %BGP-3-NOTIFICATION: sent to neighbor 1.1.1.1 2/2 (peer in wrong AS) 2 bytes 0064

Of course, the solution is to correct the AS number for the peer under the BGP process: “neighbor 1.1.1.1 remote-as 100“.

2.2. MD5 Password Mismatch

In my opinion, the MD5 Password mismatch is the most common cause of unsuccessful external BGP sessions.

R1#sh run | s router bgp
router bgp 123
no synchronization

bgp log-neighbor-changes

neighbor 1.1.1.1 remote-as 100
neighbor 1.1.1.1 password 123456
no auto-summary
ISP-1#sh run | s router bgp
router bgp 100
no synchronization

bgp log-neighbor-changes

neighbor 1.1.1.2 remote-as 123
neighbor 1.1.1.2 password cisco123
no auto-summary

Again, debugging the BGP session would show the problem:

R1#
*Mar 1 02:06:34.003: %TCP-6-BADAUTH: Invalid MD5 digest from 1.1.1.1(25725) to 1.1.1.2(179)
R1#
*Mar 1 02:06:35.995: %TCP-6-BADAUTH: Invalid MD5 digest from 1.1.1.1(25725) to 1.1.1.2(179)
R1#
*Mar 1 02:06:40.007: %TCP-6-BADAUTH: Invalid MD5 digest from 1.1.1.1(25725) to 1.1.1.2(179)
R1#
*Mar 1 02:07:55.991: BGP: 1.1.1.1 open failed: Connection timed out; remote host not responding, open active delayed 31776ms (35000ms max, 28% jitter)

Solve this problem by correcting the MD5 password, in this case: “neighbor 1.1.1.1 password cisco123“.

2.3. Multihop

Another common mistake is that engineers forget to configure the multihop option. Let’s see an incomplete configuration on R1 in attempt to establish an external BGP with another router inside ISP-1:

R1#
router bgp 123
neighbor 57.57.57.7 remote-as 100
R7#
router bgp 100
neighbor 1.1.1.2 remote-as 100

By default, eBGP packets are sent with a TTL of 1, since most eBGPs are established between directly connected peers. Unfortunately the above configuration mistake is hard to troubleshoot if you don’t know what’s wrong, because debugs (bgp or packet) do not show anything! Cisco IOS recognizes the mistake and it does not attempt to establish the eBGP multihop peering, so this is why nothing is seen in the debugs.

The only evidence of the mistake is the output of “show ip bgp neighbor” command:

R7#sh ip bgp neigh 1.1.1.2
BGP neighbor is 1.1.1.2, remote AS 123, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle
Last read 00:00:00, last write 00:00:00, hold time is 180, keepalive interval is 60 seconds

Message statistics:

InQ depth is 0

OutQ depth is 0

Sent Rcvd
Opens:                0      0
Notifications:        0      0
Updates:              0      0
Keepalives:          0      0
Route Refresh:     0      0
Total:                  0      0
Connections established 0; dropped 0
Last reset never
External BGP neighbor not directly connected.
No active TCP connection
R7#

The solution is to tell the router how may hops are between the eBGP peers; usually people configure the maximum value of 255, especially during lab testing or exams:

R1#
router bgp 123
neighbor 57.57.57.7 remote-as 100
neighbor 57.57.57.7 ebgp-multihop 2
R7#
router bgp 100

neighbor 1.1.1.2 remote-as 100
neighbor 1.1.1.2 ebgp-multihop 2

Pay attention if this problem appears also when eBGP peers uses their loopbacks to establish the BGP session. Even though they may be directly connected, you will still have to use the “ebgp-multihop” option.

2.5. TLL-Security

TTL-Security is a security feature that is used to protect external BGP sessions.

Let’s imagine the following situation: Someone in the Internet has found that your router R1 runs eBGP on the external interface (fa0/0) and also knows your external IP address and the AS number.

With this information in hand, that person can try to establish BGP sessions with R1 with the intent of crashing your device (a denial-of-service attack).

R8#sh run int lo0
Building configuration…

Current configuration : 66 bytes
!
interface Loopback0
ip address 57.57.57.7 255.255.255.255
end

R8#sh run | s router bgp
router bgp 100
no synchronization
bgp log-neighbor-changes
neighbor 1.1.1.2 remote-as 123
neighbor 1.1.1.2 ebgp-multihop 3
neighbor 1.1.1.2 update-source Loopback0
no auto-summary
R8#

In this case study, R8 (the attacker) sends TCP connection attempts (SYN) for BGP (tcp port 179) toward R1, spoofing an R7 address (57.57.57.7). The victim, R1, passes all these packets to its CPU and sends back TCP RST (RESET). These resets are sent to R7 (because the attacker R8 uses same IP on its loopback):

—— debugs from attacker, R8——
R8#
*Mar 1 01:33:19.115: IP: tableid=0, s=57.57.57.7 (local), d=1.1.1.2 (FastEthernet0/0), routed via FIB
*Mar 1 01:33:19.115: IP: s=57.57.57.7 (local), d=1.1.1.2 (FastEthernet0/0), len 44, sending
R8#
*Mar 1 01:33:23.115: IP: tableid=0, s=57.57.57.7 (local), d=1.1.1.2 (FastEthernet0/0), routed via FIB
*Mar 1 01:33:23.115: IP: s=57.57.57.7 (local), d=1.1.1.2 (FastEthernet0/0), len 44, sending

—— debugs from attacker, R1——
R1#
*Mar 1 01:40:43.103: IP: s=57.57.57.7 (FastEthernet0/0), d=1.1.1.2, len 44, rcvd 0
*Mar 1 01:40:43.107: TCP src=34609, dst=179, seq=1209774289, ack=0, win=16384 SYN
*Mar 1 01:40:43.111: IP: tableid=0, s=1.1.1.2 (local), d=57.57.57.7 (FastEthernet0/0), routed via FIB
*Mar 1 01:40:43.115: IP: s=1.1.1.2 (local), d=57.57.57.7 (FastEthernet0/0), len 40, sending
*Mar 1 01:40:43.119: TCP src=179, dst=34609, seq=0, ack=1209774290, win=0 ACK RST

—— debugs from attacker, R1——
R7#
*Mar 1 01:40:33.079: IP: tableid=0, s=1.1.1.2 (FastEthernet0/0), d=57.57.57.7 (FastEthernet0/0), routed via RIB
*Mar 1 01:40:33.079: IP: s=1.1.1.2 (FastEthernet0/0), d=57.57.57.7 (FastEthernet0/0), len 40, rcvd 3
*Mar 1 01:40:33.083: TCP src=179, dst=34609, seq=0, ack=1209774290, win=0 ACK RST

At this moment, using the “ebgp-multihop” command, the incoming TTL can have any value.

In order to protect against similar attacks, you should configure R1 to accept only requests that have a TTL value of 254. This is achieved by configuring the ttl-security feature for that neighbor:

R1#
router bgp 123
neighbor 57.57.57.7 remote-as 100
neighbor 57.57.57.7 ttl-security hops 2
R7#
router bgp 100
neighbor 1.1.1.2 remote-as 100
neighbor 1.1.1.2 ttl-security hops 2

This way, the eBGP between R1 and R7 gets this additional layer of security, because an attacker (such R8) that sits farther away from R1 would not be able to send packets with such high TTL value.

After this protection is activated, debugs show that R1 still receives the attack (spoofed SYN packets) but this time it does not pass it to the CPU and it drops it immediately (so, no RST is sent back):

R1#
*Mar  1 01:53:04.855: IP: s=57.57.57.7 (FastEthernet0/0), d=1.1.1.2, len 44, rcvd 0
*Mar  1 01:53:04.855:     TCP src=15554, dst=179, seq=3929529580, ack=0, win=16384 SYN
R1#
R1#
*Mar  1 01:53:08.859: IP: s=57.57.57.7 (FastEthernet0/0), d=1.1.1.2, len 44, rcvd 0
*Mar  1 01:53:08.863:     TCP src=15554, dst=179, seq=3929529580, ack=0, win=16384 SYN
R1#

You need to remember that “neighbor ttl-security hops” and “neighbor ebgp-multihop” commands are mutually exclusive, so you cannot configure them at the same time.

This concludes our “Beginner’s Guide to Troubleshooting BGP” and opens the way for a new article, in which more complex scenarios will be presented.