BGP for Multi Site Fiber The Non Nerd Guide to Better Failover

BGP for Multi Site Fiber The Non Nerd Guide to Better Failover

If you run more than a couple of sites, “failover” usually becomes two different problems: outbound traffic (your users reaching the internet and cloud) and inbound traffic (customers and partners reaching you). Most small networks only solve outbound. BGP is the tool that can solve inbound and outbound at the same time, but only when you actually need that level of control and you are ready for the operational responsibility. BGP itself is the internet’s interdomain routing protocol.

A plain-English description of BGP
BGP is the language networks use to tell other networks, “If you want to reach these IP addresses, send traffic to me.” It runs between autonomous systems and carries routes as announcements and withdrawals, which lets the internet shift traffic when paths change.
The outcome BGP is best at
Inbound plus outbound failover
This is the gap most dual-WAN and many SD-WAN setups do not fully solve for public services.

Two problems that get mixed up in meetings

  • Outbound failover is your users and systems getting out to the internet and cloud when a circuit fails.
  • Inbound failover is everyone else reaching your public IPs when one provider fails.
BGP is usually considered when inbound matters, or when you need policy control that simple failover cannot provide.

The three levels of failover, from simplest to most capable

1️⃣ Dual WAN without BGP

Great for outbound continuity. Common at branches. Fast to deploy. Often the best first step.
Limitation
Inbound traffic to your public services may still break or take a long time to shift if a provider fails.

2️⃣ SD-WAN or policy-based routing

Strong outbound control and brownout steering. Excellent for multi-site standardization. Still not the same as controlling how the internet reaches your public IPs.
Best fit
Most multi-site businesses that do not host critical public services.

3️⃣ BGP multihoming

You connect to two providers and use BGP to advertise your IP prefixes so the internet can route to you through either one. You can also influence which path is preferred under normal conditions. BGP is the interdomain routing protocol described in the standard specification.
Tradeoff
More operational responsibility, more coordination with providers, and more “rules of the road” around routing security and filtering.

The minimum concept set that makes BGP feel simple

Term Non-nerd meaning Why it matters for failover
ASN Your network’s identity on the internet Needed for BGP with upstream providers
Prefix The block of public IPs you want reachable Failover works when the same prefix is reachable via either provider
Announcement “Send traffic for this prefix to me” When a link fails, routes are withdrawn and the internet shifts
Path selection How a router chooses which route to use Controls which provider is preferred when both are up
BFD A fast heartbeat between routers Detects failures quickly so BGP can react faster than default timers
The failover speed reality
BGP can fail over quickly inside your edge when a link drops, but internet-wide convergence is not instant. If you need sub-second or “always on” behavior for specific applications, pair BGP with app-layer and DNS-layer resilience patterns.

When BGP is actually worth it

Strong BGP candidates
  • You host public services and inbound reachability matters: VPN endpoints, APIs, portals, VoIP SBCs, public web properties
  • You require predictable inbound routing and cannot tolerate long inbound outages during a provider incident
  • You want to avoid a hard dependency on one carrier for your public IP identity
  • You have staff or a partner who can operate routing policy safely

The usual blockers that surprise teams

🧩 Public IP space and authorization

Providers typically require proof you are allowed to announce the prefixes you want routed. Industry practice often uses routing registries and sometimes LOA documentation, and routing security tooling like RPKI origin validation is widely discussed as a safer direction.
Why this matters
Without proper authorization and registry hygiene, your routes may be filtered and never propagate correctly.

🔒 Routing security and filtering expectations

Many networks build filters from Internet Routing Registries, and origin validation concepts like those described in RFC 6811 exist to reduce route leaks and mis-announcements. If your prefixes are not represented correctly in the systems upstreams use, you may have unpredictable reachability.
Practical takeaway
“BGP is configured” is not the same as “the rest of the internet will accept my routes.”

The levers you can safely explain to leadership

Lever What it does in plain language Typical use Risk if misused
Local preference Choose your preferred outbound path inside your network Primary ISP for most traffic, secondary as backup Traffic hairpins or overloads a link
AS path prepending Make a path look longer so others prefer the other ISP Steer inbound traffic away from a link Unpredictable results because the internet is policy-driven
Communities Tags you attach to routes so a provider applies a known policy Control propagation or preference inside a carrier network Wrong community can blackhole reachability
BFD Detect a failure fast Reduce failover time at the edge Too aggressive settings can flap sessions
A reality check on path selection
BGP path selection is policy-driven and implementations use a best-path decision process. Vendors document the common algorithm steps such as preferring certain attributes when multiple routes exist.

Interactive BGP readiness and value scorer

This tool helps decide whether BGP is likely to be the right investment, or whether SD-WAN and dual-WAN gets you 90 percent of the benefit with less operational risk.
Results
BGP fit score
0
Recommended direction
Implementation notes

Short procurement checklist for a clean BGP deployment

  • Two providers with BGP support and clear handoff requirements
  • Your ASN and your prefixes are documented and approved for announcement
  • Filtering and security hygiene including route registry alignment and origin validation expectations
  • Fast failure detection plan such as BFD if you need faster than default convergence
  • Documented preference policy for normal operations, plus a tested incident mode