At the aggregation layer of a regional science and technology network, a broadband access server was installed. Due to the high volume of user access connections, the network was designed with dual uplinks connecting the broadband access server to two core switches to ensure connection stability. These dual links use gigabit fiber optic connections, and the network employs static routing for path selection. During the initial network construction, considering the heavy traffic load under normal conditions, the two gigabit fiber links were specifically chosen to achieve load balancing. The design ensured that if one fiber link failed or its upstream switch experienced a fault, all broadband access traffic would automatically failover to the other link for redundant path protection, guaranteeing uninterrupted internet access for users. After the construction was completed, the debugging results met the established requirements.
Initial Encounter with Failed Automatic Failover
Recently, many users accessing the science and technology network through this broadband access server reported that internet speeds were significantly slower than before. Even opening a simple webpage without images or multimedia content took a long time. Using the ping command to test target website addresses revealed severe packet loss, sometimes reaching 50%. Clearly, a major fault existed in the broadband access network. To quickly diagnose the issue, the network administrator remotely logged into the backend management interface of the core switch using Telnet. The inspection revealed that one of the uplink gigabit fiber links from the broadband access server to the core router was down. However, the network route did not automatically switch from the faulty fiber link to the functioning one, which explained the 50% packet loss in broadband access.
The Mystery of the Failed Automatic Failover
To quickly restore normal internet access for broadband users, the network administrator first adjusted the routing parameters on the core devices, appropriately lowering the static route priority associated with the faulty communication link. This ensured all network routes used the normally operating fiber link. After completing the parameter modification, broadband users could indeed access the internet normally again.
Later, the network administrator simulated the actual working environment of the science and technology network by setting up a switch and implementing dual-line uplinks using spare fiber links. A static route was configured on each of the two core switches pointing to this newly set up switch. Simultaneously, the state of the interconnected port on the switch connected to the faulty fiber link was set to “shutdown”. At this point, upon checking the corresponding routing status on the newly set up switch, the network administrator found it was also in a down state. However, on the core switch connected to the faulty fiber link, although the virtual work subnet under the corresponding switch interface port was down, the static route entry pointing to the newly set up switch still existed.
After a detailed examination of the configuration parameters on the core switch, the network administrator discovered a significant difference between the core switch used in the local science and technology network and those in other regions. Following network adjustments between the provincial science and technology network and the provincial telecom network, the local core switch was directly connected to a high-end router in the local telecom network. This core switch was uplinked to a router running the BGP protocol, which imported routes from the local routing table via the network. However, when trying to import local route entries via the network, they had to pre-exist in the routing table; otherwise, the import would fail. But in reality, only specific route entries existed in the routing table, without any aggregate routes. Therefore, the network administrator deliberately configured a static route pointing to the null0 interface to “trick” the Border Gateway Protocol, using the network configuration to allow the BGP router to successfully import the aggregate route.
At this point, the fault was fully revealed. When a fiber link at the aggregation layer of the science and technology network suddenly disconnected, the static route pointing to null0 in the BGP router remained in an active/up state. Since static routes have a higher administrative distance priority than other routes, a static route pointing to null0 persistently existed on the core switch connected to the faulty fiber link. Crucially, the earlier testing for the broadband access renovation project was conducted before the network adjustments between the provincial science and technology network and the provincial telecom network, when there was no static route pointing to null0. Therefore, the test results appeared normal at that time.
Considering that static routes have the highest priority among all route types, the network administrator immediately modified the routing parameters on the core switch, slightly lowering the administrative distance priority of the static route pointing to null0. This way, when a fiber link at the aggregation layer disconnected, the core switch associated with that link could automatically learn the route from the other core switch via the OSPF routing protocol. After completing the adjustment to the priority of the static route pointing to null0, the network administrator ran another test. This time, the redundant route performed the automatic failover successfully, indicating that the fault described above had been smoothly resolved.
Final Summary
During the routing configuration process for vertical networks between similar science and technology systems, mutual interference between network configurations can easily occur. These effects might not be immediately visible during a certain period of network operation. This serves as a reminder to network administrators: whenever adjustments are made within the network, we should promptly test key network access projects to ensure hidden problems are exposed to the greatest extent possible, allowing for targeted investigation and resolution.
銆怑ditor’s Picks銆?/strong>