1. Introduction of background and transformation scheme
1.1 Introduction to Qunar K8S Network
K8S Network Description:
- The ACCESS switch runs IBGP with the K8S NODE. The K8S NODE advertises routes in the POD network segment to the access switch through BGP.
- EBGP runs between the access switch and the core switch. RACK advertises the LEARNED POD network segment routes to the core switch through BGP.
- The core switch advertises default routes only to each access switch through routing policies.
1.2 Introduction to Cisco Nexus VPC
A VPC is short for Virtual port-channel. A VPC allows interfaces on two switches to form aggregation ports. The two switches are viewed as one device by the peer device of the aggregation Port.
A VPC provides the following benefits:
- Two switches are virtualized as one at the L2 interface level to eliminate STP loops connected to L2 and improve network stability.
- Combining VPC with HSRP, both HSRP switches can forward data packets using HSRP VIP as the gateway at Layer 3, increasing outbound bandwidth and bandwidth usage of upstream interfaces.
- The two switches in a VPC are independent at the control layer and the L3 layer, facilitating system maintenance and routing operations.
1.3 Why Is A VPC Modified to Enable Mixed Layer 2 and Layer 3 Network Deployment
- The original L2 access switch cannot run BGP with the K8S NODE server.
- VPC + HSRP Enables both HSRP switches to forward data packets using HSRP VIP as the gateway at Layer 3, increasing outbound bandwidth and bandwidth usage of upstream interfaces.
- The access network of L2 is gradually transferred to the access network of L3. The stability of L3 network is better than that of L2 network. After the transition, the IDC network is more stable.
- After the renovation, the existing cabinets can be used for K8S server deployment, reducing the number of new cabinets and saving the operation cost of the equipment room.
1.4 Changes of network structure before and after renovation
Before transformation:
- Before the transformation, access switch 1 and access switch 2 serve as two independent L2 access switches and connect to a group of VPC core switches through L2 port-channel.
- Access switches 1 and 2 work as a group to provide dual-uplink access for the Server. The Server connects to the two L2 access switches through bond.
- The gateway of the Server is on the core switch.
After renovation:
-
Configure two independent L2 access switches as one VPC. \
-
Add L3 interconnection between two access devices and two cores to establish EBGP neighbors.
-
The GATEWAY of the K8S server is on the access switch. The K8S server establishes IBGP neighbor relationships with the two access switches.
1.5 Ideas of transformation
- Cut off the traffic of the switch that needs to be modified;
- Configure a VIRTUAL private cloud (VPC) on a switch without traffic, restart the switch, and use the AUTO-recovery feature of the VPC to make the VPC take effect.
- Switch the server traffic to the transformed switch.
- Configure the VPC on the other switch and connect the TWO access switches to the VPC peer-link interface.
- Add L3 interconnection between access switches and core switches and configure BGP.
- On the access switch, configure the VLAN interface and BGP between the server and the access switch.
2. Specific transformation steps
2.1 Planning Active/Standby VPC Roles for Two L2 access Switches
You can configure priorities to control role allocation between active and standby VPCS.
2.2 Maintaining a VPC master switch
Cut off traffic from the switch’s uplink server and close its uplink port on the core switch.
2.3 Disabling the Port connecting to the Server on the Master Switch
This operation prevents the bond interface of the server from being accidentally switched back during maintenance.
Record the information about the interfaces in UP state in advance and check whether any interface is not recovered after the switch restarts.
2.4 Configuring a VPC for the Active Switch
In this case, you do not need to connect the VPC peer-link line and keep the VPC peer-link interface shutdown after it is configured.
“**Rack1`
feature vpc
vpc domain XX
``peer-``switch
``role priority ``4096
``peer-keepalive destination x.x.x.x source y.y.y.y
``delay restore ``150
``auto-recovery
``ip arp synchronize
interface port-channel4002
``switchport
``switchport mode trunk
``spanning-tree port type network
``vpc peer-link
interface Ethernet1/``51
``switchport
``switchport mode trunk
``channel-group ``4002 mode active
``shutdown
interface Ethernet1/``52
``switchport
``switchport mode trunk
``channel-group ``4002 mode active
` shutdown
2.5 Save the configuration and restart the master switch
The VPC takes effect after the switch is restarted.
- Delay-restroe SVI status The timer starts to count down 10 seconds.
- Delay-restroe SVI status After the countdown ends, the delay-restore status timer starts to count down for 150s.
- Delay-restore Status After the countdown ends, the auto-recovery status timer starts to count down for 240s. \
After three timers for 400 seconds, the VPC enters the Primary state.
2.6 Configuring a VPC ID for port-channel and Enabling uplink Ports on the core switch (STP Jitter once)
**Rack1`
conf t
interface port-channel1
``vpc ``1
``switchport trunk allowed vlan except xx
**core[``1``-``2``]
conf t
interface port-channel9
``switchport trunk allowed vlan except xx
Check the trunk allowed VLAN configuration on the interface
# open interface:
**core[``1``-``2``]
conf t
interface Ethernet1/``9
``no shutdown
# check Rack1 port-channel and VPC ‘1’ status, check STP
- Upper interface port-channel1 VPC ID 1 The status is normal
2.7 Switching Server Traffic back to the VPC primary Switch
After L2 on the active SWITCH of the VPC is restored, enable the ports connecting to the server, check whether the ports in UP state are the same before and after the restart, and switch the server traffic from the standby switch to the active switch.
2.8 Disabling the uplink port on the standby Switch on the Core Switch
2.9 Configuring a VPC for a Standby Switch
The SAME VPC ID must be configured for the port-channel on the standby switch and the port-channel on the active switch to prevent inconsistent VPC consistency checks from affecting the port-channel status on the active switch.
“**Rack2`
feature vpc
vpc domain XX
``peer-``switch
``peer-keepalive destination y.y.y.y source x.x.x.x
``delay restore ``150
``auto-recovery
``ip arp synchronize
interface port-channel4002
``switchport
``switchport mode trunk
``spanning-tree port type network
``vpc peer-link
interface Ethernet1/``51
``switchport
``switchport mode trunk
``channel-group ``4002 mode active
``no shutdown
interface Ethernet1/``52
``switchport
``switchport mode trunk
``channel-group ``4002 mode active
``no shutdown
interface port-channel1
``vpc ``1
` switchport trunk allowed vlan except xx
- The VPC uses the management port IP address as the keep-alive address. The VPC keep-alive status is peer is alive because the management IP addresses can communicate with each other.
- Because the peer-link line is not connected, the peer status is Peer Link is Down, and the vPC role is None Established.
2.10 Connecting to the Peer-Link and Enabling the Interface (STP Jitter Once)
- After VPC peer-link is enabled, the VPC role of the primary switch changes to Primary.
- The VPC role of the standby switch is secondary.
- The standby switch enters the normal state after the preceding 400 seconds, and the VPC between Rack1 and Rack2 is established
2.11 Adding the interface connecting to Rack2 to the Port-channel connecting to Rack1 on the core switch
- Rack1&Rack2 interconnects with Core1 & Core2 via an L2 port-channel.
**core[1-2“]`
conf t
interface Ethernet1/``35
``switchport trunk allowed vlan except xx
‘ ‘channel-group’ ‘9 mode active # channel group’ ‘9 is the channel group ID of the upper interface of the primary switch, so that the primary and secondary switches are connected to the same VPC
2.12 Opening the Uplink port on the standby Switch The VPC is transformed
2.13 Adding L3 Interconnection and EBGP configuration between access Switches and core switches
2.14 Adding IBGP Configurations for the Access Switch and K8S Server
3. Summary and matters needing attention
-
During the transformation, ensure that VPC Domain ids of multiple access switches do not conflict.
-
In the process of transformation, L2 network will undergo spanning tree jitter twice.
-
Before the transformation, the two access switches are connected to two different port-channels of the core switch. After the transformation, the two access switches are connected to the same port-channel of the core switch.
-
After a VPC is configured on the first access switch, the VPC does not take effect. You need to enable the VPC auto-recovery function and restart the switch for the VPC to take effect.