[TOC]

Container network and log selection and implementation

Network Selection (K8S and MESOS)

Thinking && pain points

  1. Can I access it across machines? Cross-domain access?

    • Flannel can communicate across containers
    • Interconnect containers across hosts
    • The container is connected to the outside

  2. Do you support static OR fixed IP addresses? Domain access?

    • With a fixed IP, you need the IP to remain the same every time you deploy or update or reboot
    • Overlay Network, Docker 1.6 can communicate across hosts

  3. Is DNS supported?

  4. Layer 4 / layer 7 access

  5. Network after container storage capacity

  6. IP port, it is best not to manually plan

  7. Network policy, defense, isolation?

    • Network isolation and traffic limiting between different applications in a container cluster

plan

  1. Scheme category

    • Tunnel scheme, using tunnels, or Overlay Networking:
      • Weave, UDP broadcast, local set up new BR, interconnect via PCAP.
      • Open vSwitch (OVS) is based on VxLAN and GRE protocols, but suffers serious performance loss.
      • Flannel, UDP broadcast, VxLan
    • Routing scheme
      • Calico, a routing scheme based on BGP, supports detailed ACL control and has a high affinity for hybrid cloud.
      • Macvlan is the solution with the best isolation and performance from the logic and Kernel layers. It is based on layer 2 isolation and requires the support of layer 2 routers. Most cloud service providers do not support it, so it is difficult to implement on hybrid cloud.
      • Good performance, no NAT, high efficiency, but limited by the routing table, in addition, each container has an IP, service IP may be used up.

  2. There are two camps on the web

    • Docker Libnetwork Container Network Model (CNM) camp (Docker Libnetwork advantage is native, and closely combined with the Docker Container lifecycle)

      • Docker Swarm overlay
      • Macvlan & IP network drivers
      • Calico
      • Contiv (from Cisco)

    • Container Network Interface (CNI) camp (CNI has the advantage of being compatible with other Container technologies (e.g. RKT) and upper-layer programming systems (Kuberneres & Mesos)

      • Kubernetes
      • Weave
      • Macvlan
      • Flannel
      • Calico
      • Contiv
      • Mesos CNI

  3. Common solutions are:

    • Flannel VXLAN, Overlay mode
    • Calico: Layer-3 isolation, cross-subnet deployment. If the gateway does not support BGP, use overlay mode of the IPIP tunnel
    • Ipvlan MacVLAN: Physical Layer 2 / Layer 3 isolation. Currently, it needs to be configured on a single node using the Pipework tool. Only VLAN isolation is implemented, but ARP broadcast is not resolved
    • Swarm native VXLAN is similar to flannel VXLAN
    • Neutron SDN: ml2+ Ovsplugin, MIDONET, VLAN or VXLAN
    • Contiv, Cisco led, SDN solution, can use either pure soft OVS or OVS + Cisco hardware SDN Controller
    • Linux Bridge + Layer-3 switch: Set the Linux Bridge on host to the subnetwork segment of layer-3 switches. Containers communicate with each other through Layer-2 switches, and containers communicate with external layer-3 switches through the gateway.

  4. Network selection is commonly used in the industry

    • kubernetes + flannel

      • Flannel supports data forwarding modes such as UDP, VxLAN, AWS VPC, and GCE routing.
      • Under Kubernetes, Flannel, OpenVSwitch and Weave can implement Overlay Network
      • Vipshop Contiv NetPlugin solution (fixed external IP) + Flannel
      • Jd Flannel + Neutron + OVS

    • Mesos + Calico

      • Mesos supports the CNI standard specification
      • One container one IP, network isolation, IP allocation, L3 virtual network
      • Where to Mesos + Calico
      • Bridge+ NAT + Open vSwitch

    • Major users of contiv can access contiv directly based on the instance IP

    • Meizu cloud OVS & VLAN + SR-IOV

    • Ucloud: VSwitch Overlay “Large Layer 2” SDN networking solution + ipvLAN

Log monitoring selection (including monitoring, statistics)

Due to the hierarchical design mode of Docker, the data in the container cannot be solidified, and the data in the container will be lost when the container is destroyed. Therefore, the log needs to be mounted to the host or distributed storage such as CEPh is used

  1. Monitoring Optional solution

    • cAdvisor + InfluxDB + Grafana
    • cAdvisor + Prometheus + Grafana
    • Graphite
    • Zabbix
    • Datadog

  2. This option is optional

    • logstash
    • ELK
    • Graylog
    • flume
    • heka
    • fluentd

  3. The industry solution

    • Aliyun: cAdvisor + InfuxDB + Prometheus
    • Coroutines: ELK
    • Graphite + cAdvisor