Calico的ip池对象ipPool 解析

已邀请:

本文主要分析calico中的ipPool资源

关于calico的集群部署,可以参考文章:

https://www.jianshu.com/p/2f8d8b4d5296


一、环境介绍




  1. 物理环境介绍



























    主机名称 系统版本 IP
    master centos7.5 172.16.91.205
    slave1 centos7.5 172.16.91.206
    slave2 centos7.5 172.16.91.207



  2. 服务部署介绍























    服务名称 版本
    docker v17.03.2-ce
    etcd v3.2.22
    calico v2.6



二、calico支持的模式?



  • BGP模式

    • 路由规则直接使用物理机网卡作为路由器转发

    • 路由即纯bgp模式,理论上ipip模式的网络传输性能低于纯bgp模式



  • IPIP模式

    • 一种妥协的overlay机制,在宿主机创建1个”tunl0”虚拟端口

    • 通过tun10作为路由转发

    • 分为两种模式:

      • ipip always模式(纯ipip模式)

      • ipip cross-subnet模式(ipip-bgp混合模式),指“同子网内路由采用bgp,跨子网路由采用ipip”






三、如何管理ipPool的生命周期?


3.1 如何创建ipPool 资源对象?



  • 第一步:自定义yaml,如myPool.yaml

  • 第二步:通过calicoctl工具,进行资源创建

    • calicoctl create -f myPool.yaml




3.2 如何删除ipPool 资源对象?



  • 方法一:根据文件名删除

    • calicoctl delete -f myPool.yaml



  • 方法二:根据cidr删除

    • calicoctl delete ipPool 10.254.0.0/24




3.3 如何更新ipPool 资源对象?



  • 方法一:calicoctl apply -f myPool.yaml

  • 方法二:calicoctl replace -f myPool.yaml




    //upload-images.jianshu.io/upload_images/13868689-1db2d56759ef7485?imageMogr2/auto-orient/strip|imageView2/2/w/753/format/webp


    image




四、ipPool


4.1 默认ipPool





//upload-images.jianshu.io/upload_images/13868689-4a2224b4b90b139f?imageMogr2/auto-orient/strip|imageView2/2/w/630/format/webp


ipPool默认情况


4.2 删除默认ipPool资源(选做)



  1. 删除之前,可以先进行备份一份:
    calicoctl get ipp -yaml > ipPool-default.yaml



  2. 删除







    //upload-images.jianshu.io/upload_images/13868689-4032fc3e555441d0?imageMogr2/auto-orient/strip|imageView2/2/w/916/format/webp


    删除默认ipPool资源




4.3 BGP模式


4.3.1 创建bgp模式下的ipPool资源对象




  1. 自定义bgp模式的ipPool资源对象

    vim ipPool-bgp.yaml


    - apiVersion: v1
    kind: ipPool
    metadata:
    cidr: 192.168.0.0/16
    spec:
    nat-outgoing: true
    disabled: false



  2. 创建ipPool资源


    calicoctl create -f ipPool-bgp.yaml




    //upload-images.jianshu.io/upload_images/13868689-10fafe398bbbd60f?imageMogr2/auto-orient/strip|imageView2/2/w/749/format/webp


    image




4.3.2 查看物理机网卡信息



  1. 查看master节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-0f96cd4c63b3afbf?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp


    master节点-bgp模式-网卡信息



  2. 查看slave1节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-758e9b1da9356d4b?imageMogr2/auto-orient/strip|imageView2/2/w/1067/format/webp


    slave1节点-bgp模式-网卡信息



  3. 查看slave2节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-eeffa3ed14eb2263?imageMogr2/auto-orient/strip|imageView2/2/w/1022/format/webp


    slave2节点-bgp模式-网卡信息




4.3.3 查看物理机路由表信息



  1. 查看master节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-bdcc6cc4e9fb64fd?imageMogr2/auto-orient/strip|imageView2/2/w/1003/format/webp


    master节点-bgp模式-路由信息



  2. 查看slave1节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-03632bd7fee19f6e?imageMogr2/auto-orient/strip|imageView2/2/w/933/format/webp


    slave1节点-bgp模式-路由信息



  3. 查看slave2节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-0e7b3056e347e263?imageMogr2/auto-orient/strip|imageView2/2/w/950/format/webp


    slave2节点-bgp模式-路由信息




4.3.4 清理ipPool资源



calicoctl delete -f ipPool-bgp.yaml







//upload-images.jianshu.io/upload_images/13868689-3cc4f853ffdbc63f?imageMogr2/auto-orient/strip|imageView2/2/w/743/format/webp


清理资源ipPool



4.3.5 BGP模式总结:



  • BGP模式下,不会在服务器上创建tun10,

  • 路由转发功能是通过服务器的网卡对流量进行转发的


4.4 IPIP模式


4.4.1 如何设置成IPIP模式


通过显示的设置ipip属性; 

4.4.2 纯ipip模式


4.4.2.1 创建纯ipip模式下的ipPool资源对象



  1. 自定义纯ipip模式的ipPool资源对象

    vim ipPool-ipip-always.yaml
    apiVersion: v1
    kind: ipPool
    metadata:
    cidr: 10.254.0.0/24
    spec:
    ipip:
    enabled: true
    mode: always
    nat-outgoing: false
    #默认即使false, 也就是启用此ipPool
    disabled: false


  2. 创建ipPool资源
    calicoctl create -f ipPool-ipip-always.yaml




    //upload-images.jianshu.io/upload_images/13868689-98c82bc171438d93?imageMogr2/auto-orient/strip|imageView2/2/w/860/format/webp


    ipip-always模式




4.4.2.2 查看物理机网卡信息



  1. 查看master节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-70d4ba5ca4e45b3f?imageMogr2/auto-orient/strip|imageView2/2/w/1055/format/webp


    master节点-ipip-always模式-网卡信息



  2. 查看slave1节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-e9f0a42c8225b987?imageMogr2/auto-orient/strip|imageView2/2/w/991/format/webp


    slave1节点-ipip-always模式-网卡信息



  3. 查看slave2节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-e366d888a28d7865?imageMogr2/auto-orient/strip|imageView2/2/w/1049/format/webp


    slave2节点-ipip-always模式-网卡信息




4.4.2.3 查看物理机路由表信息



  1. 查看master节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-cd13a9e829dd8e50?imageMogr2/auto-orient/strip|imageView2/2/w/953/format/webp


    master节点-ipip-always模式-路由信息



  2. 查看slave1节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-82a03eb3631c89f4?imageMogr2/auto-orient/strip|imageView2/2/w/996/format/webp


    slave1节点-ipip-always模式-路由信息



  3. 查看slave2节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-23f5a0661289ee8b?imageMogr2/auto-orient/strip|imageView2/2/w/971/format/webp


    slave2节点-ipip-always模式-路由信息




4.4.2.4 清理ipPool资源


calicoctl delete -f ipPool-ipip-always.yaml







//upload-images.jianshu.io/upload_images/13868689-95392c77ecf974dd?imageMogr2/auto-orient/strip|imageView2/2/w/997/format/webp


清理ipip-always资源


4.4.2.5 总结:


ipip-always模式,



  • 会在每个节点上,都创建tun10,

  • 通过tun10进行流量转发


4.4.3 混合ipip-bgp模式


4.4.3.1 创建混合ipip-bgp模式下的ipPool资源对象




  1. 自定义混合ipip-bgp模式的ipPool资源对象

    vim ipPool-ipip-cross-subnet.yaml


    apiVersion: v1
    kind: ipPool
    metadata:
    cidr: 10.254.0.0/24
    spec:
    ipip:
    enabled: true
    mode: cross-subnet
    nat-outgoing: true
    #默认即使false, 也就是启用此ipPool
    disabled: false




  2. 创建ipPool资源


    calicoctl create -f ipPool-ipip-cross-subnet.yaml




    //upload-images.jianshu.io/upload_images/13868689-0a3a0a141f1316f9?imageMogr2/auto-orient/strip|imageView2/2/w/923/format/webp


    创建ipip-cross-subenet模式资源




4.4.3.2 查看物理机网卡信息



  1. 查看master节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-2fa598e159c2221e?imageMogr2/auto-orient/strip|imageView2/2/w/998/format/webp


    master节点-ipip-cross-subnet模式-网卡信息



  2. 查看slave1节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-bb874b90ae7bd3ef?imageMogr2/auto-orient/strip|imageView2/2/w/1002/format/webp


    slave1节点-ipip-cross-subnet模式-网卡信息



  3. 查看slave2节点的网卡信息




    //upload-images.jianshu.io/upload_images/13868689-0f435eabf97504d7?imageMogr2/auto-orient/strip|imageView2/2/w/1009/format/webp


    slave2节点-ipip-cross-subnet模式-网卡信息




4.4.3.3 查看物理机路由表信息



  1. 查看master节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-836d69e091ba6a2e?imageMogr2/auto-orient/strip|imageView2/2/w/1042/format/webp


    master节点-ipip-cross-subnet模式-路由信息



  2. 查看slave1节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-0c6494419004d438?imageMogr2/auto-orient/strip|imageView2/2/w/961/format/webp


    slave1节点-ipip-cross-subnet模式-路由信息



  3. 查看slave2节点的路由表信息




    //upload-images.jianshu.io/upload_images/13868689-0c99e6c7512edcf4?imageMogr2/auto-orient/strip|imageView2/2/w/988/format/webp


    slave2节点-ipip-cross-subnet模式-路由信息




4.4.3.4 清理ipPool资源


calicoctl delete -f ipPool-ipip-cross-subnet.yaml

4.4.4 总结


混合ipip-bgp模式 
- 有些节点创建tun10, 有些节点不创建tun10
- 通过tun10和网卡进行流量转发

4.5 总结整理:


1. BGP模式总结:  
- BGP模式下,不会在服务器上创建tun10,
- 路由转发功能是通过服务器的网卡对流量进行转发的

2. ipip-always模式,
- 会在每个节点上,都创建tun10,
- 通过tun10进行流量转发

3. 混合ipip-bgp模式
- 有些节点创建tun10, 有些节点不创建tun10
- 通过tun10和网卡进行流量转发
- 如果calico集群里,存在多个混合ipip-bgp模式资源对象的话,只有第一个创建的ipip-bgp资源对象有效,
后面创建一直处于阻塞状态, 经测试,只有删除第一个后,第二个会自动创建的;
- 也就是说,在calico集群中,同一时刻只能是第一个ipPool资源对象有效
4. libnetwork插件负责从ipPool池里,获取IP,指定给docker容器的

五、如果calico集群里,存在多个bgp模式的ipPool资源对象的话,哪个生效?


简单的测试了几次,没发现什么规律。

肯定是测试不够充分。

查看calico源码:


func (i IpamDriver) RequestPool(request *ipam.RequestPoolRequest) (*ipam.RequestPoolResponse, error) {
logutils.JSONMessage("RequestPool", request)

// Calico IPAM does not allow you to request a SubPool.
if request.SubPool != "" {
err := errors.New(
"Calico IPAM does not support sub pool configuration " +
"on 'docker create network'. Calico IP Pools " +
"should be configured first and IP assignment is " +
"from those pre-configured pools.",
)
log.Errorln(err)
return nil, err
}

if len(request.Options) != 0 {
err := errors.New("Arbitrary options are not supported")
log.Errorln(err)
return nil, err
}
var poolID string
var pool string
var gateway string
if request.V6 {
// Default the poolID to the fixed value.
poolID = i.poolIDV6
pool = "::/0"
gateway = "::/0"
} else {
// Default the poolID to the fixed value.
poolID = i.poolIDV4
pool = "0.0.0.0/0"
gateway = "0.0.0.0/0"
}

// If a pool (subnet on the CLI) is specified, it must match one of the
// preconfigured Calico pools.
if request.Pool != "" {
poolsClient := i.client.IPPools()
_, ipNet, err := caliconet.ParseCIDR(request.Pool)
if err != nil {
err := errors.New("Invalid CIDR")
log.Errorln(err)
return nil, err
}

pools, err := poolsClient.List(context.Background(), options.ListOptions{})
if err != nil {
log.Errorln(err)
return nil, err
}

f := false
for _, p := range pools.Items {
if p.Spec.CIDR == ipNet.String() {
f = true
pool = p.Spec.CIDR
poolID = p.Name
break
}
}

if !f {
err := errors.New("The requested subnet must match the CIDR of a " +
"configured Calico IP Pool.",
)
log.Errorln(err)
return nil, err
}
}

// We use static pool ID and CIDR. We don't need to signal the
// The meta data includes a dummy gateway address. This prevents libnetwork
// from requesting a gateway address from the pool since for a Calico
// network our gateway is set to a special IP.
resp := &ipam.RequestPoolResponse{
PoolID: poolID,
Pool: pool,
Data: map[string]string{"com.docker.network.gateway": gateway},
}

logutils.JSONMessage("RequestPool response", resp)

return resp, nil
}

其中,有一块代码逻辑是:




//upload-images.jianshu.io/upload_images/13868689-c8ec9743da0f5afe?imageMogr2/auto-orient/strip|imageView2/2/w/806/format/webp


RequestPool


六、问题列表



  1. docker: Error response from daemon: IpamDriver.RequestAddress: IP assignment error: No configured Calico pools.




    //upload-images.jianshu.io/upload_images/13868689-712199153a6caaae?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp


    image




    查看libnetwork的日志:




    //upload-images.jianshu.io/upload_images/13868689-c2d6813f1c9a5490?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp


    image



    或者说,没有定义ip 池,

    解决措施,创建一个ipPool就可以了,

要回复问题请先登录注册