jenkins-gitlab-harbor-ceph基于Kubernetes的CI/CD运用（二）

一张网图

因为我们使用了Docker in Docker技术，就是把jenkins部署在k8s里。jenkins master会动态创建slave pod，使用slave pod运行代码克隆，项目构建，镜像构建等指令操作。构成完成以后删除这个slave pod。减轻jenkins-master的负载，可以极大地提高资源利用率。

配置连接kubernetes

我们已经安装了Kubernetes插件，我们直接在jenkins中点击

manage jenkins -- > configure system -- > 拉到最底下有一个cloud。

新增一个cloud --> kubernetes

测试

podTemplate(label: 'pod-golang',

    containers: [

        containerTemplate(

            name: 'golang',

            image: 'golang',

            ttyEnabled: true,

            command: 'cat'

        )

    ]

) {

    node ('pod-golang') {

        stage 'Switch to Utility Container'

        container('golang') {

          sh ("go version")

        }

    }

}

输出

# kubectl get pods -A -w

assembly      pod-golang-qbcxr-fbc73                       2/2     Running             0          28s

assembly      pod-golang-qbcxr-fbc73                       2/2     Terminating         0          69s

Started by user zisefeizhu

Running in Durability level: MAX_SURVIVABILITY

[Pipeline] Start of Pipeline

[Pipeline] podTemplate

[Pipeline] {

[Pipeline] node

Created Pod: pod-golang-qbcxr-gkbx2 in namespace assembly

Still waiting to schedule task

‘pod-golang-qbcxr-gkbx2’ is offline

Created Pod: pod-golang-qbcxr-4tn61 in namespace assembly

Created Pod: pod-golang-qbcxr-sqfqk in namespace assembly

Created Pod: pod-golang-qbcxr-lncp9 in namespace assembly

Created Pod: pod-golang-qbcxr-czccr in namespace assembly

Created Pod: pod-golang-qbcxr-3d6wq in namespace assembly

Created Pod: pod-golang-qbcxr-x9q5m in namespace assembly

Created Pod: pod-golang-qbcxr-fbc73 in namespace assembly

Agent pod-golang-qbcxr-fbc73 is provisioned from template pod-golang-qbcxr

---

apiVersion: "v1"

kind: "Pod"

metadata:

  annotations:

    buildUrl: "http://jenkins.assembly.svc.cluster.local/job/gitlab-citest-pipeline/4/"

    runUrl: "job/gitlab-citest-pipeline/4/"

  labels:

    jenkins: "slave"

    jenkins/label: "pod-golang"

  name: "pod-golang-qbcxr-fbc73"

spec:

  containers:

  - command:

    - "cat"

    image: "golang"

    imagePullPolicy: "IfNotPresent"

    name: "golang"

    resources:

      limits: {}

      requests: {}

    securityContext:

      privileged: false

    tty: true

    volumeMounts:

    - mountPath: "/home/jenkins/agent"

      name: "workspace-volume"

      readOnly: false

  - env:

    - name: "JENKINS_SECRET"

      value: "********"

    - name: "JENKINS_AGENT_NAME"

      value: "pod-golang-qbcxr-fbc73"

    - name: "JENKINS_NAME"

      value: "pod-golang-qbcxr-fbc73"

    - name: "JENKINS_AGENT_WORKDIR"

      value: "/home/jenkins/agent"

    - name: "JENKINS_URL"

      value: "http://jenkins.assembly.svc.cluster.local/"

    image: "jenkins/jnlp-slave:3.35-5-alpine"

    name: "jnlp"

    volumeMounts:

    - mountPath: "/home/jenkins/agent"

      name: "workspace-volume"

      readOnly: false

  nodeSelector:

    beta.kubernetes.io/os: "linux"

  restartPolicy: "Never"

  securityContext: {}

  volumes:

  - emptyDir:

      medium: ""

    name: "workspace-volume"

Running on pod-golang-qbcxr-fbc73 in /home/jenkins/agent/workspace/gitlab-citest-pipeline

[Pipeline] {

[Pipeline] stage (Switch to Utility Container)

Using the ‘stage’ step without a block argument is deprecated

Entering stage Switch to Utility Container

Proceeding

[Pipeline] container

[Pipeline] {

[Pipeline] sh

+ go version

go version go1.14.1 linux/amd64

[Pipeline] }

[Pipeline] // container

[Pipeline] }

[Pipeline] // node

[Pipeline] }

[Pipeline] // podTemplate

[Pipeline] End of Pipeline

Finished: SUCCESS

以上podTemplate是在Pipeline里面定义的

*注意name必须jnlp，如果不为jnlp pod会启两个容器

name必须jnlp Kubernetes 才能用自定义 images 指定的镜像替换默认的 jenkinsci/jnlp-slave 镜像

构建jenkins-slave镜像

github官方构建slave文档

https://github.com/jenkinsci/docker-jnlp-slave

# ll

总用量 80900

-rw-r--r-- 1 root root      596 3月  23 09:14 Dockerfile

-rwxr-xr-x 1 root root 38457344 3月  22 20:55 helm

-rw-r--r-- 1 root root     2229 3月  21 20:04 jenkins-slave

-rwxr-xr-x 1 root root 43491328 3月  23 09:13 kubectl

-rw-r--r-- 1 root root      619 3月  21 20:05 settings.xml

-rw-r--r-- 1 root root   877037 3月  23 08:40 slave.jar

# cat Dockerfile

FROM centos:7

LABEL maintainer lizhenliang

# 使镜像具有拖git仓库，编译java代码的能力

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \

    yum clean all && \

    rm -rf /var/cache/yum/* && \

    mkdir -p /usr/share/jenkins

# 将获取到slave.jar放入镜像

COPY slave.jar /usr/share/jenkins/slave.jar

# jenkins-slave执行脚本

COPY jenkins-slave /usr/bin/jenkins-slave

# settings.xml中设置了aliyun的镜像

COPY settings.xml /etc/maven/settings.xml

RUN chmod +x /usr/bin/jenkins-slave

COPY helm kubectl /usr/bin/

ENTRYPOINT ["jenkins-slave"]

# cat jenkins-slave

#!/usr/bin/env sh

if [ $# -eq 1 ]; then

    # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image

    exec "$@"

else

    # if -tunnel is not provided try env vars

    case "$@" in

        *"-tunnel "*) ;;

        *)

        if [ ! -z "$JENKINS_TUNNEL" ]; then

            TUNNEL="-tunnel $JENKINS_TUNNEL"

        fi ;;

    esac

    # if -workDir is not provided try env vars

    if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then

        case "$@" in

            *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;

            *)

            WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;

        esac

    fi

    if [ -n "$JENKINS_URL" ]; then

        URL="-url $JENKINS_URL"

    fi

    if [ -n "$JENKINS_NAME" ]; then

        JENKINS_AGENT_NAME="$JENKINS_NAME"

    fi  

    if [ -z "$JNLP_PROTOCOL_OPTS" ]; then

        echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"

        JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"

    fi

    # If both required options are defined, do not pass the parameters

    OPT_JENKINS_SECRET=""

    if [ -n "$JENKINS_SECRET" ]; then

        case "$@" in

            *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;

            *)

            OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;

        esac

    fi

    OPT_JENKINS_AGENT_NAME=""

    if [ -n "$JENKINS_AGENT_NAME" ]; then

        case "$@" in

            *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;

            *)

            OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;

        esac

    fi

    #TODO: Handle the case when the command-line and Environment variable contain different values.

    #It is fine it blows up for now since it should lead to an error anyway.

    exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"

fi

# cat settings.xml

<?xml version="1.0" encoding="UTF-8"?>

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"

          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">

  <pluginGroups>

  </pluginGroups>

  <proxies>

  </proxies>

  <servers>

  </servers>

  <mirrors>

    <mirror>

      <id>central</id>

      <mirrorOf>central</mirrorOf>

      <name>aliyun maven</name>

      <url>https://maven.aliyun.com/repository/public</url>

    </mirror>

  </mirrors>

  <profiles>

  </profiles>

</settings>

构建镜像并打上标签

# docker build . -t harbor.linux.com/jenkinsci/jenkins-slave-jdk:1.8

推送镜像到harbor

#docker push harbor.linux.com/jenkinsci/jenkins-slave-jdk:1.8

创建应用Pipeline动态构建测试

// 镜像仓库地址

def registry = "harbor.linux.com"

podTemplate(label: 'jenkins-agent', cloud: 'kubernetes',

    containers: [

    containerTemplate(

        name: 'jnlp',

        image: "${registry}/jenkinsci/jenkins-slave-jdk:1.8"

    )],

    volumes: [

        hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),

        hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker')

    ])

{

  node("jenkins-agent"){

        stage('拉取代码') { // for display purposes

            echo 'ok'

        }

        stage('代码编译') {

            echo 'ok'

        }

        stage('部署') {

            echo 'ok'

        }

    }

}

输出

Started by user zisefeizhu

Running in Durability level: MAX_SURVIVABILITY

[Pipeline] Start of Pipeline

[Pipeline] podTemplate

[Pipeline] {

[Pipeline] node

Still waiting to schedule task

‘Jenkins’ doesn’t have label ‘jenkins-agent’

Created Pod: jenkins-agent-xnj85-cq1vx in namespace assembly

Agent jenkins-agent-xnj85-cq1vx is provisioned from template jenkins-agent-xnj85

---

apiVersion: "v1"

kind: "Pod"

metadata:

  annotations:

    buildUrl: "http://jenkins.assembly.svc.cluster.local/job/gitlab-citest-pipeline/18/"

    runUrl: "job/gitlab-citest-pipeline/18/"

  labels:

    jenkins: "slave"

    jenkins/label: "jenkins-agent"

  name: "jenkins-agent-xnj85-cq1vx"

spec:

  containers:

  - env:

    - name: "JENKINS_SECRET"

      value: "********"

    - name: "JENKINS_AGENT_NAME"

      value: "jenkins-agent-xnj85-cq1vx"

    - name: "JENKINS_NAME"

      value: "jenkins-agent-xnj85-cq1vx"

    - name: "JENKINS_AGENT_WORKDIR"

      value: "/home/jenkins/agent"

    - name: "JENKINS_URL"

      value: "http://jenkins.assembly.svc.cluster.local/"

    image: "harbor.linux.com/jenkinsci/jenkins-slave-jdk:1.8"

    imagePullPolicy: "IfNotPresent"

    name: "jnlp"

    resources:

      limits: {}

      requests: {}

    securityContext:

      privileged: false

    tty: false

    volumeMounts:

    - mountPath: "/var/run/docker.sock"

      name: "volume-0"

      readOnly: false

    - mountPath: "/usr/bin/docker"

      name: "volume-1"

      readOnly: false

    - mountPath: "/home/jenkins/agent"

      name: "workspace-volume"

      readOnly: false

  nodeSelector:

    beta.kubernetes.io/os: "linux"

  restartPolicy: "Never"

  securityContext: {}

  volumes:

  - hostPath:

      path: "/var/run/docker.sock"

    name: "volume-0"

  - hostPath:

      path: "/usr/bin/docker"

    name: "volume-1"

  - emptyDir:

      medium: ""

    name: "workspace-volume"

Running on jenkins-agent-xnj85-cq1vx in /home/jenkins/agent/workspace/gitlab-citest-pipeline

[Pipeline] {

[Pipeline] stage

[Pipeline] { (拉取代码)

[Pipeline] echo

ok

[Pipeline] }

[Pipeline] // stage

[Pipeline] stage

[Pipeline] { (代码编译)

[Pipeline] echo

ok

[Pipeline] }

[Pipeline] // stage

[Pipeline] stage

[Pipeline] { (部署)

[Pipeline] echo

ok

[Pipeline] }

[Pipeline] // stage

[Pipeline] }

[Pipeline] // node

[Pipeline] }

[Pipeline] // podTemplate

[Pipeline] End of Pipeline

Finished: SUCCESS

# kubectl get pods  --all-namespaces -w -o wide

NAMESPACE     NAME                                         READY   STATUS    RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Pending   0          0s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Pending   0          0s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     ContainerCreating   0          0s

assembly      jenkins-agent-7hbn2-5kwwv                    1/1     Running             0          3s

assembly      jenkins-agent-7hbn2-5kwwv                    1/1     Terminating         0          68s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Terminating         0          72s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Terminating         0          72s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Terminating         0          77s

assembly      jenkins-agent-7hbn2-5kwwv                    0/1     Terminating         0          77s

会发现 jenkins-agent-xnj85-cq1vx  构建完成后自动删除

jenkins-gitlab-harbor-ceph基于Kubernetes的CI/CD运用（二）的相关教程结束。