最近在学习如何使用jenkins结合阿里云k8s自动化集成和部署springcloud微服务和前端next.js项目,如今记录下来分享给你们,文章有什么不妥的地方,但愿大佬们批评指正。css
一、购买阿里云ACK集群(或者自行搭建)html
我购买的是阿里云ACK托管版,建立集群地址前端
注意:建立ACK集群不收费,收费的是里面的NAT网关、SLB负载均衡、ECS服务器等vue
二、安装gitlabjava
这里我以前写了一篇阿里云ECS搭建gitlab,安装gitlab地址node
三、安装jenkinsreact
这里我以前写了一篇阿里云ECS搭建jenkins,安装jenkins地址linux
四、安装dockernginx
使用官网推荐yum 安装比较方便,安装以下:git
sudo yum install -y yum-utils \ device-mapper-persistent-data \ lvm2 sudo yum-config-manager \ --add-repo \ https://download.docker.com/linux/centos/docker-ce.repo sudo yum install docker-ce docker-ce-cli containerd.io
Next.js项目:
服务:react+next.js 服务端口:3000, K8S:Deployment+Server+Ingress Pod名:demo-webapp
Dockerfile
FROM node:12 # 设置工做路径。全部的路径都会关联WORKDIR WORKDIR /usr/src/app # 安装依赖 COPY package*.json ./ RUN npm install # 拷贝源文件 COPY . . # 构建应用 RUN npm run build # 运行应用 CMD [ "npm", "start" ]
一、服务发现:SpringCloud Eureka
微服务名:demo-eureka-server 微服务端口:8761, K8S:Deployment+Service+Ingress(由于要经过网址查看服务注册状况,因此要添加ingress) Pod名:demo-eureka-server
二、服务配置:SpringCloud Config
微服务名:demo-config-server 微服务端口:8888, K8S:Deployment+Service Pod名:demo-config-server
三、服务认证和受权:SpringSecurity + Oauth2
微服务名:demo-auth-service 微服务端口:8901, K8S:Deployment+Service Pod名:demo-auth-service
四、服务网关:SpringCloud Zuul
微服务名:demo-auth-service 微服务端口:5555, K8S:Deployment+Service+Ingress(服务网关是全部服务对外惟一入口,因此要配置ingress) Pod名:demo-auth-service
五、编写Dockerfile、K8S yaml
为了方便起见,上面的微服务名和Pod名都设置相同名称,具体的服务开发过程这里就暂时忽略,后面有时间再写篇搭建企业级微服务的文章吧,相关微服务的目录结构以下所示:
注意:不一样服务之间Dockerfile和k8s yaml文件大同小异,咱们以Eureka为例:
Eureka Dockerfile:
FROM openjdk:8-jdk-alpine MAINTAINER "zhangwei"<zhangwei900808@126.com> RUN mkdir -p /usr/local/configsvr ARG JAR_FILE ADD ${JAR_FILE} /usr/local/configsvr/app.jar ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/usr/local/configsvr/app.jar"] EXPOSE 8888
Eureka K8S Yaml:
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30
注意:不一样的策服务bootstrap.yml和application.yml文件仍是有所区别,好比下面这些:
demo-auth-service、demo-zuul-server 配置bootstrap.yml文件添加SpringCloud Config地址:
spring: application: name: authservice cloud: config: enabled: true # config的Server名 uri: http://demo-config-server-svc:8888
demo-auth-service、demo-zuul-server、demo-config-server 配置application.yml文件添加Eureka地址:
eureka: instance: preferIpAddress: true hostname: demo-eureka-server-svc client: register-with-eureka: true fetch-registry: true service-url: # eureka的Server名 defaultZone: http://demo-eureka-server-svc:8761/eureka/
按照Git工做流,通常咱们能够分为:开发环境(develop)、测试环境(release)、预生产环境(uat)和生产环境(prop),对应的分支分别是:dev、test、release、master,所以不一样阶段提交的代码所属分支也不相同,且dockerfile、jenkins pipline、k8s yaml文件也不相同,这个要注意下。
jenkins能够按照git工做流添加:测试视图、预生产视图、生产视图,以下所示:
咱们先建立流水线任务并编写pipline script脚原本建立CI/CD流水线步骤
先编写环境变量
environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" }
步骤一:克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成本身的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值作为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } }
步骤二:代码测试
stage("Test") { steps { echo "2.Test Stage" } }
步骤三:制做docker镜像
stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." } }
步骤四:推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的帐号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } }
步骤五:k8s部署docker镜像
stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } }
sed 语法你们能够自行百度,这里我使用的是#分隔,而没有使用/分隔,缘由是分隔的字符中包含/,因此不能再用。
首先,咱们来看看前端k8s yaml文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt-prod-http01 kubernetes.io/ingress.class: nginx nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/service-weight: '' generation: 3 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 3000 path: / pathType: ImplementationSpecific tls: - hosts: - baidu.com secretName: <podname>-ingress --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: ports: - port: 3000 protocol: TCP targetPort: 3000 selector: app: <podname> sessionAffinity: None type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app:<podname> spec: containers: - image: <imagename> imagePullPolicy: IfNotPresent name: <podname> resources: requests: cpu: 250m memory: 512Mi
上面的yaml文件中包含:deployment、service和ingress,ingress中配置了tls,所以可使用https来访问域名,而且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
因此能够自动将http 跳转到https,其中里面的<podname>和<imagename>是要被sed替换成真实名称和地址。
完整的pipline script脚本:
pipeline { agent any environment { GIT_REPOSITORY="前端代码仓库地址" K8S_YAML="k8s yaml文件所在目录" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = '阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成本身的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值作为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Docker Image Stage" sh "docker build -t ${DOCKER_REPOSITORY_TAG} -f docker/Dockerfile ." } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的帐号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } } } }
点击当即构建,以下所示:
先编写环境变量
environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" }
步骤一:克隆代码
stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() // 测试分支,jenkins-gitlab-ssh-hash是ssh密钥,替换成本身的就好 git branch: 'test', credentialsId: 'jenkins-gitlab-ssh-hash', url: "${GIT_REPOSITORY}" script { // 获取git提交的hash值作为docker镜像tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() // 组装成完整地址 DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } }
步骤二:代码测试
stage("Test") { steps { echo "2.Test Stage" } }
步骤三:制做docker镜像
stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" } }
这里说明一下,由于咱们使用的是maven 多模块因此打包编译的时候要分模块来打包,因此要使用-pl指定模块名,而后咱们pom.xml中使用了dockerfile-maven-plugin,以下所示:
<build> <plugins> <plugin> <groupId>com.spotify</groupId> <artifactId>dockerfile-maven-plugin</artifactId> <version>1.4.10</version> <configuration> <!-- 指定dockerfile所在目录--> <dockerfile>src/main/docker/Dockerfile</dockerfile> <buildArgs> <!--提供参数向Dockerfile传递--> <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE> </buildArgs> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build>
步骤四:推送docker镜像
stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的帐号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } }
步骤五:k8s部署docker镜像
stage("Deploy") { steps { echo "5.发布镜像" // 使用sed替换k8s yaml文件中的<imagename>和<podname> sh "sed -i 's#<imagename>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" // 执行应用k8s yaml sh "kubectl apply -f ${K8S_YAML}" } }
sed 语法你们能够自行百度,这里我使用的是#分隔,而没有使用/分隔,缘由是分隔的字符中包含/,因此不能再用。
首先,咱们来看看Eureka k8s yaml文件
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: nginx.ingress.kubernetes.io/configuration-snippet: | rewrite ^/eureka/css/(.*)$ /eureka/eureka/css/$1 redirect; rewrite ^/eureka/js/(.*)$ /eureka/eureka/js/$1 redirect; nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/service-weight: '' generation: 4 name: <podname>-ingress namespace: default spec: rules: - host: baidu.com http: paths: - backend: serviceName: <podname>-svc servicePort: 8761 path: /eureka(/|$)(.*) pathType: ImplementationSpecific --- apiVersion: v1 kind: Service metadata: name: <podname>-svc namespace: default spec: externalTrafficPolicy: Local ports: - nodePort: 31061 port: 8761 protocol: TCP targetPort: 8761 selector: app: <podname> sessionAffinity: None type: NodePort --- apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: '1' generation: 1 labels: app: <podname> name: <podname> namespace: default spec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: <podname> strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: <podname> spec: containers: - env: - name: LANG value: C.UTF-8 - name: JAVA_HOME value: /usr/lib/jvm/java-1.8-openjdk image: <imagename> imagePullPolicy: IfNotPresent name: <podname> ports: - containerPort: 8761 protocol: TCP resources: requests: cpu: 250m memory: 512Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30
上面的yaml文件中包含:deployment、service和ingress,ingress中配置了tls,所以可使用https来访问域名,而且配置了nginx.ingress.kubernetes.io/force-ssl-redirect: 'true'
因此能够自动将http 跳转到https,其中里面的<podname>和<imagename>是要被sed替换成真实名称和地址。
完整的pipline script脚本:
pipeline { agent any environment { GIT_REPOSITORY="代码仓库地址" MODULE_NAME="maven 模块名" POD_NAME="k8s pod name" K8S_YAML="${MODULE_NAME}/src/main/k8s/eurekasvr.yaml" DOCKER_USERNAME="docker 仓库用户名" DOCKER_PWD="docker仓库密码" ALIYUN_DOCKER_HOST = 阿里云docker仓库域名' ALIYUN_DOCKER_NAMESPACE="阿里云docker仓库命名空间" ALIYUN_DOCKER_REPOSITORY_NAME="阿里云docker仓库命名空间下的仓库名" } stages { stage("Clone") { steps { echo "1.Clone Stage" // 删除文件夹 deleteDir() git branch: 'test',credentialsId: '1297dda3-e592-4e70-8fb0-087a26c08db0', url: "${GIT_REPOSITORY}" script { // 获取git代码tag为docker仓库tag // GIT_TAG = sh(returnStdout: true,script: 'git describe --tags --always').trim() // 获取git提交hash作为docker仓库tag GIT_TAG = sh(returnStdout: true, script: 'git rev-parse --short HEAD').trim() DOCKER_REPOSITORY = "${ALIYUN_DOCKER_HOST}/${ALIYUN_DOCKER_NAMESPACE}/${ALIYUN_DOCKER_REPOSITORY_NAME}" // docker 阿里镜像仓库 DOCKER_REPOSITORY_TAG = "${DOCKER_REPOSITORY}:${GIT_TAG}" } } } stage("Test") { steps { echo "2.Test Stage" } } stage("Build") { steps { echo "3.Build Server" sh "mvn -e -U -pl ${MODULE_NAME} -am clean package -Dmaven.test.skip=true dockerfile:build -Ddockerfile.tag=${GIT_TAG} -Ddockerfile.repository=${DOCKER_REPOSITORY}" } } stage("Push") { steps { echo "4.Push Docker Image Stage" //推送Docker镜像,username 跟 password 为 阿里云容器镜像服务的帐号密码 sh "docker login --username=${DOCKER_USERNAME} --password=${DOCKER_PWD} ${ALIYUN_DOCKER_HOST}" // 开始推送镜像到阿里云docker镜像仓库 sh "docker push ${DOCKER_REPOSITORY_TAG}" // 删除jenkins生成的image sh ''' docker images | grep seaurl | awk '{print $3}' | xargs docker rmi -f ''' } } stage("Deploy") { steps { echo "5.发布镜像" sh "sed -i 's#<dockerrepository>#${DOCKER_REPOSITORY_TAG}#g;s#<podname>#${POD_NAME}#g' ${K8S_YAML}" sh "kubectl apply -f ${K8S_YAML}" } } } }
点击当即构建,以下所示:
能够经过命令
kubectl get deploy kubectl get pod kubectl get svc kubectl get ingress
还能够查看pod日志,对其成功或失败进行分析:
kubectl logs podname
能够经过postman接口访问对外的zuul地址,查看是否能够认证经过:
能够经过next.js的部署地址,查看是否部署成功:
一、jenkins pipline script脚本和Dockerfile写法网上各式各样,你们找到一个符合标准的就好
二、kubectl apply -f 的做用是:若是没有建立deployment就会建立,不然就更新
三、微服务k8s yaml文件中Service要设置type: NodePort不然,微服务间不能通讯
四、jenkins 编译微服务的maven模块要用到-pl 和 dockerfile-maven-plugin
五、微服务若是要用k8s ingress要设置https,以及http自动跳转到https
JenkinsPipeline部署一个Kubernetes 应用
采用jenkins pipeline实现自动构建并部署至k8s
Configuring-CI-CD-on-Kubernetes-with-Jenkins
spring-k8s
jenkins pipeline 自动构建并部署至k8s
微服务实战(一)基于OAUTH2.0统一认证受权的微服务基础架构
使用cert-manager申请免费的HTTPS证书
SPRINGBOOT利用SPRING.PROFILES.ACTIVE=@SPRING.ACTIVE@不一样环境下灵活切换配置文件
https://kuboard.cn/learning/k8s-practice/ocp/eureka-server.html#%E6%9F%A5%E7%9C%8B%E9%83%A8%E7%BD%B2%E7%BB%93%E6%9E%9C
kubernetes部署微服务spring cloud的简单例子
k8s-nginx-ingress eureka二级路径转发的问题