1 Jenkinsfile概念
Jenkinsfile 使用两种语法进行编写,分别是声明式和脚本式。
声明式和脚本式的流水线从根本上是不同的。
声明式是 jenkins 流水线更友好的特性。
脚本式的流水线语法,提供更丰富的语法特性。
声明式流水线使编写和读取流水线代码更容易设计
2 Pipeline概念
Jenkins 的 Pipeline 通过 Jenkinsfile 进行描述(类似于 Dockerfile)
Jenkinsfile 是 Jenkins 的特性(pipeline as code)
Pipeline 是 Jenkins 的核心功能,提供一组可扩展的工具。
通过 Pipeline 的 DSL 语法可以完成从简单到复杂的交付流水线实现。
流水线组成:
1 Pipeline 流水线,整个构建过程
2 Agent 节点或代理,是一个机器, Jenkins环境的一部分
3 Stage 阶段,一般是不同子集,构建、测试、发布
4 Step 步骤,单一任务,用的较少
5 Post 运行后处理
pipeline{
//指定运行此流水线的节点
agent { node { label "build"}}
//流水线的阶段
stages{
//阶段 1 获取代码
stage("CheckOut"){
steps{
script{
println("获取代码")
}
}
}
stage("Build"){
steps{
script{
println("运行构建")
}
}
}
}
post {
always{
script{
println("流水线结束后,经常做的事情")
}
}
success{
script{
println("流水线成功后,要做的事情")
}
}
failure{
script{
println("流水线失败后,要做的事情")
}
}
aborted{
script{
println("流水线取消后,要做的事情")
}
}
}
}
声明式 Pipleine 是官方推荐的语法,声明式语法更加简洁,在声明式 Pipeline 中的基本语句和表达式遵循 Groovy 的语法。但是有以下例外:
- 流水线顶层必须是一个块,特别是 pipeline{}。
- 不需要分号作为分割符,是按照行分割的。
- 语句块只能由阶段、指令、步骤、赋值语句组成
3 Jenkins 流水线语法
- agent 代理
- post 运行后处理
- stages 阶段
- environment 环境变量
- options 运行选项
- parameters 参数
- trigger 触发器
- tool 构建工具
- input 交互输入
- when 条件判断
- parallel 并行
- script 脚本标签
Jenkinsfile 内置变量
BUILD_NUMBER //构建号
BUILD_ID //构建号
BUILD_DISPLAY_NAME //构建显示名称
JOB_NAME //项目名称
EXECUTOR_NUMBER //执行器数量
NODE_NAME //构建节点名称
WORKSPACE //工作目录
JENKINS_HOME //Jenkins home
JENKINS_URL //Jenkins地址
BUILD_URL //构建地址
JOB_URL //项目地址
4 Jenkins扩展-ShareLibrary
共享库这并不是一个全新的概念,其实具有编程能力的同学应该清楚一些。例如在编程语言 Python 中,我们可以将 Python 代码写到一个文件中,当代码数量增加,我们可以将代码打包成模块然后再以 import 的方式使用此模块中的方法。
在 Jenkins 中使用 Groovy 语法,共享库中存储的每个文件都是一个 groovy 的类,每个文件(类)中包含一个或多个方法。每个方法包含 groovy 语句块
5 Groovy 基础语法
- 数据类型
- 字符串 string
- 列表 list
- 映射 map
- 条件语句
- if 语句
- switch 语句
- 循环语句
- for 循环语句
- while 循环语句
- 函数
https://www.jenkins.io/doc/book/pipeline
pipeline {
agent {
node {
label 'maven'
}
}
parameters {
string(name:'TAG_NAME',defaultValue: '',description:'')
}
options{
buildDiscarder(logRotator(numToKeepStr: '5'))
quietPeriod(15)
}
environment {
APP_NAME = 'scp-job'
NAMESPACE = 'scp-pre'
CONTAINER_PORT = '8210'
JAVA_OPTS = '-Xmx1024m'
KUBECONFIG_CREDENTIAL_ID = 'pipeline-kubeconfig'
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'scp-pre'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
NACOS_HOST = 'nacos-hs.infrastructure-pre:8848'
NACOS_NAMESPACE = 'd9b32e8c-a835-4df7-a287-f4514ce3f1df'
NACOS_GROUP = 'scp'
SW_BACKEND = 'skywalking-oap.infrastructure-pre:11800'
HEALTH_PATH = '/health/check'
}
stages {
stage ('checkout scm') {
steps {
checkout(scm)
script {
sh 'wget http://192.168.13.78/pipeline-user/pipeline/-/raw/master/docker/maven/Dockerfile'
sh 'wget http://192.168.13.78/pipeline-user/pipeline/-/raw/master/maven/setting/settings.xml'
sh 'wget http://192.168.13.78/pipeline-user/pipeline/-/raw/master/deploy/maven/pre/deployment.yaml'
}
}
}
stage ('build & push') {
steps {
container ('maven') {
sh 'mvn -Dmaven.test.skip=true -gs `pwd`/settings.xml clean package'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage('deploy to develop') {
when{
branch 'develop'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy to pre') {
when{
branch 'pre'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
}
}
pipeline {
agent {
node {
label 'maven'
}
}
parameters {
string(name:'TAG_NAME',defaultValue: '',description:'')
}
options{
buildDiscarder(logRotator(numToKeepStr: '5'))
quietPeriod(5)
}
environment {
KUBECONFIG_CREDENTIAL_ID = 'pipeline-user-kubeconfig'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
APP_NAME = 'ufm-center-basic-service'
CONTAINER_PORT = '8080'
HEALTH_PATH = '/health/check'
}
stages {
stage ('checkout scm') {
steps {
checkout(scm)
script {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/docker/ufm/maven/Dockerfile'
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/maven/setting/settings.xml'
}
}
}
stage ('build develop') {
when{
branch 'develop'
}
environment {
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-dev'
}
steps {
container ('maven') {
sh 'mvn -U -Dmaven.test.skip=true -gs `pwd`/settings.xml clean package'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage ('build test') {
when{
branch 'test'
}
environment {
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-test'
}
steps {
container ('maven') {
sh 'mvn -U -Dmaven.test.skip=true -gs `pwd`/settings.xml clean package'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage ('build pre') {
when{
branch 'master'
}
environment {
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-pre'
}
steps {
container ('maven') {
sh 'mvn -U -Dmaven.test.skip=true -gs `pwd`/settings.xml -Ppre clean package'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage('bulid release') {
when{
tag "release-*"
}
environment {
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-prod'
}
steps {
container ('maven') {
sh 'mvn -U -Dmaven.test.skip=true -gs `pwd`/settings.xml -Pprod clean package'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage('deploy to develop') {
when{
branch 'develop'
}
environment {
NAMESPACE = 'ufm-dev'
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-dev'
NACOS_HOST = 'nacos-hs.infrastructure-test:8848'
NACOS_NAMESPACE = 'e3b19280-b268-4b52-b217-8cc0a983574d'
NACOS_GROUP = 'ufm-group'
SW_BACKEND = 'skw-oap.skywalking.svc.devcluster.local:11800'
JAVA_OPTS = '-Dspring.profiles.active=dev -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/maven/dev/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy to test') {
when{
branch 'test'
}
environment {
NAMESPACE = 'ufm-test'
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-test'
NACOS_HOST = 'nacos-hs.infrastructure-test:8848'
NACOS_NAMESPACE = 'c4ec1565-2971-4237-bdb6-87a20856c99c'
NACOS_GROUP = 'ufm-group'
SW_BACKEND = 'skw-oap.skywalking.svc.devcluster.local:11800'
JAVA_OPTS = '-Dspring.profiles.active=test -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/maven/dev/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy to pre') {
when{
branch 'master'
}
environment {
NAMESPACE = 'ufm-pre'
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-pre'
NACOS_HOST = 'nacos-hs.infrastructure-pre:8848'
NACOS_NAMESPACE = 'd9b32e8c-a835-4df7-a287-f4514ce3f1df'
NACOS_GROUP = 'ufm-group'
SW_BACKEND = 'skywalking-oap.infrastructure-pre:11800'
JAVA_OPTS = '-Dspring.profiles.active=pre'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/maven/pre/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy release') {
when{
tag "release-*"
}
environment {
NAMESPACE = 'ufm-prod'
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-prod'
NACOS_HOST = 'nacos-hs.infrastructure-prod:8848'
NACOS_NAMESPACE = '1f8da0a1-929d-42cc-b3d5-239cc068f3e8'
NACOS_GROUP = 'ufm-group'
SW_BACKEND = 'skywalking-oap.infrastructure-prod:11800'
JAVA_OPTS = '-Dspring.profiles.active=prod'
}
steps {
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/maven/prod/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
}
}
pipeline {
agent {
node {
label 'nodejs'
}
}
parameters {
string(name:'TAG_NAME', defaultValue: '', description:'')
}
options {
buildDiscarder(logRotator(numToKeepStr: '5'))
quietPeriod(5)
}
environment {
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
KUBECONFIG_CREDENTIAL_ID = 'pipeline-user-kubeconfig'
APP_NAME = 'ufm-mgr'
}
stages {
stage ('checkout scm') {
steps {
checkout(scm)
script {
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/docker/vue/Dockerfile'
}
}
}
stage ('build develop') {
when {
branch 'develop'
}
environment {
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-dev'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
}
steps {
container('nodejs') {
sh 'yarn config set ignore-engines true'
sh 'yarn install --frozen-lockfile'
sh 'yarn build:dev'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER'
}
}
}
}
stage ('build test') {
when {
branch 'test'
}
environment {
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-test'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
}
steps {
container('nodejs') {
sh 'yarn config set ignore-engines true'
sh 'yarn install --frozen-lockfile'
sh 'yarn build:test'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER'
}
}
}
}
stage ('build pre') {
when {
branch 'pre'
}
environment {
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-pre'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
}
steps {
container('nodejs') {
sh 'yarn config set ignore-engines true'
sh 'yarn install --frozen-lockfile'
sh 'yarn build:pre'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME.$TAG_NAME.$BUILD_NUMBER'
}
}
}
}
stage ('build release') {
when {
tag 'release-*'
}
environment {
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-prod'
HARBOR_CREDENTIAL_ID = 'pipeline-user-harbor'
}
steps {
container('nodejs') {
sh 'yarn config set ignore-engines true'
sh 'yarn install --frozen-lockfile'
sh 'yarn build:prod'
sh 'docker build -f `pwd`/Dockerfile -t $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(credentialsId : "$HARBOR_CREDENTIAL_ID" ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
sh 'echo "$HARBOR_PASSWORD" | docker login $HARBOR_HOST -u "$HARBOR_USERNAME" --password-stdin'
sh 'docker push $HARBOR_HOST/$HARBOR_NAMESPACE/$APP_NAME:$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage('deploy to develop') {
when {
branch 'develop'
}
environment {
NAMESPACE = 'ufm-dev'
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-dev'
}
steps {
container ('nodejs') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')])
{
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/vue/dev/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy to test') {
when {
branch 'test'
}
environment {
NAMESPACE = 'ufm-test'
HARBOR_HOST = 'bytest-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-test'
}
steps {
container ('nodejs') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')])
{
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/vue/dev/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage('deploy to pre') {
when {
branch 'pre'
}
environment {
NAMESPACE = 'ufm-pre'
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-pre'
}
steps {
container ('nodejs') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')])
{
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/vue/pre/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
stage ('deploy release') {
when {
tag 'release-*'
}
environment {
NAMESPACE = 'ufm-prod'
HARBOR_HOST = 'hw-harbor.ur.com.cn'
HARBOR_NAMESPACE = 'ufm-prod'
}
steps {
container ('nodejs') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')])
{
sh 'wget http://192.168.13.78/paas-pub/pipeline/-/raw/master/deploy/ufm/vue/prod/deployment.yaml'
sh 'envsubst < `pwd`/deployment.yaml | cat -'
sh 'envsubst < `pwd`/deployment.yaml | kubectl apply -f -'
}
}
}
}
}
}