Kubernetes Ephemeral Container

Kubernetes Plugin extension that adds the ability to run steps in an Ephemeral Container. This plugin provides similar behavior to withDockerContainer but for Kubernetes. Many modern Kubernetes cluster environments do not have access to the Docker daemon required by the Docker plugin. This plugin aims to bridge that gap.

• Getting Started
• Limitations
• Best Practices
• Container Entrypoints
• Environment Variables
• Nested Ephemeral Container Steps
• Step Rules
• Advanced Settings
• Metrics
• Contributing
• License

The plugin adds an Ephemeral Container to the current Kubernetes Cloud agent Pod, executes some steps then the container is terminated at the end of the block. This means pipelines may run step inside a container not explicitly defined in the cloud Pod Template.

pipeline {
    agent {
        // Reference a kubernetes pod template label
        label 'k8s-agent'
        // Use inline kubernetes agent 
        // kubernetes { ... }
    }
    stages {
        stage('build') {
            steps {
                withEphemeralContainer(image: 'maven:3') {
                    // Shares the build workspace
                    sh 'mvn clean install'
                }
                // Ephemeral container is terminated here
            }
        }
    }
}

The Kubernetes Cloud service account role must have the pod/ephemeralcontainers permission.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: jenkins-agent
rules:
  - apiGroups:
      - ""
    resources:
      - events
      - pods/logs
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/exec
      - pods/ephemeralcontainers
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
      - delete

The Ephemeral Container spec does not allow for specifying resources, but will still be subject to the Pod overall resource limits. The kubelet may evict a Pod if an ephemeral container causes the Pod to exceed its resource allocation. To avoid eviction it is advisable to increase the limits of the primary container to account for potential ephemeral container resource requirements.

In Kubernetes, the size limit for objects stored in etcd, including pod specs, is 1.5MB per key-value pair. This means there is a finite number of ephemeral containers that may be added to a build agent. This is usually not much of an issue for most pipelines, but could become an issue for long-running Pods. For this reason it is not recommended to re-use agents node for multiple builds. Setting the Pod Template idle minutes property ("Time in minutes to retain agent when idle") to 0 is a good practice.

• Windows containers are not supported
• Container must have a shell available

• Set the Pod Template idle minutes property ("Time in minutes to retain agent when idle") to 0. This prevents Pod spec container accumulation across builds which could lead to Pod spec size limits.
• Do not loop over withEphemeralContainer

  // Bad - Could trigger Pod spec size limits
  for (def file in files) {
    withEphemeralContainer("koalaman/shellcheck") {
      sh "shellcheck ${file}"
    }
  }
  
  // Better - Move the loop inside the container block
  withEphemeralContainer("koalaman/shellcheck") {
    for (def file in files) {
      sh "shellcheck ${file}"
    }
  }
  
  // Best - Execute a single shell command in the container
  withEphemeralContainer("koalaman/shellcheck") {
    sh "shellcheck ${files.join(' ')}"
  }

For containers to work properly the container entrypoint, if defined, must be able to accept and execute a monitor command provided by this plugin. By default, the monitor command will be passed as container command args.

#!/bin/bash
# /opt/docker/entrypoint.sh

# do some custom entrypoint stuff

# execute monitor command provided by jenkins
exec "$@"

If the container has an entrypoint that does not respect command args, the entrypoint can be disabled by passing an empty command line.

// disable container entrypoint
withEphemeralContainer(image: image, command: []) {
    sh 'echo "hello"'
}

When ephemeral containers are enabled for a cloud a build environment variable will be added. This is useful for script library authors that may need to provide alternate execution options, for example fallback to withDockerContainer.

• KUBERNETES_CLOUD_EPHEMERAL_CONTAINERS_ENABLED=true

// vars/withMaven.groovy
def call(config = [:], body = null) {
    def image = "maven:${config.image?:'3'}"
    if (env.KUBERNETES_CLOUD_EPHEMERAL_CONTAINERS_ENABLED == "true") {
        withEphemeralContainer(image: image) {
            body()
        }
    } else {
        withDockerContainer(image: image) {
            body()
        }
    }
}

Container environment variables may also be configured. This may be necessary for the container entrypoint, but otherwise work the same as withEnv.

withEphemeralContainer(image: 'maven', envVars: [envVar(key: 'FOO', value: 'bar')]) {
    withEnv(["BAR=baz"]) {
        sh 'echo "$FOO $BAR"'
    }
}

withEphemeralContainer steps may be nested. This does not mean one container runs inside another. It just means that multiple running ephemeral containers are added to the agent Pod.

• All the containers share the same network, see Pod Networking. This enables use cases where one container starts a service (i.e. http server) and another container calls it via localhost as a pair of nested steps.
• If the user arg is set, the next nested container will inherit the uid:gid of the parent.

withEphemeralContainer(image: 'redis:8-alpine') {
  timeout(time: 5, unit: 'SECONDS') {
    sh '''
       set -e
       redis-server &
       while ! nc -z localhost 6379; do sleep 1; done
    '''
  }
  withEphemeralContainer(image: 'redis:8-alpine') {
    sh 'redis-cli ping'
  }
}

Ephemeral container step rules can be defined globally or by cloud.

Use wildcard patterns to with either allow or reject specific images or registries that can be run.

• Use * to match any character sequence (i.e. docker.io/library/*)
• Image names are normalized (i.e. maven → docker.io/library/maven)
• If Allow, non-matches throw abort exception
• If Reject, matches throw abort exception

For clusters with high load or pipelines with lots of concurrency the following system property settings may be tweaked to improve container creation reliability.

• io.jenkins.plugins.kubernetes.ephemeral.EphemeralContainerStepExecution.startMaxRetry

  Number of retry attempts if container fails to start due to context deadline miss. May be increased for clusters with kubelet load.

  Default: 3

• io.jenkins.plugins.kubernetes.ephemeral.EphemeralContainerStepExecution.startRetryMaxWaitSecs

  Max wait time between retries. Used to spread kubelet load.

  Default: 2

• io.jenkins.plugins.kubernetes.ephemeral.EphemeralContainerStepExecution.pathMaxRetry

  Number of retry attempts to patch Pod spec. May be increased for highly concurrent pipelines create a lot patch conflicts.

  Default: 10

• io.jenkins.plugins.kubernetes.ephemeral.EphemeralContainerStepExecution.patchRetryMaxWaitSecs

  Max wait time between retries. Used to minimize patch conflicts.

  Default: 2

Container metrics are available via the metrics plugin endpoint. Metric keys start with kubernetes.cloud.containers.ephemeral..

$JENKINS_URL/metrics/currentUser/metrics?pretty=true

Refer to our contribution guidelines

This project is licensed under the MIT License - see the LICENSE file for details.