BTC DSL for Pipeline

In order to use BTC EmbeddedPlatform steps in your Jenkins Pipeline, here's what should be prepared upfront:

The Jenkins Automation Plugin for BTC EmbeddedPlatform provides easy to use steps for Jenkins Pipeline enabling you to execute test workflow steps from Jenkins. The BTC EmbeddedPlatform HTML reports become available automatically on the Jenkins job page and a JUnit XML report is generated that feeds into Jenkins test trends and status overview dashboards.

The following workflow steps are available for Jenkins Pipeline:

• Profile Load / Create / Update
• Import of Test Cases and Stimuli Vectors
• Import / Export of Tolerances and Input Restrictions
• Execution of Functional Tests
• Automatic Vector Generation & Analysis
• Back-to-Back Test Execution
• Regression Test Execution
• Formal Verification
• Formal Test
• Configuration of Additional Coverage Goals
• Export of different reports
• BTC Migration Suite

For these workflow steps many additional settings can be configured (optional) in order to adapt the automated test runs to your individual project needs. In addition, two utility methods help you to connect to instances of BTC EmbeddedPlatform and to close them if again required. In addition, the Migration Suite use case can be addressed (see section Migration Suite below).

Table of Contents

• Description

• Release Notes

• Prerequisites

• Jenkins Pipeline

  • Overview
  • Licensing for Jenkins Integration
  • Configuration

• Workflow Steps

  • Basics
    • Step "startup"
    • Step "profileLoad"
    • Step "profileCreateTL"
    • Step "profileCreateEC"
    • Step "profileCreateSL"
    • Step "profileCreateC"
    • Step "wrapUp"
  • Import & Export
    • Step "vectorImport"
    • Step "vectorExport"
    • Step "toleranceImport"
    • Step "toleranceExport"
    • Step "inputRestrictionsImport"
    • Step "inputRestrictionsExport"
    • Step "executionRecordExport"
  • Analysis, Verification & Validation
    • Step "rbtExecution"
    • Step "vectorGeneration"
    • Step "backToBack"
    • Step "regressionTest"
    • Step "rangeViolationGoals"
    • Step "domainCoverageGoals"
    • Step "addDomainCheckGoals"
    • Step "addInputCombinationGoals"
    • Step "defaultTolerances"
    • Step "formalTest"
    • Step "formalVerification"
  • Reporting
    • Step "testExecutionReport"
    • Step "xmlReport"
    • Step "codeAnalysisReport"
    • Step "modelCoverageReport"
    • Step "interfaceReport"
    • Overall Report
  • Misc
    • Step "getStatusSummary"
    • "Error handling"

• BTC Migration Suite

  • Step "migrationSource"
  • Step "migrationTarget"
  • Migration Suite Example: Jenkinsfile

• Adding the BTC Plugin to Jenkins

This plugin only works in combination with BTC EmbeddedPlatform which needs to be installed and licensed separately.

Integrating test runs with BTC EmbeddedPlatform in your Jenkins workflows combines the automation and traceability concepts and results in great benefits:

1. The automated workflows scale for multiple components / projects with low configuration effort

2. You are easily able to trace changes made to your system under test from the Source Code Management to the integrated product and recognize test failures early in the process

3. The pipeline visualization intuitively shows how much time each phase of the testing process takes
   

4. The Jenkins Automation Plugin produces an XML report in the JUnit format that can be analyzed by Jenkins to provide test status trends over multiple executions and projects
   

5. Comprehensive HTML Reports from BTC EmbeddedPlatform are available directly from the Jenkins job page

   Jenkins' Content Security Policy can prevent the reports from being displayed properly. See Configuring Content Security Policy for further details.

6. Relevant artifacts like the test profile are accessible for easy debugging and analysis

   

In addition to the basic license requirements that depend on the chosen workflow steps which require EmbeddedTester or EmbeddedValidator the Jenkins Automation use case requires the "Test Automation Server" floating network license (ET_AUTOMATION_SERVER).

Since v2.4.0 it's possible to run ET_BASE use cases (Architecture Import, Test Case Import, Test Execution, Reporting) with an EmbeddedTester BASE installation and the license ET_AUTOMATION_SERVER_BASE.

In a Jenkins Pipeline the configuration of a job can be defined as simple groovy code which can be versioned alongside the main source files of the component. A full documentation of the Jenkins Pipeline can be found here. The following example shows of how BTC EmbeddedPlatform can be automated from Jenkins via the BTC DSL for Pipeline Plugin. The Plugin needs to be installed in Jenkins and dedicated BTC methods to create a test automation workflow.

Pipeline Example

pipeline {
    agent none // agent can be defined per stage
    stages {
        stage ('BTC Unit Test') {
            // Stage must run on an agent with the required software, e.g.:
            // - Matlab Simulink + Code Generator
            // - Compiler (Visual Studio / MinGW)
            // - BTC EmbeddedPlatform incl. JenkinsAutomation plugin
            agent { label 'my_btc_agent' }

            steps {
                // checkout files from repository referenced in this pipeline item
                checkout scm
                // Tests with BTC EmbeddedPlatform
                script {
                    // start BTC EmbeddedPlatform on Agent
                    btc.startup {
                        additionalJvmArgs = '-Xmx2g'
                    }
                    
                    // load and update test project
                    btc.profileLoad {
                        profilePath = "my_module.epp"
                        tlModelPath = "my_module.slx"
                        tlScriptPath = "start.m"
                        matlabVersion = "2020b"
                        updateRequired = true
                    }

                    // Execute requirements-based tests
                    btc.rbtExecution {
                        createReport = true
                    }

                    // Generate vectors for statement & mcdc coverage
                    btc.vectorGeneration {
                        pll = "STM; MCDC"
                    }
                    
                    // Generate coverage reports
                    btc.codeAnalysisReport { useCase = "RBT" }
                    btc.codeAnalysisReport { useCase = "B2B" }
                    
                    // B2B test model vs. code
                    btc.backToBack {
                        reference = "TL MIL"
                        comparison = "SIL"
                    }
                
                    // BTC: close EmbeddedPlatform and store reports
                    btc.wrapUp {}
                }
            }
            post {
                always {
                    // make sure that EP is closed, also in case of errors or if the build is aborted
                    script { btc.killEp {} }
                }
            }
        }
    }
}   

DSL Command: btc.startup {...}

Description

Method to connect to BTC EmbeddedPlatform with a specified port. If BTC EmbeddedPlatform is not available it is started and the method waits until it is available. The following optional settings are available:

Possible Return values

Jenkins will always connect to the active version of EmbeddedPlatform since many tasks will only work with the version that is integrated into Matlab. Please ensure that the correct EP version is active by choosing Activate BTC EmbeddedPlatform in your start menu for the desired version and also ensure that the Jenkins Automation Plugin is installed for this version of EmbeddedPlatform.

DSL Command: btc.profileLoad {...}

Description

Opens the profile if the specified profile exists, otherwise creates a new profile. A profile update is only performed if this is required. Profile Creation requires either a TargetLink model or C-Code in combination with a CodeModel.xml architecture description.

The "profileLoad" step or any of the "profileCreate" steps are a mandatory starting point for all automation workflows.

Possible Return values

DSL Command: btc.profileCreateTL {...}

Description

Creates a new profile for a TargetLink model.

Please note: The listed properties only show the TargetLink specific properties. Each of the properties listed in the "profileLoad" step also apply here.

DSL Command: btc.profileCreateEC {...}

Description

Creates a new profile for an EmbeddedCoder model.

Please note: The listed properties only show the EmbeddedCoder specific properties. Each of the properties listed in the "profileLoad" step also apply here.

DSL Command: btc.profileCreateSL {...}

Description

Creates a new profile for a Simulink model.

Please note: The listed properties only show the Simulink specific properties. Each of the properties listed in the "profileLoad" step also apply here.

DSL Command: btc.profileCreateC {...}

Description

Creates a new profile for supported ansi C-Code.

Please note: The listed properties only show the C-Code specific properties. Each of the properties listed in the "profileLoad" step also apply here.

DSL Command: btc.wrapUp {...}

Description

Publishes HTML reports and the JUnit XML report to Jenkins and closes BTC EmbeddedPlatform. The following optional settings are available:

DSL Command: btc.vectorImport {...}

Description

Imports test cases or stimuli vectors from the specified location. The following settings are available:

Possible Return values

DSL Command: btc.vectorExport {...}

Description

Exports test cases or stimuli vectors to the specified location. The following settings are available:

Possible Return values

DSL Command: btc.toleranceImport {...}

Description

Imports tolerance settings from the specified file. The following options are available:

Possible Return values

Possible Return values

DSL Command: btc.toleranceExport {...}

Description

Exports tolerance settings to the specified file. The following options are available:

Possible Return values

Possible Return values

DSL Command: btc.inputRestrictionsImport {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Imports Input Restrictions from the specified file. The following options are available:

Possible Return values

Possible Return values

DSL Command: btc.inputRestrictionsExport {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Exports Input Restrictions to the specified file. The following options are available:

Possible Return values

Possible Return values

DSL Command: btc.executionRecordExport {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Exports Execution Records to the specified directory. The following options are available:

Possible Return values

You can define whitelists and blacklists for scopes, folders and test cases. Everything will be merged resulting in a filtered set of test cases. Blacklists always have precedence over whitelists (i.e. if something is whitelisted and blacklisted it will be excluded).

Possible Return values

DSL Command: btc.rbtExecution {...}

Required License

EmbeddedTester (ET_BASE)

Description

Executes all functional test cases in the profile. A Test Execution Report will be exported to the "exportDir" specified in the "profileLoad" step. The following optional settings are available:

Filtering via White- & Blacklists

You can define whitelists and blacklists for scopes, folders and test cases. Everything will be merged resulting in a filtered set of test cases. Blacklists always have precedence over whitelists (i.e. if something is whitelisted and blacklisted it will be excluded).

Possible Return values

DSL Command: btc.vectorGeneration{...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes the engines for analysis and stimuli vector generation for structural coverage. The following optional settings are available:

Possible Return values

DSL Command: btc.backToBack {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes a Back-to-Back Test between the specified reference and comparison configuration (e.g. TL MIL vs. SIL). This step requires stimuli vectors or functional test cases in the profile. A Back-to-Back Test Report will be exported to the "exportDir" specified in the "profileLoad" step.

In automated scenarios the effort for manual reviews of frequently executed Back-to-Back tests can become quite high. The BTC plugin "ApplyFailedAccepted" deals with this challenge by applying your manually accepted deviations to all future Back-to-Back Tests as long as the deviating values are equal. For more information, please contact support@btc-es.de.

The following optional settings are available:

Possible Return values

DSL Command: btc.regressionTest {...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Executes a Regression Test between the current SUT and old Execution Records that have been saved. If no saved Execution Records are available, the vectors will only be executed on the current SUT and the Execution Records will be stored for a later Regression Test. This requires stimuli vectors or functional test cases in the profile. A Regression Test Report will be exported to the "exportDir" specified in the "profileLoad" step. The following optional settings are available:

Possible Return values

DSL Command: btc.rangeViolationGoals{...}

Required License

EmbeddedTester (ET_COMPLETE)

Required Plugin

Plugin RangeViolationGoals

Description

Adds Range Violation Goals to the profile which contribute to the Code Analysis Report and can be considered during vector generation (pll: "RVG"). The following optional settings are available:

Possible Return values

DSL Command: btc.domainCoverageGoals{...}

Required License

EmbeddedTester (ET_COMPLETE)

Required Plugin

Plugin DomainCoverageGoals

Description

Adds Domain Coverage Goals to the profile which contribute to the Code Analysis Report and can be considered during vector generation (pll: "DCG"). The following optional settings are available:

Possible Return values

DSL Command: btc.addDomainCheckGoals{...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Requires EP 2.9p0 or higher

Adds Domain Check Goals to the profile which contribute to the Code Analysis Report and can be considered during vector generation (pll: "VDCG;IDCG"). The following optional settings are available:

Possible Return values

DSL Command: btc.addInputCombinationGoals{...}

Required License

• EmbeddedTester (ET_COMPLETE)
• User Defined Coverage Goals Add-On (ET_UDEF_COVGOAL)

Description

Adds Input Combination Goals to the profile using the User Defined Coverage Goals feature. You can define certin value regions that shall be covered and this step will add all combinations of those values for all Inputs. Due to the big number of goals this can produce (#ValueRegions^#Inputs), this step is recommended for library functions or small modules and is not advised bigger systems such as software components.

• The goals created by this step will be listed in the "User Defined Coverage Goals" section of the Code Analysis Report.
• You can add the PLL "UDCG" to the btc.vectorGeneration step to generate vectors for these goals
• ZERO will only be applied if ZERO is part of the value range. Otherwise the minimum value will be selected.

Possible Return values

DSL Command: btc.defaultTolerances{...}

Description

Adds default tolerances for floating point and fixed point outputs / locals. The default setting aims to cover minimal deviations that occur due to the precisions involved. You can change in which cases the tolerances are applied via the applyTo option. When calling this step without parameters the following tolerances will be set:

• a relative tolerance of 1E-10%
• an abolute tolerance of 1E-10 for floating point outputs/locals
• an abolute tolerance of 2*LSB for fixed point outputs/locals

Possible Return values

DSL Command: btc.formalTest{...}

Required License

EmbeddedTester (ET_BASE) + Formal Test Add-On

Description

Executes a Formal Test based on all formal requirements in the profile. A Formal Test Report will be exported to the "exportDir" specified in the "profileLoad" step (and will be linked in the overview report). The following optional settings are available:

Possible Return values

DSL Command: btc.formalVerification {...}

Required License

EmbeddedValidator (EV)

Description

Executes all existing proofs in the profile and generates a Formal Verification Report. The following optional settings are available:

Possible Return values

DSL Command: btc.testExecutionReport{...}

Required License

EmbeddedTester (ET_BASE)

Description

Creates the Test Execution Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. If no reportName is specified the reports will be placed into a subdirectory in order to avoid multiple reports overwriting each other.

Possible Return values

DSL Command: btc.xmlReport{...}

Required License

EmbeddedTester (ET_COMPLETE)

Description

Creates the XML Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. Requires BTC Plugin for XMLReports. The following optional settings are available:

Possible Return values

DSL Command: btc.codeAnalysisReport{...}

Required License

EmbeddedTester (ET_COMPLETE) for B2B use case.

Description

Creates the Code Analysis Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. The following optional settings are available:

Possible Return values

In addition, this step creates a CSV file "BTCCoverageOverview_USECASE.csv" (with USECASE being B2B or RBT) which can be used by other Jenkins Plugins like the Plot Plugin to report coverage.

Example content of the CSV File:

Statement Coverage, Decision Coverage, MC/DC Coverage
100.0, 90.0, 91.98

Example of plots created by the Plot Plugin:

plot csvFileName: 'plot-b2b-codecoverage.csv', csvSeries: [[displayTableFlag: false, exclusionValues: '', file: "reports/BTCCoverageOverview_B2B.csv", inclusionFlag: 'OFF', url: '']], group: 'BTC Code Coverage Overview', style: 'line', title: 'B2B Code Coverage (Structural)', yaxis: 'Coverage Percentage'

DSL Command: btc.modelCoverageReport{...}

Required License

EmbeddedTester (ET_COMPLETE) for B2B use case

Simulink Coverage (formerly V&V)

Description

Creates the Model Coverage Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. The following optional settings are available:

Possible Return values

DSL Command: btc.finalWrapUp{...}

Description

Creates an Overall Report to provide an overview over multiple test projects. In cases where you have a set of models and are running tests in multiple test projects, this report will summarize the overall status and provide links to the *.epp files and the individual reports. Requires all projects to have been tested on the same agent (for inter-report linking).

This is a somewhat special step that requires some preparation:

• btc.collectProjectOverview{} needs to be called for each project before the wrapUp method is called. This stores the projects overview data.
• btc.wrapUp needs to be called for each project in order to generate the individual project reports (Test Automation report + detailed reports)
• btc.finalWrapUp{} will take the data collected previously to create the overview report. Since the report generation works based on the BTC EmbeddedPlatform report template engine, this requires an instance of BTC EmbeddedPlatform to be available. The finalWrapUp step will then close BTC EmbeddedPlatform.

node {
    // checkout changes from SCM
    checkout scm

    // one profile for each *.slx file
    def models = findFiles glob: '*.slx'
    // start EmbeddedPlatform and connect to it
    btc.startup {}

    // create a test project for each model (for Back-to-Back Test MIL vs. SIL)
    for (modelFile in models) {
      def modelName = "$modelFile".substring(0, "$modelFile".lastIndexOf('.'))
      // load / create / update a profile
      btc.profileCreateTL {
        profilePath = "${this.modelName}.epp"
        tlModelPath = "${this.modelFile}"
        matlabVersion = "2020b"
      }
      // generate stimuli vectors
      btc.vectorGeneration {
        pll = "STM, D, MCDC"
        createReport = true
      }
      // execute back-to-back test MIL vs. SIL
      btc.backToBack {
        reference = "TL MIL"
        comparison = "SIL"
      }
      /*
      * Collect the status information regarding the current
      * project for an overview report (this enables you to
      * get an OverviewReport when you call btc.finalWrapUp later)
      */
      btc.collectProjectOverview{}
      // generate reports for this project but skip archiving, etc.
      // keep EP alive
      btc.wrapUp {
        closeEp = false
        archiveProfiles = false
        publishReports = false
        publishResults = false
      }
    }

    // create the overall report based on the collectedProjectOverview data
    btc.finalWrapUp {
      path = 'reports'
    }
}

Possible Return values

DSL Command: btc.interfaceReport{...}

Description

Creates the Interface Report and exports it to the "exportDir" specified in the "profileLoad" / "profileCreate" step. The following optional settings are available:

Possible Return values

DSL Command: btc.getStatusSummary()

Required License

EmbeddedTester (ET COMPLETE)

Description

Retrieves a struct as a json text (see below) which can be passed on to external post-processing or evalulated directly (via readJSON of the Pipeline Utility Steps plugin):

• ProfileName
• Coverage
  • RBT
    • RequirementsCoverage
    • StatementCoverage
    • ConditionCoverage
    • DecisionCoverage
    • ModifiedConditionDecisionCoverage
    • RelationalOperatorCoverage
    • FunctionCoverage
    • FunctionCallCoverage
    • SwitchCaseCoverage
    • UserDefinedCoverageGoalsCoverage
    • DomainCoverageGoalsCoverage
    • RangeViolationGoalsCoverage
    • DivisionByZeroRobustnessCheck
    • DowncastRobustnessCheck
  • B2B
    • StatementCoverage
    • ConditionCoverage
    • DecisionCoverage
    • ModifiedConditionDecisionCoverage
    • RelationalOperatorCoverage
    • FunctionCoverage
    • FunctionCallCoverage
    • SwitchCaseCoverage
    • UserDefinedCoverageGoalsCoverage
    • DomainCoverageGoalsCoverage
    • RangeViolationGoalsCoverage
    • DivisionByZeroRobustnessCheck
    • DowncastRobustnessCheck
  • TestCases (list of objects with the following properties)
    • Name
    • Description
    • Result
    • VerifiedRequirements (list of strings)
    • Length
    • CreatedOn
    • CreatedBy

It is important that the EP process is closed to free the used resources and that the reserved port on the agent is released. To ensure this in an easy manner, the DSL command btc.handleError(errorMsg) is provided. It will safely close EP, release the used port and then call the Jenkins error(...) command with the given error message.

The BTC Migration Suite allows you to perform a fully automatic regression test between different Matlab or TargetLink versions. This makes it easy to document that the change of a tool version in a particular project does not influence the behavior of software components on model and code level.

Required License

EmbeddedTester (ET_COMPLETE)

DSL Command: btc.migrationSource {...}

Description

Creates a profile on the source configuration (e.g. old Matlab / TargetLink version), generates vectors for full coverage and exports the simulation results.

DSL Command: btc.migrationTarget {...}

Description

Creates a profile on the target configuration (e.g. newMatlab / TargetLink version), imports the simulation results from the source config and runs a regression test.

For both steps (migrationSource and migrationTarget) the following parameters are mandatory:

In Addition, you can add all other the parameters from the steps btc.profileLoad and btc.vectorGeneration if required.

The following example shows a Jenkinsfile for a complete migration of a TargetLink model. The labels that are specified for the node describe the resource requirements for each configuration and allow Jenkins to provide a suitable agent. Any data produced by the migrationSource step which is needed by the migrationTarget step will be automatically handled by Jenkins. This way, the agent for the target config (which is obviously a different one machine due to the different Windows versions) will have access to the simulation results from the source config required for the regression test.

Migration Suite Example

node ('Win7 && TL41 && ML2015b') {

    checkout scm
 
    stage ("Migration Source") {
        btc.migrationSource {
            matlabVersion = "2015b"
            tlModelPath = "source/model.mdl"
            tlScriptPath = "start.m"
        }
    }
}

node ('Win10 && TL43 && ML2017b') {

    checkout scm
 
    stage ("Migration Target") {
        btc.migrationTarget {
            matlabVersion = "2017b"
            tlModelPath = "target/model.mdl"
            tlScriptPath = "start.m"
        }
    }
}

In order to use the convenient pipeline syntax described above you need to add the BTC Plugin to Jenkins. This is very easy and can be done with the following steps:

On your Jenkins web UI go to Jenkins > Manage Jenkins > Manage Plugins

Click on the Available tab and search for btc-embeddedplatform

Select the plugin btc-embeddedplatform and install it

1. Once the plugin is installed new versions which are available will appear in the Updates section
2. Updating a plugin usually requires you to restart your Jenkins master

The BTC Pipeline plugin is now available in Jenkins and you can benefit from the convenient BTC pipeline methods described in the sections above. Enjoy!

If your Jenkins master can't access the internet you can manually download the plugin here and upload it to the server via the advanced section of the "Manage Plugins" page.