This plugin helps you integrate a performance testing stage in your CI/CD pipeline for proactive issue detection and analysis of mainframe applications.

Along with the MAT Performance Benchmarking plugin by Broadcom, the plugin installation procedure automatically installs on your Jenkins machine the latest versions of the following components:

• Zowe CLI

• MAT Detect plug-in for Zowe CLI

• MAT Analyze plug-in for Zowe CLI

• Endevor SCM plug-in for Zowe CLI

The above listed components are required for the operation of the MAT Performance Benchmarking plugin by Broadcom.

If some of the required components are already installed on your Jenkins machine, verify that the current versions of the installed components comply with the minimum compatible versions that the MAT Performance Benchmarking plugin by Broadcom requires:

• Zowe CLI - version 6.25.0

• MAT Detect plug-in for Zowe CLI - version 1.0.1

• MAT Analyze plug-in for Zowe CLI - version 1.0.2

• Endevor SCM plug-in for Zowe CLI - version 5.7.2

After the successful installation of the MAT Performance Benchmarking plugin by Broadcom, the Performance Benchmarking option appears in the main Jenkins menu. By clicking the Performance Benchmarking option, you access the Configure Performance Benchmarking page. When you enter the Performance Benchmarking page, the plugin re-verifies whether you have all the necessary components installed.

The MAT Analyze profile enables the communication of the plugin with Mainframe Application Tuner (MAT) that is running on mainframe and ensures the analysis of the detected performance issue. To define the MAT Analyze profile, use the details of the MAT REST API server. You can create multiple profiles and switch between them by assigning one default profile.

Follow these steps:

1. In the Configure Performance Benchmarking window, click Define the MAT Analyze Profile. The Define the MAT Analyze Profile page opens. The page contains the list of all available MAT Analyze profiles.

2. In the Entries section, click Add and select MAT Analyze Profile. A new entry with the MAT Analyze profile fields appears.

3. Provide values for the following fields:

   • Profile Name
     Specify the name of your MAT Analyze profile that you create, for example, MATPROF1.

   • Protocol
     Select the protocol type that you defined for your MAT REST API server: http or https.

   • Host Name
     Specify the name or IP address that you defined for your MAT REST API server, for example, 127.0.0.0 or localhost.

   • Port Number
     Provide the port number that you defined for your MAT REST API server, for example, 8080.

   • Username
     Specify your mainframe username.

   • Password
     Specify your mainframe username.

   • Zowe Discoverable
     Select this option only if you have the Zowe API Mediation Layer properties configured in your MAT REST API server settings.

   • Default Profile
     Select this option to set the profile as default.

4. Click Save to apply all changes.
   A notification about the save and update process appears in the message section above all profile entries.

The MAT Analyze profile is defined.

To update an existing profile, provide new values in the respective fields and click Save. To delete a profile, click Delete, then click Save to apply the changes.

Ensure that you have one default MAT Analyze profile. If you delete a default profile or do not mark any profile as default, the plugin automatically assigns the first profile as the default one when you click save Save.

The MAT Detect profile enables the communication of the plugin with the Performance Management Assistant component (PMA) of MAT that is running on mainframe and ensures the automatic performance issue detection. To define the MAT Detect profile, use your PMA customization settings and your mainframe account details. You can create multiple profiles and switch between them by assigning one default profile.

Follow these steps:

1. In the Configure Performance Benchmarking window, click Define the MAT Detect Profile. The Define the MAT Detect Profile page opens. The page contains the list of all available MAT Detect profiles.

2. In the Entries section, click Add and select MAT Detect Profile. A new entry with the MAT Detect profile fields appears.

3. Provide values for the following fields:

   • Profile Name
     Specify the name of your MAT Detect profile that you create, for example, PMAPROF1.

   • Job Account Number
     Specify your z/OS TSO/E account information. Only numbers are allowed. For example, 123456789.

   • Job Class
     Specify your z/OS class information. Allowed values: alphanumeric characters (A-Z, 0-9). For example, A.

   • Message Class
     Specify the MSGCLASS parameter value to be assigned to the output class in the job log. The specified MSGCLASS value is used in all JCLs that PMA runs while you execute the commands. Values: alphanumeric characters (A-Z, 0-9). For example, A.
     Default: A.

   • Load Library
     Specify PMA loadlib information. To provide the loadlib name, refer to your PMA installation details (&HLQ.CEETLOAD).

   • PMA HLQ
     Specify the PMA high-level qualifier to access the KSDSALT, KSDSJOB, and KSDSEXC VSAM files to collect the necessary data. For example, PMA.V12.

   • Default Profile
     Select this option to set the profile as default.

4. Click Save to apply all changes.
   A notification about the save and update process appears in the message section above all profile entries.

The MAT Detect profile is defined.

To update an existing profile, provide new values in the respective fields and click Save. To delete a profile, click Delete, then click Save to apply the changes.

Ensure that you have one default MAT Detect profile. If you delete a default profile or do not mark any profile as default, the plugin automatically assigns the first profile as the default one when you click Save.

The z/OSMF profile enables the communication of the plugin with your instance of the IBM z/OS Management Facility. To define the z/OSMF profile, use your z/OSMF server details. You can create multiple profiles and switch between them by assigning one default profile.

Follow these steps:

1. In the Configure Performance Benchmarking window, click Define the z/OSMF Profile.
   The Define the MAT z/OSMF Profile page opens. The page contains the list of all available z/OSMF profiles.

2. In the Entries section, click Add and select z/OSMF Profile.
   A new entry with the z/OSMF profile fields appears.

3. Provide values for the following fields:

   • Profile Name
     Specify the name of your z/OSMF profile that you create, for example, ZOSPROF1.

   • Host Name
     Specify your z/OSMF server host name, for example, myhost.com.

   • Port Number
     Specify your z/OSMF server port number, for example, 443.
     Default: 443.

   • Username
     Provide your mainframe (z/OSMF) user name, which can be the same as your TSO login.

   • Password
     Provide your mainframe (z/OSMF) user name, which can be the same as your TSO password.

   • Reject Unauthorized
     Select this option to reject self-signed certificates.

   • Base Path
     Specify the base path for your API mediation layer instance. Specify this option to prepend the base path to all z/OSMF resources when making REST requests.
     Leave blank if you are not using an API mediation layer.

   • Encoding
     (Optional) Specify encoding for download and upload of z/OS data sets and USS files. If not specified, the default encoding is 1047.

   • Response Timeout
     (Optional) Specify the maximum amount of time in seconds the z/OSMF Files TSO servlet should run before returning a response. Any request exceeding this amount of time will be terminated and return an error. Allowed values: 5 - 600.

   • Default Profile
     Select this option to set the profile as default.

4. Click Save to apply all changes.
   A notification about the save and update process appears in the message section above all profile entries.

The z/OSMF profile is defined.

To update an existing profile, provide new values in the respective fields and click Save. To delete a profile, click Delete, then click Save to apply the changes.

Ensure that you have one default z/OSMF profile. If you delete a default profile or do not mark any profile as default, the plugin automatically assigns the first profile as the default one when you click Save.

The Endevor profile enables the communication of the plugin with Endevor® and enables you to remotely interact with your source code. Define this profile if you use Endevor® for managing your source code. To define the Endevor profile, use the details of your Endevor Web services session and credentials. You can create multiple profiles and switch between them by assigning one default profile.

Follow these steps:

1. In the Configure Performance Benchmarking window, click Define the Endevor Profile.
   The Define the Endevor Profile page opens. The page contains the list of all available Endevor profiles.

2. In the Entries section, click Add and select Endevor Profile.
   A new entry with the Endevor profile fields appears.

3. Provide values for the following fields:

   • Profile Name
     Specify the name of your Endevor profile that you create, for example, ENDVPROF.

   • Protocol
     Select the protocol type that you defined for your Cndevor® REST API server: http or https.

   • Host Name
     Specify the host name for your Endevor® Web services server, for example, endevorhost.com.

   • Port Number
     Specify your Endevor® Web services server port number, for example, 8080.

   • Username
     Specify your username for Endevor session.

   • Password
     Specify the password for your Endevor session.

   • Base Path
     Specify the base path for connecting to Endevor® REST API.
     Default: EndevorService/rest

   • Reject Unauthorized
     Select this option to reject self-signed certificates.

   • Default Profile
     Select this option to set the profile as default.

4. Click Save to apply all changes.
   A notification about the save and update process appears in the message section above all profile entries.

The Endevor profile is defined.

To update an existing profile, provide new values in the respective fields and click Save. To delete a profile, click Delete, then click Save to apply the changes.

Ensure that you have one default Endevor profile. If you delete a default profile or do not mark any profile as default, the plugin automatically assigns the first profile as the default one when you click save Save.

The PMA component of MAT performs a constant monitoring of all jobs on mainframe. To narrow down the monitoring activity only to the group of jobs that you want to focus on, you must specify the monitoring scope.

Note: Define the monitoring scope after you have successfully defined the communication profiles.

To define the monitoring scope, you include specific jobs or groups of jobs and exclude specific programs or groups of programs. You can create multiple entries for job inclusions and program exclusions. The resulting scope becomes a subject of permanent performance monitoring by the PMA component of MAT that is running on mainframe. From the resulting monitoring scope, you further include particular jobs for performance benchmarking while configuring specific pipelines and analyze the performance of the jobs under test.

Follow these steps:

1. In the Configure Performance Benchmarking window, click Define the Monitoring Scope.
   The Define the Monitoring Scope page opens.
   Note: When you access or update the Define the Monitoring Scope page, the plugin communicates with mainframe to retrieve the current monitoring scope details. The retrieval process may take time depending on your connection and mainframe response speed. The progress bar above the Entries section indicates the current status of data retrieval. Please wait for the process to complete before you update the monitoring scope details.

2. In the Entries section, click Add, and select one of the scope definition entry types:

   • Job Inclusion
     Specify the mandatory parameters of the job that you want to specifically include in the monitoring scope:

   • Job Name
     Provide the name of the job that you want to include in the monitoring scope, for example, ABCJOB. The maximum length is 8 characters. You can use wildcard characters * and _ to define an inclusion pattern for the job names. Examples:

     • Specify TEST* to include all job names that start with TEST.

     • Specify \_TEST to include all job names that end with TEST.

     • Specify \_TEST* to include all job names that contain TEST.

       Click **Advanced** to specify optional parameters of the excluded program:

   • Step Name
     Specify the step name of the job. The maximum length is 8 characters. The Step Name field supports the same use of wildcards as the Job name.

   • Proc Step
     Specify the procedure step of the job. The maximum length is 8 characters. The Proc Step field supports the same use of wildcards as the Job name.

   • Description
     Provide a description of the included job, for example, Test job included. The maximum length is 24 characters. The description that you provide appears in the inclusion entry in PMA.

   • Program Exclusion
     Specify the mandatory parameters of the program that you want to specifically exclude from the monitoring scope:

   • Program Name
     Provide the name of the program that you want to exclude from the monitoring scope, for example, ABCPGM. The maximum length is 8 characters. You can use wildcard characters * and _ to define an exclusion pattern for the program names. Examples:

     • Specify TEST* to exclude all program names that start with TEST.

     • Specify \_TEST to exclude all program names that end with TEST.

     • Specify \_TEST* to exclude all program names that contain TEST.

       Click **Advanced** to specify optional parameters of the excluded program:

   • Description Provide a description of the excluded program, for example, Backup program excluded. The maximum length is 24 characters. The description that you provide appears in the exclusion entry in PMA.

3. Confirm the changes using one of the following options:

   • Click Save to save all changes and exit.

   • Click Apply to save all changes and stay on the page.

You have defined the general scope of jobs to be monitored. You can define multiple entries for included jobs and excluded programs.

You can modify the monitoring scope at any time. To update an existing entry, provide new values in the respective fields and save the page. To delete an entry, click Delete, then save the page.

From the resulting scope, you add the specific job that you want to test for performance during the configuration of a particular pipeline. For more information, see #using-the-plugin[Using the Plugin].

You include the performance benchmarking tools in the Build stage of a Jenkins pipeline. MAT Performance Benchmarking by Broadcom monitors the performance of your changed application after the compilation of your updated source code. To perform the compilation step prior to performance benchmarking, you can include in the pipeline a compiler of your choice that is specific to your source code and SCM, or use the built-in Autogen step that compiles the job from a specific sandbox of Endevor®.

The following sequence of the pipeline steps within the Build stage ensures the proper performance benchmarking of your application upon each code change:

1. Compilation
   This step ensures the compilation of executables from the latest version of the source code in your SCM.+ You have the following options to include the latest executable in your pipeline:

   • Use a compiler of your choice that is integrated with your SCM. To include the compilation step in your Jenkins pipeline, refer to the documentation of the chosen compiler.

   • Add the built-in Autogen step that enables you to use the source code from your Endevor® sandbox. If you use Endevor®, you can include in the Build stage the Autogen step that is provided by the MAT Performance Benchmarking plugin by Broadcom. The Autogen step collects the source code from the specified Endevor® sandbox and compiles the job from this source after each update.

2. Performance Benchmarking
   In this step, the MAT Performance Benchmarking plugin by Broadcom runs the test job on mainframe, measures the performance KPIs of the executable compiled from the changed source code, and evaluates the results.

To integrate performance benchmarking in your Jenkins pipeline using the MAT Performance Benchmarking plugin by Broadcom, create a new Jenkins pipeline or edit an existing pipeline and perform the following actions:

1. Define the compilation step details

2. Specify the job for performance benchmarking

3. Define the recipients of the performance benchmarking reports

Trigger the pipeline execution using either of the following options:

• Automated execution
  Change the source code of the tested job in your SCM or in the IDE synchronized with the SCM. For automated execution, you need to configure a mainframe-specific pipeline that connects the local developer IDE, SCM, and Jenkins into a common automated solution. Refer to Performance Testing with MAT in DevOps CI/CD Pipelines in the MAT documentation for the details of the complete architecture of mainframe-specific CI/CD pipelines.

• Manual execution
  In the pipeline details menu, click Build Now.

If the pipeline execution results in an error, check the Jenkins log for details.

With the compilation and performance benchmarking steps properly configured, upon each pipeline execution the MAT Performance Benchmarking plugin by Broadcom automatically performs the following actions:

1. Compile the source code
   The plugin triggers the compiler that you specified, or uses the Endevor® sandbox details that you provided for the Autogen step.

2. Run the job under test
   The plugin submits the compiled tested job to run on mainframe and verifies whether the job falls within the defined monitoring scope.

3. Get performance KPIs
   The plugin obtains the performance metric values of the tested job from mainframe.

4. Check for performance alerts
   The plugin obtains from mainframe the information whether the current run of the tested job has raised a performance alert.

5. (If alerts found) Measure the alerted application
   Mainframe Application Tuner is automatically triggered to measure the alerted job on mainframe.

6. Send an email with performance analysis results for your review
   The plugin evaluates and summarizes the obtained results and sends you an email with the performance benchmarking report.

After the pipeline execution successfully completes, you can review the performance analysis metrics that the plugin measured and analyzed during the job run. The plugin evaluates the performance of mainframe jobs based on the following KPIs:

• CPU time

• Service Units (SRVU)

• Execute Channel Program count (EXCP)

• Elapsed time

During each job run, the plugin measures the current values of the key performance metrics of the job, compares the measurement results of the current run with the previous and average values (if available), calculates the deviations of each parameter, and analyzes the results. If the deviation of any metric exceeds 75% 2 times in a row, the plugin creates a performance alert. The performance alert triggers an immediate measurement of the job by Mainframe Application Tuner, which provides you with details for a quick, yet comprehensive analysis, and enables you to easily identify the cause of the performance degradation.

You can access the performance analysis results in the following locations: * Pipeline build log * Performance benchmarking reports