How to Retrieve Info Using Jenkins Remote Access API



We can use REST-like Remote Access API in Jenkins to post or retrieve information in XML or JSON format.  General format of the URL is <Jenkins-URL>/<data-context>/api/xml|json.
Below are few examples:

1) Retrieve Build Result.
Below examples shows the build result in JSON format for job 'web-apps-build-pkgvalidator' and build number ''6'. Credential is passed using '--user <user-name>:<password>'
result is piped to 'jq' to pretty print.
Notes:



$> curl -X GET http://localhost:9090/job/web-apps-build-pkgvalidator/6/api/json --user ppoudel:<password> | jq

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  100  1765  100  1765    0     0  14120   0 --:--:-- --:--:-- --:--:-- 16192

{
  "_class": "hudson.maven.MavenModuleSetBuild",
  "actions": [
    {},
    {
      "_class": "hudson.model.CauseAction",
      "causes": [
        {
          "_class": "hudson.model.Cause$UserIdCause",
          "shortDescription": "Started by user Purna Poudel",
          "userId": "ppoudel",
          "userName": "Purna Poudel"
        }
      ]
    },
    {
      "_class": "hudson.plugins.git.util.BuildData",
      "buildsByBranchName": {
        "refs/remotes/origin/release/release-2.0.3": {
          "_class": "hudson.plugins.git.util.Build",
          "buildNumber": 6,
          "buildResult": null,
          "marked": {
            "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
            "branch": [
              {
                "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
                "name": "refs/remotes/origin/release/release-2.0.3"
              }
            ]
          },
          "revision": {
            "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
            "branch": [
              {
                "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
                "name": "refs/remotes/origin/release/release-2.0.3"
              }
            ]
          }
        }
      },
      "lastBuiltRevision": {
        "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
        "branch": [
          {
            "SHA1": "2c60eff4fef13e9346ae7c6b848efdc2fbf31026",
            "name": "refs/remotes/origin/release/release-2.0.3"
          }
        ]
      },
      "remoteUrls": [
        "https://pppoudel@bitbucket.org/pppoudel/pkgvalidator.git"
      ],
      "scmName": ""
    },
    {
      "_class": "hudson.plugins.git.GitTagAction"
    },
    {},
    {
      "_class": "hudson.maven.reporters.MavenAggregatedArtifactRecord"
    },
    {},
    {},
    {}
  ],
  "artifacts": [],
  "building": false,
  "description": null,
  "displayName": "#6",
  "duration": 37633,
  "estimatedDuration": 39130,
  "executor": null,
  "fullDisplayName": "web-apps-build-pkgvalidator #6",
  "id": "6",
  "keepLog": false,
  "number": 6,
  "queueId": 150,
  "result": "SUCCESS",
  "timestamp": 1546274196419,
  "url": "http://localhost:8080/job/web-apps-build-pkgvalidator/6/",
  "builtOn": "",
  "changeSet": {
    "_class": "hudson.plugins.git.GitChangeSetList",
    "items": [],
    "kind": "git"
  },
  "culprits": [
    {
      "absoluteUrl": "http://localhost:8080/user/ppoudel",
      "fullName": "Purna Poudel"
    }
  ],
  "mavenArtifacts": {},
  "mavenVersionUsed": "3.5.4"
}

2) Below example shows use of /api/xml with 'xpath' to get just the build status from the build report.

$> curl -X GET http://localhost:9090/job/web-apps-build-pkgvalidator/6/api/xml?xpath=/*/result --user ppoudel:<password>

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 24 0 24 0 0 102 0 --:--:-- --:--:-- --:--:-- 110

 <result>SUCCESS</result>

3) Getting build status using /api/json. The following example shows retrieving job name, build number, build status and timestamp.

$> curl -X GET http://localhost:9090/job/web-apps-build-pkgvalidator/6/api/json?tree=fullDisplayName,number,result,timestamp --user ppoudel:<password>

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   152  100   152    0     0   1216      0 --:--:-- --:--:-- --:--:--  1394

{"_class":"hudson.maven.MavenModuleSetBuild","fullDisplayName":"web-apps-build-pkgvalidator #6","number":6,"result":"SUCCESS","timestamp":1546274196419}

4) Retrieving all jobs under certain view:
Note: here I'm piping the result through 'jq' and 'grep', which is optional.

curl -X GET http://localhost:9090/job/Web/job/mobile-apps/view/mobile-apps/api/json --user ppoudel:<password> | jq | grep name

 "name": "mobile-apps-xyzmportal",
"name": "mobile-apps-holportal",
"name": "mobile-apps-tpl",
...
...
"name": "mobile-apps-bcs_jpj",


 5) Retrieving JUnit test summary report:

$> curl http://localhost:9090/job/web-apps-build-pkgvalidator/6/testReport/api/json?tree=failCount,skipCount,totalCount,urlName --user ppoudel:<password>

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   127  100   127    0     0    451      0 --:--:-- --:--:-- - 579

{"_class":"hudson.maven.reporters.SurefireAggregatedReport","failCount":0,"skipCount":0,"totalCount":20,"urlName":"testReport

6) Below steps can be used to retrieve the SonarQube analysis result using Jenkins' remote REST like API.
6.1) Get the taskId from the build providing build number
Note: http://localhost:9090 is Jenkins server URL.

curl -X GET http://localhost:9090/job/web-apps-build-pkgvalidator_sonarqube/6/api/json --user ppoudel:<password> | jq | grep ceTaskId

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5182  100  5182    0     0  18507      0 --:--:-- --:--:-- --:--:-- 20811
      "ceTaskId": "AWfXjftAinfFqLzOhqqe",

6.2)  Get the analysisId using taskID:
Note: http://localhost:8000 is SonarQube URL.

$> curl -X GET http://localhost:8000/api/ce/task?id=AWfXjftAinfFqLzOhqqe --user ppoudel:<password> | jq | grep analysisId

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   518  100   518    0     0   2770      0 --:--:-- --:--:-- --:--:--  3029
    "analysisId": "AWfXjgNNVNzkngEjXoPD",

6.3) Get the analysis report using analysisId:

$> curl -X GET http://localhost:8000/api/qualitygates/project_status?analysisId=AWfXjgNNVNzkngEjXoPD --user ppoudel:<password> | jq

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed
100  1438  100  1438    0     0   4608      0 --:--:-- --:--:-- --:--:--  4841

{
  "projectStatus": {
    "status": "ERROR",
    "conditions": [
      {
        "status": "OK",
        "metricKey": "new_maintainability_rating",
        "comparator": "GT",
        "periodIndex": 1,
        "warningThreshold": "1",
        "actualValue": "1"
      },
      {
        "status": "OK",
        "metricKey": "new_reliability_rating",
        "comparator": "GT",
        "periodIndex": 1,
        "warningThreshold": "1",
        "actualValue": "1"
      },
      {
        "status": "OK",
        "metricKey": "new_security_rating",
        "comparator": "GT",
        "periodIndex": 1,
        "errorThreshold": "1",
        "actualValue": "1"
      },
      {
        "status": "OK",
        "metricKey": "sqale_rating",
        "comparator": "GT",
        "warningThreshold": "3",
        "actualValue": "1"
      },
      {
        "status": "ERROR",
        "metricKey": "security_rating",
        "comparator": "GT",
        "errorThreshold": "1",
        "actualValue": "5"
      },
      {
        "status": "WARN",
        "metricKey": "reliability_rating",
        "comparator": "GT",
        "warningThreshold": "3",
        "actualValue": "5"
      },
      {
        "status": "ERROR",
        "metricKey": "blocker_violations",
        "comparator": "GT",
        "errorThreshold": "0",
        "actualValue": "180"
      },
      {
        "status": "WARN",
        "metricKey": "critical_violations",
        "comparator": "GT",
        "warningThreshold": "0",
        "actualValue": "3806"
      },
      {
        "status": "WARN",
        "metricKey": "major_violations",
        "comparator": "GT",
        "warningThreshold": "0",
        "actualValue": "2878"
      },
      {
        "status": "WARN",
        "metricKey": "coverage",
        "comparator": "LT",
        "warningThreshold": "80",
        "actualValue": "0.1"
      },
      {
        "status": "ERROR",
        "metricKey": "vulnerabilities",
        "comparator": "GT",
        "errorThreshold": "0",
        "actualValue": "107"
      }
    ],
    "periods": [
      {
        "index": 1,
        "mode": "previous_version",
        "date": "2018-12-15T15:11:09-0500",
        "parameter": "1.5.8-SNAPSHOT"
      }
    ],
    "ignoredConditions": false
  }
}

7) If you need to get all the configured projects (paginated output) in SonarQube you can use the following URL:
Note: http://localhost:8000 is SonarQube URL.


# First page:
$> curl -X GET http://localhost:8000/api/components/search?qualifiers=TRK&p=1

8) Or get detail of a specific project:
Note: http://localhost:8000 is SonarQube URL.

$> http://localhost:8000/api/components/show?key=<project-key&gt

For more information on Remote access API visit https://wiki.jenkins.io/display/JENKINS/Remote+access+API

You may be interested reading the following Jenkins related blog:
Jenkins Job Chain and How to Resolve Issue with Parameter Passing
Jenkins Pipeline - Few Troubleshooting Tips
Posted by 2 comments:
Labels: , ,

How I achieved my PSM I Certification

I'm writing this post not only to express my joy but also to share the techniques that I employed in order to prepare myself to pass the Professional Scrum Master™ (PSM) I test. I'm extremely delighted to have this certification. But don't get me wrong, employing the knowledge gained, in daily practice is more important than earning a certification!
Just to give you some background, I'm a DevOps specialist and work with different agile teams (using Scrum framework), as part of my daily job. I attended a Agile boot camp just last week, which also gave me a lot of insight on how to use Scrum framework in the real world.

In terms of preparation, I spent around 5 days including 3 days of boot camp. Below is the list of resources, I used to acquire the mastery of PSM I.

  1. Scrum Guide - in my humble opinion, this is the most important source of information. It is short but written very concisely and to the point. I read it just once top to bottom, but spent about four hours reviewing and taking notes (imaging I spent 4 hours and it (the 2017 version) only has 19 pages). I did spend another 15 minutes or so to review it again just before taking my real test. You can download this guide free from www.scrumguides.org
  2. Took free Open Assessments provided by scrum.org. Even though, I was targeting for Scrum Master, I took all available Open Assessments there just to get a bigger picture from the Scrum Master, the Product Owner, the Development Team, and the Scaled Scrum perspectives. I took some of these assessments twice to get close to a 100% passing mark and also took note of individual failed quizzes and reviewed it all again just before taking my real test.
  3.  Big thanks go to Mikhail Lapshin for maintaining such a wonderful site (https://mlapshin.com/index.php/scrum-quizzes/) dedicated to Scrum quizzes. In my opinion you find here sufficient (quantitatively and qualitatively) quizzes. Mikhail also maintains very good Scrum Questions page where he provides detail insight.  
  4. I also reviewed materials from www.volkerdon.com and did their quizzes. I must say that the site has high quality materials and quizzes. I repeated some of the quizzes here twice.
  5. I also downloaded and reviewed The Scrum Master Training Manual (The-Scrum-Master-Training-Manual-Vr1.61.pdf) produced by Management Plaza (https://mplaza.pm). As name suggests, it is written as a manual with nice details. I also noticed that the manual gives too much emphasis on word "Project" (as a noun). My understanding is that Scrum is more product oriented rather than project. Scrum Guide (2017 version) has just a brief touch on project, which states "... Each Sprint may be considered a project with no more than a one-month horizon. Like projects, Sprints are used to accomplish something...". So, make sure you are not confused when reviewing the manual.
  6. If you are a audio visual learner, Scrum Training Series modules from http://www.scrumtrainingseries.com/ are very good. These modules not only explain Scrum in real world like scenarios but also have good set of quizzes. I went through top five modules and did the quizzes. 
  7. Along with the Scrum Guide, I also reviewed the Nexus Guide and got knowledge about the Nexus framework. Nexus is one of the Scrum Scaling frameworks.
  8. I also studied some material (from the internet) to get better understanding on how to do Product Backlog Items (PBIs) ordering/prioritization, PBI work estimation etc. Evidence-Based Management Guide is a good read. I also gained some knowledge on consensus-based estimating technique using Planning Poker®, estimating using Story points etc.
In terms of giving the actual test, I chose the morning time (I heard that the site becomes slower during the day). I woke up Sunday morning around 6:00 AM, prepared myself a cup of coffee (I actually consumed two cups in two hours of time), did around 40 minutes of review before starting the actual test. Once you start the test, time management is very important. I was about to miss few, hadn't I rushed the last 30 minutes. I did mark few questions for later review (you have to manually make a note in separate notepad, as the system does not allow you to mark the question for review), but at the end, did not get enough time to go back and review again. It took me around 58 minutes to finish the test. So, instead of going back and reviewing, I just hit the finish button. I felt that I was confident in my answers, however, I was still nervous for my results.Yes, I got 96.3%.
   Hope this post is helpful to all of you - either considering to get certification or just trying to learn and use Scrum in your daily practice!

Posted by 18 comments:
Labels: , , ,

Jenkins Pipeline - Few Troubleshooting Tips



1)  ERROR: Error cloning remote repo ... fatal: I don't handle protocol 'ssh'. Detail stack trace below:

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress ssh://<gituser>@<githost>:<port>/<GIT_REPO_NAME>.git
+refs/heads/*:refs/remotes/origin/*" returned status code 128:
stdout:
stderr: fatal: I don't handle protocol '
ssh'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1996)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1715)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$300(CliGitAPIImpl.java:72)

Root cause and possible solution: The above issue can be caused by many things, but one you never suspect from the error message itself is that it is caused by some extra character(s) in GIT URL that you can't see on the Jenkins UI. Specifically if you have copied your GIT URL from another web page. Just delete the GIT URL in your pipeline script and retype it manually (instead of copying), it may solve the issue.

2) Error: java.lang.NoSuchMethodError: No such DSL method 'findFiles' found among steps...

 Root cause and possible solution: The DSL methods are related to the Jenkins DSL execution engine or one of the Plugins. In this particular case, make sure pipeline-utility-steps (pipeline-utility-steps.jpi) plugin is installed. For more info visit https://plugins.jenkins.io/pipeline-utility-steps

3) Error: java.lang.NoSuchMethodError: No such DSL method 'httpRequest' found among steps...

Root cause and possible solution: In this particular case, make sure httpRequest (http_request.hpi) plugin is installed. For more info visit https://jenkins.io/doc/pipeline/steps/http_request/

4) Error: java.lang.NoSuchMethodError: No such DSL method 'sshagent' found among steps...

 Root cause and possible solution: In this particular case, make sure sshAgent (ssh-agent.hpi ) plugin (http_request.hpi) plugin is installed. For more info visit https://wiki.jenkins.io/display/JENKINS/SSH+Agent+Plugin

5) Error: java.lang.RuntimeException: [ssh-agent] Could not find a suitable ssh-agent provider...
java.lang.RuntimeException: [ssh-agent] Could not find a suitable ssh-agent provider. at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.initRemoteAgent(SSHAgentStepExecution.java:175) at com.cloudbees.jenkins.plugins.sshagent.SSHAgentStepExecution.start(SSHAgentStepExecution.java:63) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source)

Root cause and possible solution: This error means that the sshAgent plugin is not able to locate the sshAgent provider in the path. Mostly, you may encounter this issue if you are running Jenkins on Windows. In order to resolve this issue on Windows, I downloaded the Portable GIT (https://git-scm.com/download/win) and put %PORTABLE_GIT_HOME%\usr\bin directory in my System path. This directory has ssh-agent.exe file. Make sure to launch new command window and restart the Jenkins.
You may be interested reading the following Jenkins related blog:
Jenkins Job Chain and How to Resolve Issue with Parameter Passing
How to Retrieve Info Using Jenkins Remote Access API

6) You are using Maven Pipeline Plugin and you get  [ERROR] Java heap space

[ERROR] Java heap space -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError

Root cause and possible solution: Because of insufficient Java heap space, it is running out of memory. The easiest solution is to use Maven JVM Options to specify the maximum (or min & max both) heap size that your process needs. You can use 'mavenOpts' within 'withMaven' step. Below is an example:

stage('Build') {
withMaven(
  mavenSettingsConfig: '9d2a7048-91b1-47a8-8788-be4b89b71128', jdk: jdk.toString(), maven: 'Maven 3.3.9', mavenOpts: '-Xmx2048m') {
    bat 'mvn clean package'
      }
}

Important: before increasing the heap size, make sure you have sufficient physical memory (RAM) available. If your Java process can not reserve specified heap space, then you may get error saying "Error occurred during initialization of VM. Could not reserve enough space for 2097152KB object heap". Which means either there is not enough physical memory available in the server for Java to reserve specified heap space (2 GB in this case) or other server/OS settings (like 32 JVM on Windows has around 1.6 GB heap limitation) preventing Java to reserve specified heap size. 
Posted by 1 comment:
Labels: , ,

Welcoming the New Year - the Polar Bear Way

Looking back, 2018 has been an amazing year, full of ups and downs and sweet and sours experiences. Today, I just wanted to say HAPPY NEW YEAR to everyone and wishing you all the best in 2019! How was your first day of 2019? I started my New Year by taking dip into the freezing water of lake Ontario organized by Toronto Polar Bear Club in support of Boost Child & Youth Advocacy Centre (CYAC). I thought, there was no better way to start the new year than participating in an event for charity.


People rushing to chilly water of lake Ontario


Boost CYAC is a Toronto based organization that answers calls when a child/youth under the age of 18 becomes a victim of sexual abuse, physical abuse, emotional abuse, or neglect and help them with their services.
Boost CYAC has a noble mission which states, "Boost CYAC is committed to eliminating abuse and violence in the lives of children, youth, and their families...We are dedicated to the prevention of child abuse and violence through education and awareness, and to collaborating with our community partners to provide services to children, youth, and their families." For more information, visit their website https://boostforkids.org

Thank you to everyone who supported me and donated to Boost CYAC. Good news! As per the latest figure from @TOPolarBearClub, today's participants have raised over $33,000.

Today's event was amazing and definitely something I can talk about for a long time. Participating in this event was a goal of mine for a long time. Two years ago, I had chance to read Scott Carney's famous book “What doesn't kill Us,” where he explained techniques based on cold therapy, breathing and commitment, developed by Wim Hof. I became an instant fan of it, and started practicing a lighter version of it. So, today was my chance to experience how my body reacts in natural freezing water and it did just great!

The day overall was chilly with a temperature of around -3°C and with wind chill it felt like -6°C. I went to this event with my and friend's families. The event was a buzz of excitement with people cheering and taking pictures. I was one of the first few people to touch the freezing cold water and experience the thrilling chill.

I'm hoping to make the polar bear dip as my New Year's Day tradition. If I'm able inspire you enough through this writing, I'll be seeing you all  in January 1st, 2020 at Sunnyside beach, Toronto. Stay warm until then.

Here are some of the pictures and a short YouTube Video that my family captured during the event.

After the dip



CityNews coverage

Posted by 2 comments:
Labels: ,
Subscribe to: Posts (Atom)