net.sf.json.JSONException: JSONObject["return"] is not a JSONArray.

[rbjorklin]

This seems to be similar but not the same as in #36. This occurred when running the same job that caused #55. This job of ours unfortunately seem really fickle since upgrading to the 2.x branch of the plugin.

ERROR: Build step failed with exception
net.sf.json.JSONException: JSONObject["return"] is not a JSONArray.
at net.sf.json.JSONObject.getJSONArray(JSONObject.java:1986)
at com.waytta.SaltAPIBuilder.performRequest(SaltAPIBuilder.java:265)
at com.waytta.SaltAPIBuilder.perform(SaltAPIBuilder.java:201)
at com.waytta.SaltAPIBuilder.perform(SaltAPIBuilder.java:161)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
at hudson.model.Run.execute(Run.java:1728)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:405)
Build step 'Send a message to Salt API' marked build as failure
Finished: FAILURE

[mchugh19]

Are you saying that when you run the job, you get the error in #55 and sometimes you get this error? If you were to kick off the same calls from curl, would you get back different results?

This error seems to indicate that salt did not return with a properly formatted result or came back with a result that did not include a "return" object in the array.

Do you have any way of sending over the debug logs where we could see what the salt-api came back with?

[rbjorklin]

The error in #55 have occurred a few times, this has only happened once but both are from the same job. I'll have to look into doing calls with curl.

For this run there are no debug logs even though I followed the instructions as described on the plugin page.

[mchugh19]

Sounds like a situation where the same job may be producing different output. This would indicate an error in salt. Once we get back the debug output with the actual results from the salt-api, I can adjust the parsing to correctly interpret it.

Here's an example using curl to help troubleshoot: https://gist.github.com/mchugh19/61cd93440660ccbfc709

[mchugh19]

Can you send over examples of saltapi json output that is being marked by Jenkins as a failure?

[rbjorklin]

@mchugh19 This error has not returned and I've been unable to reproduce it. I'll close this issue until the problem resurfaces.

[rbjorklin]

This issue is back with a vengeance. Currently all our salt-jobs are failing with the output from the initial message. Started happening seemingly without reason during our nightly jobs using version 2.0.1. Upgrading to 2.1.0 did not help unfortunately. Could possibly be related to an automatic upgrade of Jenkins to 2.52 or 2.53.

EDIT: A colleague upgraded salt from 2016.11.1 to 2016.11.3. Sounds like a likely culprit.

EDIT2: The jobs still work when submitted from command line.

[rbjorklin]

Here's the http log:

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
ProxySelector Request for https://salt.example.com:8000/login

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
Looking for HttpClient for URL https://salt.example.com:8000/login and proxy value of DIRECT

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
Creating new HttpsClient with url:https://salt.example.com:8000/login and proxy:DIRECT with connect timeout:5000

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
Proxy used: DIRECT

Apr 03, 2017 6:30:13 AM FINE sun.net.www.protocol.http.HttpURLConnection
sun.net.www.MessageHeader@304d188d9 pairs: {POST /login HTTP/1.1: null}{Accept: application/json}{Content-Type: application/json}{Cache-Control: no-cache}{Pragma: no-cache}{User-Agent: Java/1.8.0_112}{Host: salt.example.com:8000}{Connection: keep-alive}{Content-Length: 70}

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
KeepAlive stream used: https://salt.example.com:8000/login

Apr 03, 2017 6:30:13 AM FINE sun.net.www.protocol.http.HttpURLConnection
sun.net.www.MessageHeader@4127cd1112 pairs: {null: HTTP/1.1 200 OK}{Content-Length: 187}{Access-Control-Expose-Headers: GET, POST}{Vary: Accept-Encoding}{Server: CherryPy/3.2.2}{Allow: GET, HEAD, POST}{Access-Control-Allow-Credentials: true}{Date: Mon, 03 Apr 2017 06:30:13 GMT}{Access-Control-Allow-Origin: *}{X-Auth-Token: 1111111111111111111111111111111111111111}{Content-Type: application/json}{Set-Cookie: session_id=1111111111111111111111111111111111111111; expires=Mon, 03 Apr 2017 16:30:13 GMT; Path=/}

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
ProxySelector Request for https://salt.example.com:8000/

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
Looking for HttpClient for URL https://salt.example.com:8000 and proxy value of DIRECT

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
KeepAlive stream retrieved from the cache, sun.net.www.protocol.https.HttpsClient(https://salt.example.com:8000/login)

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
Proxy used: DIRECT

Apr 03, 2017 6:30:13 AM FINE sun.net.www.protocol.http.HttpURLConnection
sun.net.www.MessageHeader@1609a8ce10 pairs: {POST / HTTP/1.1: null}{Accept: application/json}{Content-Type: application/json}{X-Auth-Token: 1111111111111111111111111111111111111111}{Cache-Control: no-cache}{Pragma: no-cache}{User-Agent: Java/1.8.0_112}{Host: salt.example.com:8000}{Connection: keep-alive}{Content-Length: 136}

Apr 03, 2017 6:30:13 AM FINEST sun.net.www.protocol.http.HttpURLConnection
KeepAlive stream used: https://salt.example.com:8000

Apr 03, 2017 6:30:13 AM FINE sun.net.www.protocol.http.HttpURLConnection
sun.net.www.MessageHeader@2838577712 pairs: {null: HTTP/1.1 400 Bad Request}{Content-Length: 731}{Access-Control-Expose-Headers: GET, POST}{Cache-Control: private}{Vary: Accept-Encoding}{Server: CherryPy/3.2.2}{Allow: GET, HEAD, POST}{Access-Control-Allow-Credentials: true}{Date: Mon, 03 Apr 2017 06:30:13 GMT}{Access-Control-Allow-Origin: *}{Content-Type: text/html;charset=utf-8}{Set-Cookie: session_id=1111111111111111111111111111111111111111; expires=Mon, 03 Apr 2017 16:30:13 GMT; Path=/}

and here's the saltstack logs:

Apr 03, 2017 6:30:13 AM FINE com.waytta.saltstack
Sending JSON: {"client":"local_batch","batch":"1","tgt":"host.example.com","expr_form":"list","fun":"state.sls","arg":"application.host-scraper","kwarg":{}}

[mchugh19]

The test master I'm working with is also 2016.11.3, and I've not seen this yet. In the http logs, it looks like all is fine, up until the master replies with 400. Do some local_batch jobs succeed, or are all failing every time? Is the application.host-scrapper a long running state?

With the json sent from the saltstack logs, I am able to see this behavior.

[rbjorklin]

Since this started happening all local_batch jobs are failing all the time. application.host-scraper is not a long running jobs, finishes in less than 60 seconds when run from command line and explodes as soon as it's started when starting from Jenkins.

I just tried running with local instead of local_batch and that worked so it seems to be local_batch specific.

[mchugh19]

Ah ha! Looks like batch was removed from the salt-api, until the batch api can be manged by eauth.
See saltstack/salt#40477

Adding the two lines to __init__.py from saltstack/salt@3d8f3d1 should patch you back up until upstream corrects.

[rbjorklin]

Nice find @mchugh19! We ended up wrapping the state previously run as local_batch under a runner instead. Since this issue lies with Salt and not this plugin feel free to close it.

[mchugh19]

Glad you got a workaround. Head's up, local_batch was removed in 2016.11.2

[kedare]

@rbjorklin We are having the same issue, how would you convert a local_batch job into a runner ?
I tried to convert my task to a simple "local" but I'm getting the same error

Thanks

[rbjorklin]

@kedare Our use case was pretty simple so we just looped over a provided list of minions like this:

{% for minion in pillar['data']['minions'] %}

app-{{ minion }}:
  salt.state:
    - tgt: {{ minion }}
    - sls: application.deploy
    - pillar:
        data:
          app: {{ pillar['data']['app'] }}
          version: {{ pillar['data']['version'] }}
    - failhard: True

{% endfor %}

[kedare]

Hmm ok, I need to check how that work, I'm quite new to Salt, in my case I suppose as I'm just applying a state, I would just have to run sls: state.apply and in pillar.data a name attribute with the state name ? (Basically I need to do the equivalent of a : salt "*staging*" state.apply ci.deploy.app

[mchugh19]

local_batch will hopefully return soon saltstack/salt#40598

[mchugh19]

@kedare Looks like local_batch support for saltapi cherrypy is included in the 2016.11.4 release.

[kedare]

Does the Jenkins plugin need reworking or it should work as it with the "new" local_batch ?

[rbjorklin]

Just a fair warning that the salt-api seems completely broken in 2016.11.4 for us. We rolled back.
saltstack/salt#40845

[kedare]

Any update about this ?
Looks like the original issue has been fixed

[rbjorklin]

@kedare it's probably working in 2016.11.5 but I haven't tried it.