Skip to content
This repository has been archived by the owner on Oct 23, 2024. It is now read-only.

Event subscriptions failing if one of them is not correct #3291

Closed
orimarti opened this issue Feb 22, 2016 · 1 comment
Closed

Event subscriptions failing if one of them is not correct #3291

orimarti opened this issue Feb 22, 2016 · 1 comment

Comments

@orimarti
Copy link

It seems that if a host is not correct, marathon fails to send the subscriptions to all the next nodes in the subscription list.
For example, if I have the next subscription list, where the second one is not correct:

{"callbackUrls":["http://10.2.2.216:8000/api/marathon/event_callback","10.2.2.215:8000/api/marathon/event_callback","http://10.2.0.168:8000/api/marathon/event_callback"]}

The first host on the list is receiving the events and the third one is not receiving it.

I'm seeing this logs on marathon:

[2016-02-22 11:44:48,335] INFO Removing 981253ab-59f8-442d-a78a-d779c176fe05 from list of running deployments (mesosphere.marathon.upgrade.DeploymentManager:marathon-akka.actor.default-dispatcher-53)    [47/1520]
[2016-02-22 11:44:48,335] INFO Updated groups/apps according to deployment plan 981253ab-59f8-442d-a78a-d779c176fe05 (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-15)
spray.http.IllegalUriException: Illegal URI reference, unexpected character ':' at position 10: 
10.2.2.215:8000/api/marathon/event_callback
          ^
[2016-02-22 11:44:48,336] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
[2016-02-22 11:44:48,337] INFO Deployment acknowledged. Waiting to get processed:
DeploymentPlan 2016-02-22T11:44:48.299Z NO STEPS
 (mesosphere.marathon.state.GroupManager:ForkJoinPool-2-worker-15)
[2016-02-22 11:44:48,337] INFO 10.2.1.31 - - [22/Feb/2016:11:44:48 +0000] "PUT //10.2.0.13:8080/v2/apps//hello HTTP/1.1" 200 92 "http://xxxxxxxxxxxxxxx:8080/ui/" "Mozilla/5.0 (X11
; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0" 4 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$b2a9f498:qtp697240075-30)
[2016-02-22 11:44:48,340] INFO Sending POST to:http://10.2.2.216:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)

        at spray.http.Uri$.fail(Uri.scala:775)
        at spray.http.parser.UriParser.complete(UriParser.scala:429)
[2016-02-22 11:44:48,344] INFO Sending POST to:10.2.2.215:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
[2016-02-22 11:44:48,346] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
        at spray.http.parser.UriParser.parseReference(UriParser.scala:60)
        at spray.http.Uri$.apply(Uri.scala:231)
        at spray.http.Uri$.apply(Uri.scala:203)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:36)
[2016-02-22 11:44:48,349] INFO Sending POST to:http://10.2.2.216:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:35)
        at mesosphere.marathon.event.http.HttpEventActor.post(HttpEventActor.scala:94)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at scala.collection.immutable.Set$Set3.foreach(Set.scala:145)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:78)
        at scala.util.Success.foreach(Try.scala:236)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
[2016-02-22 11:44:48,354] INFO Sending POST to:10.2.2.215:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
spray.http.IllegalUriException: Illegal URI reference, unexpected character ':' at position 10: 
10.2.2.215:8000/api/marathon/event_callback
          ^
[2016-02-22 11:44:48,360] INFO POSTing to all endpoints. (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)
[2016-02-22 11:44:48,361] INFO Sending POST to:http://10.2.2.216:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-53)

        at spray.http.Uri$.fail(Uri.scala:775)
        at spray.http.parser.UriParser.complete(UriParser.scala:429)
        at spray.http.parser.UriParser.parseReference(UriParser.scala:60)
        at spray.http.Uri$.apply(Uri.scala:231)
        at spray.http.Uri$.apply(Uri.scala:203)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:36)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:35)
        at mesosphere.marathon.event.http.HttpEventActor.post(HttpEventActor.scala:94)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at scala.collection.immutable.Set$Set3.foreach(Set.scala:145)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:78)
        at scala.util.Success.foreach(Try.scala:236)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-02-22 11:44:48,365] INFO Sending POST to:10.2.2.215:8000/api/marathon/event_callback (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-49)
spray.http.IllegalUriException: Illegal URI reference, unexpected character ':' at position 10: 
10.2.2.215:8000/api/marathon/event_callback
          ^
[2016-02-22 11:44:48,373] INFO 10.2.2.216 - - [22/Feb/2016:11:44:48 +0000] "GET //10.2.0.13:8080/v2/apps HTTP/1.1" 200 935 "-" "Go 1.1 package http" 2 (mesosphere.chaos.http.ChaosRequestLog$$EnhancerByGuice$$b2a9
f498:qtp697240075-32)

        at spray.http.Uri$.fail(Uri.scala:775)
        at spray.http.parser.UriParser.complete(UriParser.scala:429)
        at spray.http.parser.UriParser.parseReference(UriParser.scala:60)
        at spray.http.Uri$.apply(Uri.scala:231)
        at spray.http.Uri$.apply(Uri.scala:203)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:36)
        at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:35)
        at mesosphere.marathon.event.http.HttpEventActor.post(HttpEventActor.scala:94)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:86)
        at scala.collection.immutable.Set$Set3.foreach(Set.scala:145)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:86)
        at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:78)
        at scala.util.Success.foreach(Try.scala:236)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
        at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
@orlandohohmeier
Copy link
Contributor

@orimarti thanks a bunch for reporting this! I can confirm that Marathon fails to send events to the rest of the registered http endpoints if one of them is malformed.

Case 1

In this case, one of the registered endpoints (http://host:3535) doesn't respond to requests. Marathon retries to send the event data to this endpoint and proceeds with the rest as intended. All good.

Config

MARATHON_EVENT_SUBSCRIBER=http_callback
MARATHON_HTTP_ENDPOINTS=http://host:3434,http://host:3535,http://host:3636

Log

[2016-02-23 11:44:35,181] WARN Connection attempt to host:3535 failed in response to POST request to / with 1 retries left, retrying... (spray.can.client.HttpHostConnectionSlot:marathon-akka.actor.default-dispatcher-9)
[2016-02-23 11:44:35,181] WARN Connection attempt to host:3535 failed in response to POST request to / with 1 retries left, retrying... (spray.can.client.HttpHostConnectionSlot:marathon-akka.actor.default-dispatcher-9)
[2016-02-23 11:44:35,182] WARN Connection attempt to host:3535 failed in response to POST request to / with no retries left, dispatching error... (spray.can.client.HttpHostConnectionSlot:marathon-akka.actor.default-dispatcher-9)

Case 2

In the second case, one of the registered endpoints (host:3535) is malformed. Marathon fails to send the event data to the rest of the registered http endpoints due to an IllegalUriException.

Config

MARATHON_EVENT_SUBSCRIBER=http_callback
MARATHON_HTTP_ENDPOINTS=http://host:3434,host:3535,http://host:3636

Log

[2016-02-23 11:46:24,846] WARN Post DeploymentStepSuccess(DeploymentPlan 2016-02-23T11:46:24.363Z
 step 1:
   * Start(App(/sleepy, cmd="sleep 999")), instances=0)
 step 2:
   * Scale(App(/sleepy, cmd="sleep 999")), instances=1)
 ,DeploymentStep(Vector(ScaleApplication(AppDefinition(/sleepy,Some(sleep 999),None,None,Map(),1,1.0,128.0,0.0,,Set(),List(),List(),List(10000),false,1 second,1.15,3600 seconds,None,Set(),Set(),UpgradeStrategy(1.0,1.0),Map(),None,None,FullVersionInfo(2016-02-23T11:46:24.363Z,2016-02-23T11:46:24.363Z,2016-02-23T11:46:24.363Z)),1,None))),deployment_step_success,2016-02-23T11:46:24.744Z) to http://host:3434 resulted in error response 404 Not Found (mesosphere.marathon.event.http.HttpEventActor:marathon-akka.actor.default-dispatcher-8)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:34)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:34)
    at scala.util.Success.foreach(Try.scala:236)
    at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
    at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
 spray.http.IllegalUriException: Illegal URI reference, unexpected character ':' at position 11: 
 host:3535
     ^
    at spray.http.Uri$.fail(Uri.scala:775)
    at spray.http.parser.UriParser.complete(UriParser.scala:429)
    at spray.http.parser.UriParser.parseReference(UriParser.scala:60)
    at spray.http.Uri$.apply(Uri.scala:231)
    at spray.http.Uri$.apply(Uri.scala:203)
    at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:36)
    at spray.httpx.RequestBuilding$RequestBuilder.apply(RequestBuilding.scala:35)
    at mesosphere.marathon.event.http.HttpEventActor.post(HttpEventActor.scala:41)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:34)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1$$anonfun$apply$1.apply(HttpEventActor.scala:34)
    at scala.collection.immutable.Set$Set3.foreach(Set.scala:145)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:34)
    at mesosphere.marathon.event.http.HttpEventActor$$anonfun$broadcast$1.apply(HttpEventActor.scala:34)
    at scala.util.Success.foreach(Try.scala:236)
    at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
    at scala.concurrent.Future$$anonfun$foreach$1.apply(Future.scala:204)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I consider this as a minor issue as it shouldn't break "properly" configured setups but some kind of startup validation as well as exception handling would be nice. Pull request welcome!

janisz added a commit to janisz/marathon that referenced this issue Apr 15, 2016
@janisz
Fixes d2iq-archive#3291 | Event subscriptions validation
206969c
janisz added a commit to janisz/marathon that referenced this issue Apr 18, 2016
@janisz
Fixes d2iq-archive#3291 | Event subscriptions validation
5c28040
janisz added a commit to janisz/marathon that referenced this issue Apr 26, 2016
@janisz
Fixes d2iq-archive#3291 | Event subscriptions validation
3144bc9
aquamatthias added a commit that referenced this issue Apr 27, 2016
@aquamatthias
Merge pull request #3787 from janisz/event_subscriptions_validation
844f09c 
Fixes #3291 | Event subscriptions validation
@d2iq-archive d2iq-archive locked and limited conversation to collaborators Mar 27, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@orlandohohmeier @orimarti