[Bug] Incoming Webhook broken

[cb3inco]

Description:

Since upgrading to 3.16+ it seems some Slack-compatible Incoming Webhooks seem to not work any more. There isn't much detail other than what I can provide below with screenshots and relevant logs.

Steps to reproduce:

1. Setup Channel and add user(s) to Channel
2. Setup Integration to use that channel
3. Copy Webhook URL to app to send webhooks to Rocket.Chat

Expected behavior:

Webhook post content to channel

Actual behavior:

No content shows

Server Setup Information:

• Version of Rocket.Chat Server: 3.18.1
• Operating System: Ubuntu 18.04
• Deployment Method: docker
• Number of Running Instances: 1
• DB Replicaset Oplog: Enabled
• NodeJS Version: v12.22.1
• MongoDB Version: 4.0.17 / mmapv1 (oplog Enabled)

Client Setup Information

• Desktop App or Browser Version: Latest Linux Electron App
• Operating System: Pop!_OS!

Additional context

Relevant logs:

I20210915-14:57:17.184(0) Integrations ➔ Incoming WebHook.info Post integration: SupportPal Dev
I20210915-14:57:17.186(0) Integrations ➔ Incoming WebHook.debug @urlParams: { integrationId: 'AfWf54ZnXqAJf6nWF', token: 'nN2stSRrpENZdRzD3birmM28qT9WGTeKTvScf7HBDFQhAoc8' }
I20210915-14:57:17.187(0) Integrations ➔ Incoming WebHook.debug @bodyParams: {}
I20210915-14:57:17.188(0) API ➔ debug Success { statusCode: 200, body: { success: true } }
I20210915-14:57:17.189(0) API ➔ info 10.200.X.XXX - aDFNiCQWGwZX7ATD9 [2021-09-15T14:57:17.188Z] "POST /hooks/AfWf54ZnXqAJf6nWF/nN2stSRrpENZdRzD3birmM28qT9WGTeKTvScf7HBDFQhAoc8" 200 - "undefined" "GuzzleHttp/6.5.5 curl/7.58.0 PHP/7.2.24-0ubuntu0.18.04.8" |
I20210915-14:57:23.792(0) API ➔ debug Success { statusCode: 200, body: { queue: [ [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], [Object], ... 900 more items ], success: true } }

I20210915-14:58:37.488(0) Integrations ➔ Incoming WebHook.info Post integration: SupportPal Dev
I20210915-14:58:37.489(0) Integrations ➔ Incoming WebHook.debug @urlParams: { integrationId: 'AfWf54ZnXqAJf6nWF', token: 'nN2stSRrpENZdRzD3birmM28qT9WGTeKTvScf7HBDFQhAoc8' }
I20210915-14:58:37.490(0) Integrations ➔ Incoming WebHook.debug @bodyParams: {}
I20210915-14:58:37.491(0) API ➔ debug Success { statusCode: 200, body: { success: true } }
I20210915-14:58:37.492(0) API ➔ info 10.200.X.XXX - aDFNiCQWGwZX7ATD9 [2021-09-15T14:58:37.491Z] "POST /hooks/AfWf54ZnXqAJf6nWF/nN2stSRrpENZdRzD3birmM28qT9WGTeKTvScf7HBDFQhAoc8" 200 - "undefined" "GuzzleHttp/6.5.5 curl/7.58.0 PHP/7.2.24-0ubuntu0.18.04.8" |
I20210915-14:59:00.239(0) SyncedCron ➔ info Starting "Federation".

[cb3inco]

If I use webhook.site and send the slack notification from my app, I get the following response:

{
  "text": "Admin posted a new reply to ticket #1821933.",
  "channel": "#supportpal",
  "username": "SupportPal",
  "link_names": 1,
  "unfurl_links": false,
  "unfurl_media": true,
  "mrkdwn": true,
  "icon_url": "https://support.domain.com/resources/assets/operator/images/favicon/apple-touch-icon-144x144.png?v=3.6.0",
  "attachments": [
    {
      "fallback": "#1821933 - tester",
      "text": "<https://support.domain.com/admin/ticket/view/14#message-27|#1821933 - tester>\n Admin replied: This is a test",
      "pretext": null,
      "color": "#3480B6",
      "mrkdwn_in": [],
      "image_url": null,
      "thumb_url": null,
      "title": null,
      "title_link": null,
      "author_name": null,
      "author_link": null,
      "author_icon": null,
      "fields": [
        {
          "title": "User",
          "value": "FirstName LastName",
          "short": true
        },
        {
          "title": "Department",
          "value": "Support",
          "short": true
        },
        {
          "title": "Status",
          "value": "In-Progress",
          "short": true
        },
        {
          "title": "Priority",
          "value": "Low",
          "short": true
        }
      ]
    }
  ]
}

[cb3inco]

I also can confirm that this webhook works correct in Slack, leading me to believe this issue is with Rocket.Chat. I'm surprised no one else is having this problem?

[cb3inco]

So it appears that the problem is that my Ticket system wasn't pushing the data with the content-type application/json defined. I see that there was a new parser implimented in 3.15 and this was about the time things went sideways. There doesn't appear any information on RC's documentation that data needs to be defined in it's type - and so it receives the data but doesn't do anything with it. I was able to fix this on the ticket system side and it works as expected now. It's probably a good security function for RC to require this, but it's not very clear that this is needed - as it worked as expected prior to RC 3.15.

[debdutdeb]

Hi @cb3inco, sorry for the late response.

Thanks for reporting this. And yes, content type has to be set to json otherwise incoming webhooks will not work anymore.

I'd like to keep the issue open for now and update the docs first before closing thiss permanently.

[Trans-IP]

I am seeing this also. The RocketChat log gives a "status":400 in the response from the Post line in my case but the full incoming webook is there a line above. Version 4.1.2 but I have seen it for a while now. Last time it worked was early October and I have upgraded several times since then.

[AllAstronauts]

#21126
Though for us, even with content type set correctly, integrations are broken, Fine in the phone app, same code (snap - pc brwosers) and nothing. Be nice to at least get some staff to acknowledge #21126

[Trans-IP]

The solution for me was based on this note: https://github.com/RocketChat/Rocket.Chat/releases/tag/4.0.0-rc.4

My user wasn't part of the target channel and that change was breaking. Did not notice it before. I added the user and everything started to work for me.

[AllAstronauts]

Alas, my user is a bot account that has full perms everywhere and is in every channel. Text pushes but the attachments do not. Double checked just now to be sure. My big worry is this is some super edge-case CORS thing with Snaps with no errors being logged (which is the case, there is nothing at all) - and I've already toggled and tested against all available RC settings. I'll get drunk one of these nights and start tearing into things upstream and see if there is anything. Also, I misspoke before, phone app will display the audio player if you send an audio attachment, but there is no file to play. I just assumed seeing the player there that all was well... Thanks for the heads-up though, one less variable to consider now.

EDIT: Also fair to note, snaps are still on the v3 series, not sure why we aren't getting the 4 series yet.

EDIT 2: Got off my posterior and pushed the Snap line to v4 series (duh...) and tested; no change.

EDIT 3: Ditched my afternoon to poke this stupid bear some more. audio_urls are just dropped when they go into chat when pushed via api or webhook integration. You can manually paste a web url to an audio file into chat and it will not only show you the link but will helpfully stick the audio player in as well, works great. But the aforementioned api and webhook pushes? audio_url just disappears. Confirm via console logs its all there, it just gets dropped when pushed to chat. It can't be a CORS thing as I can paste those external audio files manually in and those are fine (as I just said above)

EDIT 4: Looked at my last tests this morning. Curl, direct message into an integration - phone app audio is there and working, browser nothing. API call to postMessage, nothing in either phone app or browser.

🤷 I guess unless someone takes mercy I'll have to spin up a non-snap test install and see if this is working there...