[Feature] Add Github source connector

[qqeasonchen]

Search before asking

• I had searched in the issues and found no similar issues.

Feature Request

Add Github source connector

Data Source:

• A GitHub Repo - Settings - Code and automation - Webhooks (can reuse existing HTTP Source Connector and substitute the GitHubProtocol of existing eventmesh-webhook module) May be superseded by [Feature] Unify HTTP Source Connector and eventmesh-webhook module #4825
• GitHub REST APIs (Timed query for updates and changes)

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct *

[Fungx]

Hi! I'm interested in this issue. But how can Github be a data source? Can you provide me with an example like the one in #4047 ?

[pandaapo]

Hi! I'm interested in this issue. But how can Github be a data source? Can you provide me with an example like the one in #4047 ?

Actually, it's similar to this #4047 (comment) . GitHub provides developers with REST API interfaces, GraphQL API interfaces, and Webhooks, all of which can provide rich and diverse data resources.

[Fungx]

Hi! I'm interested in this issue. But how can Github be a data source? Can you provide me with an example like the one in #4047 ?

Actually, it's similar to this #4047 (comment) . GitHub provides developers with REST API interfaces, GraphQL API interfaces, and Webhooks, all of which can provide rich and diverse data resources.

Thanks for your reply. I think that Webhooks are the most efficient way to retrieve data since they don't need any polling.

But one thing still confuses me. Webhooks require a Payload URL and will make POST requests after certain events have occurred. Is it proper to make the source connector an HTTP server to provide this URL? If we use this approach, perhaps we can implement an HTTP source connector and reuse it in the GitHub source connector.

[pandaapo]

Thanks for your reply. I think that Webhooks are the most efficient way to retrieve data since they don't need any polling.

But one thing still confuses me. Webhooks require a Payload URL and will make POST requests after certain events have occurred. Is it proper to make the source connector an HTTP server to provide this URL? If we use this approach, perhaps we can implement an HTTP source connector and reuse it in the GitHub source connector.

HTTP source is waiting for implementation. If you are interested, you can implement it first. https://github.com/apache/eventmesh/tree/master/eventmesh-connectors#connector-status

If you want to implement GitHub source first, you can first enable it to support REST API, or GraphQL API, or both. Then, after HTTP source is implemented by community, if it is suitable for GitHub source reuse, you can add Webhooks to GitHub source.

If you only want to implement GitHub source that supports Webhooks first, you can create a lightweight HTTP Server in GitHub source.

[Fungx]

Thanks for your reply. I think that Webhooks are the most efficient way to retrieve data since they don't need any polling.
But one thing still confuses me. Webhooks require a Payload URL and will make POST requests after certain events have occurred. Is it proper to make the source connector an HTTP server to provide this URL? If we use this approach, perhaps we can implement an HTTP source connector and reuse it in the GitHub source connector.

HTTP source is waiting for implementation. If you are interested, you can implement it first. https://github.com/apache/eventmesh/tree/master/eventmesh-connectors#connector-status

If you want to implement GitHub source first, you can first enable it to support REST API, or GraphQL API, or both. Then, after HTTP source is implemented by community, if it is suitable for GitHub source reuse, you can add Webhooks to GitHub source.

If you only want to implement GitHub source that supports Webhooks first, you can create a lightweight HTTP Server in GitHub source.

Ok, I want to work on HTTP source first ! I will post my design here in a few days.

[Fungx]

@pandaapo Hi, here is my design. Since I'm new to EventMesh, please let me know if there are any issues!

The HTTP source will be implemented with the native HttpServer.

In source-config.yml,we can configure the path and port. For example, the following configuration creates an API POST http://127.0.0.1:3000/test. We can call it in various ways (e.g. curl, GitHub Webhooks, etc.) to send messages.

connectorConfig:
    connectorName: httpSource
    path: /test
    port: 3000

The HTTP Source Connector converts the request body into a CloudEvent. For convenience, the request body should be provided in json format.

[pandaapo]

The HTTP source will be implemented with the native HttpServer.

In source-config.yml,we can configure the path and port. For example, the following configuration creates an API POST http://127.0.0.1:3000/test. We can call it in various ways (e.g. curl, GitHub Webhooks, etc.) to send messages.

connectorConfig:
    connectorName: httpSource
    path: /test
    port: 3000

The HTTP Source Connector converts the request body into a CloudEvent. For convenience, the request body should be provided in json format.

If you are willing, you can use Jetty, Tomcat, or Netty, which are more stable and efficient implementations of HTTP Server. Of course, if you insist on using the one provided by JDK, that's also okay.

So, this design looks fine to me. What do you think of this design of HTTP Source Connector? @xwm1992

Additionally, @Fungx plans to reuse this connector in the GitHub Source Connector that supports Webhooks. I think that's also fine. What do you think? @xwm1992

[pandaapo]

Additionally, @Fungx plans to reuse this connector (HTTP Source Connector) in the GitHub Source Connector that supports Webhooks. I think that's also fine. What do you think? @xwm1992

Considering that HTTP Source Connector has already been implemented by @Fungx , it is possible that @Fungx is preparing to implement GitHub Source Connector according to the above plan or is waiting for this reply. Therefore, I think I should draw your attention to this plan again. @xwm1992