Contributors graph

[ShalokShalom]

Implement contributor graphs: https://github.com/go-gitea/gitea/graphs/contributors

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[bkcsoft]

Is there a good graph-lib? In my opinion this can be rendered and cached server-side

[akuzia]

Any progress?

[ghost]

would be nice to have 🎉

[linusg]

I would like to start working on this feature, if no one is already on it (yeah @lafriks, I learned my lesson, +1 is not constructive 😉).

I would probably need some assistance every now and then, e.g. on how to decide about server or client side rendering, what charting library to use etc.
I also basically don't know any Go but have good frontend knowledge so it should work, and everything has a first time, also I wanted to dive into hacking on Gitea some while ago 😄

[linusg]

Let's start by taking apart existing solutions to identify required data and possible data structure.

GitHub

API endpoint for contibutions data is https://github.com/<owner>/<repo>/graphs/contributors-data.

The returned JSON data is basically a list of objects (each representing one contributor) sorted least contributions first, most contributions last:

[
  { ... }, // User with least contributions
    ...
  { ... }, // User with second most contributions
  { ... }  // User with most contributions
]

The structure is roughly similar to the one documented here and looks like this:

{
  "author": {
    "id": 12345,
    "login": "octocat",
    "avatar": "https://avatars3.githubusercontent.com/u/12345?s=60&v=4",
    "path": "/octocat",
    "hovercard_url": "/hovercards?user_id=12345"
  },
  "total": 123,
  "weeks": [
    // First week in which the repo existed
    {
      "w": 1391904000,
      "a": 6898,
      "d": 77,
      "c": 10
    },
    // Second week in which the repo existed
    {
      "w": 1392508800,
      "a": 2437,
      "d": 439,
      "c": 6
    },
    ...
    // Current week
    {
      "w": 1538265600,
      "a": 0,
      "d": 0,
      "c": 0
    }
  ]
}

Each member of the "weeks" array is contructed has the following attributes:

• w - Start of the week, given as a Unix timestamp.
• a - Number of additions
• d - Number of deletions
• c - Number of commits

All that information is used to build these cards:

The big contributions graph obviously can be built by adding up the stats from each user of a week n (0 <= n <= weeks since the repo exists) and plotting the cumulative value for each week.

GitLab

GitLab CE is Open Source, so we have the relevant files:

• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/controllers/projects/graphs_controller.rb
• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/graphs/commits.rb
• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/views/projects/graphs/show.html.haml
• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/pages/graph.scss
• https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/vue_shared/components/bar_chart.vue

API endpoint is https://gitlab.com/<owner>/<repo>/graphs/master?format=json.

The returned JSON data is much simpler:

[
  { ... }, // Latest commit
  { ... }, // Second latest commit
    ...
  { ... }, // First commit
]

Each member of the array represents one commit, sorted latest commit first, initial commit last. The structure looks as follows:

{
  "author_name": "Some User",
  "author_email": "some.user@example.com",
  "date": "2018-10-02"
}

If a user made multiple commits on the same day, there will simply be duplicate entries with the same user information and date, one for each commit.

The per-user tiles will contain less information than on GitHub, the plotting is done by taking the number of commits for one day, X-Axis is time, Y-Axis number of commits. That is done for both the whole repo (ignoring username) and each user (taking all commit entries for a specific user on a specific day).

---

In both cases rendering is done client side, which has the great advantage of being able to build dynamic charts with zooming.

[linusg]

If it works with your general workflow here, I'd be fine with being assigned to this issue.

---

Some more thoughts on this. Constructive feedback is of course highly appreciated!

Placing the page link on the UI

That should work fine, no need to restructure anything for now.

Speaking of links, the page should probably live at https://git.example.com/<owner>/<repo>/contributors, that's how all the other links up there work.

Another idea, which I do not prefer, is putting the contributor graph(s) on the Activity page.

I did some DOM editing:

I chose octicon-organization as the icon, octicon-graph might work as well.

Now some quick CSS editing on the GitHub contributors chart for Gitea and merging the images:

That's a very rough idea of how it may look like, not taking individual per-user charts into account.

[ShalokShalom]

Looks wonderful ^-^

[lunny]

@linusg great! Go ahead!

[linusg]

@lunny I'm kinda confused right now: Who is @Morlinest and what role will he play in this issue?

[Morlinest]

It is probably a mistake or maybe he has some secret plans with me :D

[lunny]

@linusg @Morlinest :( sorry. A mistake like what @Morlinest said. I want to assign this issue to @linusg but I found it cannot be assigned to non-maintainers and issue poster.

[linusg]

Ok, thanks for clarification 😄

[Morlinest]

Oh, so I will have to do it now :D

[linusg]

Short heads-up for those interested: I wanted to work on this during the Christmas holidays, but couldn't find much time. I've created the basic stuff (page, routing etc.) and plan to continue working on it!

[ShalokShalom]

Thanks a lot ^-^

[linusg]

Ok folks, yet another update. I managed it to get to this state:

---

Click to expand:

Gitea vs GitHub (real-life example)

Dark

Details:

• No data exposed over HTTP API, graphs are rendered to SVG (using https://github.com/wcharczuk/go-chart) on the server. This is really performant and keeps things simple.
• Sorting by number of commits, additions and deletions
• The UI is "slightly" based on GitHub 😄

Issues left:

• Contributors, which are not in the gitea DB (e.g. because the repo was imported) will not show up. ✅
• Performance issues with bigger repositories. Edit: not performance, but HTTP 500 with the message "http: multiple response.WriteHeader calls" written to the console. Might be just me being a golang n00b.
• Removing the AM/PM stuff from the X-Axis (can be easily done via custom formatter) ✅
• Fix the Y-Axis scale of user graphs, 1 commit should be half the height as 2 commits ✅
• Proper dark theme support (CSS for above was tweaked in the dev tools) ✅

Possible enhancements:

• Stats are for the master branch (hardcoded), this can be easily changed and exposed as an UI control

Ideas for changes and improvements welcome - I'm exited so far! I fear the upcoming code review though 😄

[linusg]

Sooo... here we go! Now it's the time for some external input, so please see below the images.

(gitea repo taken from GitHub)

Let me explain:

• Users who are not in the gitea users DB will be shown, but with no link to the profile, obv. Stats are calculated by Username (available is only "name" and "email" per each commit), that's why there's "Unknown", "Unknwon" and "无闻" vs only "Unknwon" in GitHub: The information, that this is all the same user is lost when cloning/importing the repo. I guess that's the best option available, thoughts?

• GitHub compiles stats per week, I went with daily stats. Should this be changed?

  That's the reason why the Y-Axis on GitHub ends at ~150 [commits per week] and the Gitea one at 52 [commits per day]. Also it makes the chart on Gitea appear with more "spikes". (interpolation isn't available as well)

• GitHub excludes merge commits from the stats, I didn't implement anything of this kind (and don't know how hard distinguishing one from a normal commit would be). Do we want this feature?

• Do you wish a separate color for the per-user charts?

• What else do you think can be improved?

Performance:

I fixed all of the issues noted in my last post, and I'm back to some performance issues. All stats from my dev machine:

The contributors page of the Gitea blog repo takes 1.1s to load, which is probably fine (Page: 1090ms Template: 7ms)

The one for the gitea main repo took 1min 14s and reports Page: 74443ms Template: 47ms. It has nine years of history and almost 7k commits, though.

Possible improvements: the gitea repo contributor page ends up with 602 user cards, I believe GitHub cuts off at 100. See https://github.com/go-gitea/gitea/graphs/contributors.

What do you think about that? As the charts become less useful with very few changes/commits because of the fixed Y-Axis, should we either only show top X contributors or maybe don't generate charts for more than e.g. 100 contributors?

Since the whole commit history will be traversed each time the page is visited, we can probably as well improve the situation by caching the stats. No clue if that makes sense and how implementation would look like.

I had to clear the ServiceWorker's cache for the changed CSS files to show up (normal cache refresh wouldn't work). What do I have to do here so it works OOTB?

More screenshots, click to expand

[lunny]

@linusg Great job!!! How about to let the work as a cronjob when the repository is big(i.e. over 1000 commits)? It can be run one or more days according the configuration. I think top 100 is enough, otherwise pagination is better.

[yasuokav]

@linusg

• Stats are for the master branch (hardcoded), this can be easily changed and exposed as an UI control

Maybe you can use the default branch option instead of creating another option.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[linusg]

Nope, still working on this - someone might remove the stale label!

[bryanpedini]

Nope, still working on this

That's good news, hoping to see this feature coming soon!
Hyped to see charts and graphic data representation everywhere...

[sahinakkaya]

I will try to implement this since no one seems to be working on it. I have zero experience with go but I think I will figure it out. Let's see how it goes 🤞

[sahinakkaya]

I decided to use Chart.js to draw the graphs. Right now gitea repository looks like this:

There are still some work to do but before continuing I wanted to ask the maintainers if the Chart.js dependency is OK or not. If it is not ok which library should I use?

Any other feedback is also greatly appreciated :)

[sahinakkaya]

@wxiaoguang @lunny @techknowlogick @silverwind

[lunny]

@wxiaoguang @lunny @techknowlogick @silverwind

Since it's under MIT license, I think it's acceptable. Is that library the best choice to draw the graph? But of course PRs are always welcome.

[sahinakkaya]

@lunny These were my requirements:

• Draw an area graph
• Draw on the client side
• interpolate data to smooth out the edges
• support zoom, pan etc.

Of course none of these requirements are hard requirements but they are nice to have. Chart.js supports all of them (last one available via an extension) If anyone have a simpler alternative in mind, I can take a look at that too.

[sahinakkaya]

Another update (Help needed):

I copied the Activity page and replace the content with graphs. I think it looks OK except for the top section where it shows dates and period selection menu (they are left-overs from Activity page).

I obviously need to replace that top right menu with contribution type selection (commits, additions, deletions) however, I don't have any idea about how to handle it in a single page. The Activity page's menu redirects you to other endpoints and makes relevant calls to get the data.

In my case, I already have the data and I just need to pass the selected option to my Vue component which draws the graphs.

I have zero experience with go

I also don't know any Vue either so... I think I need a little bit of help here.

Best way to help: Show me an example from this repository where dynamically selected option is passed to Vue. That's how I implemented all of this, by replicating, without knowing the details of these languages anyway :D

[lunny]

If we want to introduce a frontend chart library but not two for this requirement and future requirements, I think we have some options below:

• chartjs https://www.chartjs.org
• highcharts https://highcharts.com
• echarts https://echarts.apache.org
• d3js https://d3js.org
• plotyly https://plotly.com
• chartist-js https://github.com/gionkunz/chartist-js
• ApexChart https://github.com/apexcharts/apexcharts.js

[sahinakkaya]

In my previous comment, the graphs are already drawn with Chart.js but this is also a nice list @lunny. I examined them a little bit and I might consider switching to ApexCharts because they have exactly what we need: https://github.com/apexcharts/apexcharts.js#a-little-more-than-the-basic

A little more than the basic

You can create a combination of different charts, sync them and give your desired look with unlimited possibilities. Below is an example of synchronized charts with github style.

I still need help with the issue I mentioned in my previous comment though. (Maybe @wxiaoguang could help me)

[fnetX]

Thank you for the work on the charts. I suggest to keep the current information easily available, though. Like, make the contributors clickable easily, but also retain access to an overview of merged pull requests etc. I use this quite frequently to check what kind of activity a project has: Basic maintenance? New Features? Only users reporting bugs and no actual progress?

Another suggestion: Gitea already counts not only commits / code contributions in some places like the heatmap, and more and more people think this is the right direction for the software community, some even move over because of this. Maybe we can have an activity overview which does not only count code contributions, but also draw different kinds of contributions (in different colour), so you can check at a glance how the amount of commits, issues and pull requests change over time. Maybe even separated in opened / fixed ...

[silverwind]

Can't really comment on chart libraries but as long as you're lazy-loading them with await import only on pages that use them, it should be fine.

apexcharts does look pretty well-suited to the task yes.

[wxiaoguang]

I still need help with the issue I mentioned in my previous comment though. (Maybe @wxiaoguang could help me)

I am glad to help (if it is in my knowledge), but at the moment, I haven't got the point, the questions (experience with Go/Vue) seems general.

---

Show me an example from this repository where dynamically selected option is passed to Vue.

There are some Vue components, the data could be passed by either:

• Backend rendered "pageData", it's somewhat hacky but it does work for traditional Golang template + modern Vue
• Make Vue components use "fetch" (it is the preferred approach IMO, although it hasn't been widely used in this project)
  • If the "dynamically" in the question means it must use "fetch/AJAX", then this is the only approach IMO.

[sahinakkaya]

I suggest to keep the current information easily available, though. Like, make the contributors clickable easily...

@fnetX thanks for the suggestions. I created an endpoint which returns contributors' commits, additions and deletions for each week. I think this data can also be used to generate code frequency graph. I am not planning to implement it for this PR though as it is out of scope.

as long as you're lazy-loading them with await import only on pages that use them, it should be fine.

@silverwind, I don't do any await import stuff but I think there is no performance issue because I am doing same thing as what Activity page does. A pull request is coming soon anyway :)

And @wxiaoguang, is there a way to pass some property with pageData and not trigger a re-render of whole page? Fetching data is expensive so in Go file, I fetch all the data I need and pass it to Vue via pageData. When user change selection, I don't want to fetch data again.

Here, I have all the data for commits, additions and deletions (1 expensive call). If "commits" option is selected I draw graphs with commits data. When user change selected option to something else I just need to change the data I am feeding to graphs. Hope this makes sense and clears up the confusion.

[wxiaoguang]

And @wxiaoguang, is there a way to pass some property with pageData and not trigger a re-render of whole page? Fetching data is expensive so in Go file, I fetch all the data I need and pass it to Vue via pageData. When user change selection, I don't want to fetch data again.

If you want to get some new data without re-rendering, then I think the only way is "fetch"

[silverwind]

@silverwind, I don't do any await import stuff but I think there is no performance issue because I am doing same thing as what Activity page does. A pull request is coming soon anyway :)

Definitely do it. If you don't code-split off the chart library with await import, our JS payload for all pages will grow by its size, e.g. 200kb+. There are numerous examples in the codebase for such code-splits.

[sahinakkaya]

I opened a pull request: #27882. It would be nice if I can get some review and help from folks following this thread.

@silverwind I just saw your comment. OK, I will look into that 👍

[github-actions]

Automatically locked because of our CONTRIBUTING guidelines