Go Modules version and dependency support! #2528

tiegz · 2020-06-05T17:31:23Z

This adds support for "Releases" and "Dependencies" for Go Modules, which most Go projects are using now. It switches us from "go-search.org" (which doesn't work anymore) to "pkg.go.dev" (the new Go Module directory) and "proxy.golang.org" (the official Go Module proxy).

Example from k8s:

tiegz · 2020-06-05T17:34:52Z

app/models/package_manager/go.rb

+              number: info["Version"],
+              published_at: info["Time"].presence && Time.parse(info["Time"])
+            }
+          end


This is the only questionable part -- it's a choice between two options:

this, where we make a superfast request for each version to get it's actual published_at

or, make a single request to the pkg.go.dev page, and get each version with a Date-only-version of its published_at

🤷‍♂️

I like the approach here of doing a request per version. This is a background job and so would rather get the full and correct info

@katzj I just switched this to use the pkg.go.dev scraping instead, because I noticed pkg.go.dev will index packages that aren't technically Go Modules (i.e. don't have go.mod), e.g. https://pkg.go.dev/github.com/BurntSushi/toml , which works, whereas the Go proxy url fails 😕 :

$ curl http://proxy.golang.org/github.com/BurntSushi/toml/@v/list bad request: invalid escaped module path "github.com/BurntSushi/toml"

Seem ok?

Use chronic library to parse version timestamps like "1 day ago"

katzj

Will we want to be re-looking at all go packages after this goes in to make the data more current/correct? I could even see trying to clean up some of the old stuff

katzj · 2020-06-05T19:29:20Z

app/models/package_manager/go.rb

    end

    def self.install_instructions(project, version = nil)
      "go get #{project.name}"
    end

    def self.project_names
-      get("http://go-search.org/api?action=packages")
+      # Currently the index only shows the last <=2000 modules from the date given. (https://proxy.golang.org/)


About how many a day are there usually?

Uh oh, just realized this is actually package-specific (not module-specific), and it's version-specific too. So this list is gonna change [very frequently]. Left a note about that in the Go issue above tho.

katzj · 2020-06-05T19:33:39Z

app/models/package_manager/go.rb

    end

    def self.project(name)
-      get("http://go-search.org/api?action=package&id=#{name}")
+      if pkg_html = get_html("https://pkg.go.dev/#{name}?tab=doc")
+        go_module, _latest_version = pkg_html.css('a[data-test-id="DetailsHeader-infoLabelModule"]').first&.text&.split('@', 2)


This seems fragile... how often is name != go_module? Because then we could just use the version list from the proxy and figure out latest version ourselves

We're currently indexing both packages+modules, so name could be either the package or the parent module.

This part's not to get the latest_version tho, it's just to strip the latest_version from this module name. I might be wrong and it's not there, so I'll circle back in a bit and see if the splitting is necessary.

katzj · 2020-06-05T19:33:48Z

app/models/package_manager/go.rb

+        # NB this requires a quick request for each version, but we could alternatively fetch from
+        # https://pkg.go.dev/mod/#{go_module}?tab=versions and the '.Versions-commitTime', selector,
+        # but the dates are a combination of date-only or natural language (e.g. '1 day ago')
+        versions = get_raw("http://proxy.golang.org/#{go_module}/@v/list")


Suggested change

versions = get_raw("http://proxy.golang.org/#{go_module}/@v/list")

versions = get_raw("https://proxy.golang.org/#{go_module}/@v/list")

katzj · 2020-06-05T19:34:57Z

app/models/package_manager/go.rb

+              number: info["Version"],
+              published_at: info["Time"].presence && Time.parse(info["Time"])
+            }
+          end


I like the approach here of doing a request per version. This is a background job and so would rather get the full and correct info

app/models/package_manager/go.rb

…n pkg page.

tiegz · 2020-06-06T20:49:54Z

Will we want to be re-looking at all go packages after this goes in to make the data more current/correct? I could even see trying to clean up some of the old stuff

Def! I'm afraid of overwhelming pkg.go.dev tho -- maybe something like this?

Project.where(platform: "Go").find_each.with_index do |p, i|
  PackageManagerDownloadWorker.perform_at((i * 1.second).from_now, platform, name)
end

With 1.8 million Go packages, that'd take about 21 days.

The slowest thing in an update() I think is fetching each go.mod file for each version, about 2 sec per request. As that go Issue above says, we can fetch deps from pkg.go.dev, but they're only for the latest version :\

katzj

I don't love all the html parsing as changes to their css will obviously break us... but better than where we are now.

We should think about how to detect a change and make it so that we know that things are broken but I'm okay doing that in a follow-up

tiegz added 2 commits June 5, 2020 13:03

Go Modules version support!

568f612

pecs Please enter the commit message for your changes. Lines starting

5e99543

tiegz requested a review from katzj June 5, 2020 17:31

tiegz commented Jun 5, 2020

View reviewed changes

tiegz requested a review from jsonperl June 5, 2020 19:07

Fetch versions in a single request from pkg.go.dev instead

c206793

Use chronic library to parse version timestamps like "1 day ago"

katzj reviewed Jun 5, 2020

View reviewed changes

tiegz added 3 commits June 5, 2020 15:39

Fix

9e2df2d

Only use parent module page as a backup for versions if they aren't o…

081b1a5

…n pkg page.

Support dependency fetching for Go Modules

b7af028

tiegz changed the title ~~Go Modules version support!~~ Go Modules version and dependency support! Jun 6, 2020

Remove some debugging.

5ab73e8

tiegz added 2 commits June 8, 2020 12:52

Bump bibliothecary for Go Mod fixes

602ca7d

Remove a spec for parse_names, which was removed, to fix tests

4d39dd1

tiegz requested a review from katzj June 9, 2020 14:26

katzj approved these changes Jun 16, 2020

View reviewed changes

tiegz added 2 commits June 17, 2020 09:25

Merge branch 'master' into tiegz/go-mod-version-support

fe0d174

Add note about slow mod fetches

6e9537c

tiegz merged commit 30e0602 into master Jun 17, 2020

tiegz deleted the tiegz/go-mod-version-support branch June 17, 2020 15:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Go Modules version and dependency support! #2528

Go Modules version and dependency support! #2528

tiegz commented Jun 5, 2020 •

edited

Loading

tiegz Jun 5, 2020

katzj Jun 5, 2020

tiegz Jun 5, 2020 •

edited

Loading

katzj left a comment

katzj Jun 5, 2020

tiegz Jun 5, 2020 •

edited

Loading

katzj Jun 5, 2020

tiegz Jun 5, 2020

katzj Jun 5, 2020

katzj Jun 5, 2020

tiegz commented Jun 6, 2020

katzj left a comment

	versions = get_raw("http://proxy.golang.org/#{go_module}/@v/list")
	versions = get_raw("https://proxy.golang.org/#{go_module}/@v/list")

Go Modules version and dependency support! #2528

Go Modules version and dependency support! #2528

Conversation

tiegz commented Jun 5, 2020 • edited Loading

tiegz Jun 5, 2020

Choose a reason for hiding this comment

katzj Jun 5, 2020

Choose a reason for hiding this comment

tiegz Jun 5, 2020 • edited Loading

Choose a reason for hiding this comment

katzj left a comment

Choose a reason for hiding this comment

katzj Jun 5, 2020

Choose a reason for hiding this comment

tiegz Jun 5, 2020 • edited Loading

Choose a reason for hiding this comment

katzj Jun 5, 2020

Choose a reason for hiding this comment

tiegz Jun 5, 2020

Choose a reason for hiding this comment

katzj Jun 5, 2020

Choose a reason for hiding this comment

katzj Jun 5, 2020

Choose a reason for hiding this comment

tiegz commented Jun 6, 2020

katzj left a comment

Choose a reason for hiding this comment

tiegz commented Jun 5, 2020 •

edited

Loading

tiegz Jun 5, 2020 •

edited

Loading

tiegz Jun 5, 2020 •

edited

Loading