Update hackage-server for Cabal 3

[bgamari]

This ticket tracks the progress of porting hackage-server to Cabal-3.

[bgamari]

I started working on this a while ago (see https://github.com/bgamari/hackage-server/tree/cabal-3) but admittedly didn't make it very far. The problem was that I wasn't particularly happy with any of the options for rewriting the Pretty and Parsec instances for Day and UTCTime in Distribution.Server.Framework.Instances.

@hvr has also said that he wanted someone to verify that the Binary instances remained compatible (although I'll admit I don't know which Binary instances he was referring to; I don't see any such non-trivial instances in hackage-server).

[bgamari]

Ahh, I see. hackage-server uses cereal's Serialize class, not Binary. This is likely what @hvr was referring to. I compatibility is a concern we should presumably add some golden tests to verify this property before starting on the upgrade.

[bgamari]

PkgConfigVersionRange is unfortunately now a distinct type from VersionRange, which will be problematic in the IsDependency PkgconfigDependency instance.

[phadej]

See hackage-trustees/hackage-cli#13

[tfausak]

I started looking at this too. You can take a look at my branch, but it's still very much a work in progress.

I've spent about a day's worth of work on it so far. I have been having an extremely hard time making the upgrade to Cabal 3. So many things have changed, moved, or been deleted that it's hard to write code that's both backwards and forwards compatible. On top of that, some of the code that is forwards compatible is simply broken, like the Parsec instance for PackageIdentifier.

[tfausak]

I see a bunch of commits were recently pushed to master that seem to start addressing this issue: f3c5eb8, 11136d3, 70de659, b58a7df, 8924eca, add48b9, d003d77. Should I stop working on this?

[tfausak]

I'm not working on this anymore. As far as I can tell, the wip/cabal-3.0 branch is making good headway.

[dwijnand]

Oh. I wasn't aware of this thread or the progress reported here...

I wanted to create such a thread to mark trailing TODOs on the progress being made in wip/cabal-3.0:

• get the code passing on CI: b58d61e
• implement Pretty and Parsec for Time: 088bfed#diff-0251fd4f91872f46f89c3c26c76750b2R241-R242
• implement the flagAssignment field of BuildReport's FieldGrammar: 088bfed#diff-0251fd4f91872f46f89c3c26c76750b2R254

[bgamari]

What remains to be done here, @dwijnand?

[dwijnand]

I'm going to sync up with @hvr on this soon, but it builds already against Cabal 3 (in the wip/cabal-3.0 branch), what's left is upgrading the logic to the changes Cabal 3 brought.

[swamp-agr]

Hello @dwijnand,

I kindly ask you, do you have any news regarding this topic?
Do you know, who is planning to work on the issue?

Thanks,
Andrey

[dwijnand]

No, sorry, no news. I'm not sure who (if anyone) has taken up this task.

[alexbiehl]

@dwijnand can you summarize what's still missing? From the commits on your branch I can see you managed to build hackage with Cabal 3. What's left?

[dwijnand]

If I remember correctly from my chat with @hvr there was the whole aspect of the behavioural changes in Cabal 3 to apply and verify, with the hackage server's data. I don't recall the details and/or if there were other remaining tasks, sorry.

[alexbiehl]

Thank you for your quick response. I have a call scheduled with Herbert in the hope we can come up with a validation plan.

[tchoutri]

@alexbiehl Hi Alexander! Are there any news on this front? :)

[cartazio]

And what if anything can other contributors do to help facilitate?

[bgamari]

I would encourage anyone who works on this to proactively summarize their status as progress is made. One of the things that makes this issue quite hard to resolve is that there is generally a lack of transparency into what has been done by who, what remains to be done, and what can be punted into the future.

[recursion-ninja]

I need to upload a new version of a package to hackage.org which uses features in the .cabal file format that were introduced in Cabal-3.0, notably multiple public libraries.

How long should I expect to wait before I'm able to upload the new version of my package?

[alexbiehl]

Sorry for being late. Herbert and I had the call. Two things necessary:

1. Making Hackage compile with Cabal 3 is done thanks to @dwijnand. What's missing is proper validation.

2. Hackage as running (see hackage-central branch) requires a library cabal-parsers. cabal-parsers is being used to parse and validate Cabal files. It needs to support every version of Cabal out there. I have opened a PR to support Cabal 3 here.

@hvr Can you give us an update on where you are with the validation?

[gbaz]

So once cabal-parsers is updated, that suffices to support validation, or are we missing something else?

[gbaz]

@alexbiehl can you open a PR for that branch, or do you think that there are any further issues? i.e. should we just merge it?

[alexbiehl]

I think it makes sense to get this thing merged. However, I am unsure about deployment without proper test using current production state. The concern is that due to auto-derived instances serialization might be broken for some types. Hitting such a case would corrupt the state.

It's not the right time for it but I will say it nevertheless: never serialize data without explicit contract. (Learned the hard way at @shlevy's school of hard knocks)

[fumieval]

Is there a recommended way to test hackage server? I built an older version (86e5e02) of Hackage, started it locally with an HTTPS reverse proxy (since cabal-install can only upload through HTTPS regardless of secure), and registered a test user successfully. However I didn't get to upload a package; cabal (cabal-install 3.0.0.0 with a hack to add --insecure) is stuck at

/usr/bin/curl --insecure --config - 'https://localhost:4433/packages/candidates' --insecure --form 'package=@dist-newstyle/sdist/deriving-aeson-0.2.2.tar.gz' --write-out '
%{http_code}' --user-agent 'cabal-install/3.0.0.0 (linux; x86_64)' --silent --show-error --header 'Accept: text/plain' --location                                                       

It's hard to tell what's going on because hackage-server prints nothing.

[phadej]

@fumieval upload through the web interface (if you are not particularly testing cabal-install upload functionality).

[expipiplus1]

I think it makes sense to get this thing merged. However, I am unsure about deployment without proper test using current production state. The concern is that due to auto-derived instances serialization might be broken for some types. Hitting such a case would corrupt the state.

What's the plan for getting this merged released? Super looking forward to using this.

[Kleidukos]

@alexbiehl @hvr Hi, I hope you're alright in these difficult times.
I was wondering what were the news on this front. :)

[gbaz]

Ok this is deployed now. Thanks everyone for your help!

[mitchellwrosen]

I see hackage still rejects multi-public-library packages because cabal emits a warning. Is that intentional?

Error: Invalid package

visibility is experimental feature (issue #5660)

[expipiplus1]

@mitchellwrosen perhaps this will be enabled once haskell/cabal#5660 is closed

[philderbeast]

I tried today to upload a package candidate to hackage with multiple public libraries that uses the package:library format in some if its dependencies but this candidate was rejected:

Invalid package
plugins-for-blobs-0.1.0/plugins-for-blobs.cabal:939:25:
  colon specifier is experimental feature (issue #5660)

A snippet of the cabal file in that region:

  build-depends:
      base
    , plugins-for-blobs:uom-plugin
    , plugins-for-blobs:uom-quantity
    , plugins-for-blobs:uom-th
    , tasty
    , tasty-hunit

[andreasabel]

Maybe open a new issue for the multi-libs problem?

[philderbeast]

@andreasabel I will open a new issue with the reproduction steps I have.

I removed the package:library format use and tried uploading again finding visibility is rejected:

Invalid package
plugins-for-blobs-0.1.0/plugins-for-blobs.cabal:444:21: visibility is experimental feature (issue #5660)

[recursion-ninja]

I've been waiting for over 2 years to upload an updated version of bv-little to hackage. Long ago I refactored the codebase to utilize the multi-library package feature and am unwilling to revert the changes or maintain two branches with different package build APIs. I would greatly appreciate prioritizing at least a road map and timeline for hackage support for publishing multi-public-library packages.

[gbaz]

I believe the new cabal library should remove this warning and allow upload of such packages. Two things need to happen first. First: the new cabal library 3.8 needs an official release. Second, we need to update hackage-server to use the 3.8 library following the official cabal release, and redeploy.

Note that this was all by design, as a series of open issues on cabal meant that multi-lib packages were still experimental and not complete, and correspondingly hackage server enforced the cabal warnings. Some good work by some cabal contributors (thanks @fgaz and @grayjay !) and a fair amount of sleuthing fixed the largest outstanding issues with multi-libs. The following is the cabal PR in 3.8 which removes the warning: haskell/cabal#8089

[recursion-ninja]

@gbaz Delighted to hear the timeline is so imminent!