Semantics of index-state in cabal.project

[andreabedini]

At IOG we have started using a custom hackage repositories, which we informally call CHaP.

To add the custom repository to a project we have been doing

repository cardano-haskell-packages
  url: https://input-output-hk.github.io/cardano-haskell-packages
  secure: True
  root-keys:
    -- snip

index-state: 2022-07-01T00:00:00Z
index-state: cardano-haskell-packages 2022-10-20T00:00:00Z

We thought the first index-state would apply to all active repositories, and the second one only to the specified repository. (For context: we did this because we were working around a limination with another tool which we use to parse cabal.project).

But we were wrong! After a while it became clear that the second index-state completely overrides the first (as the first was never there).

Indeed, the correct way to express two index-states would be

index-state:
  , hackage.haskell.org 2022-07-01T00:00:00Z
  , cardano-haskell-packages 2022-10-20T00:00:00Z

1. Is this the intended behaviour?
2. Should cabal accept multiple index-state stanza (for different repos)
3. Should cabal give a warning when it sees stanzas which override previos ones?

Thanks!

Ping @michaelpj @angerman

[michaelpj]

Just checking @andreabedini, I think we found that this also doesn't work?

index-state: hackage.haskell.org 2022-07-01T00:00:00Z
index-state: cardano-haskell-packages 2022-10-20T00:00:00Z

In terms of expected semantics, my naive expectation is that cabal does something like "overriding union" when it sees a subsequent stanza of the same kind, i.e. it unions as much as possible, and where there's a conflict it takes the later one. e.g. if you have two constraints stanzas it will take the union of them unless they conflict.

So I think the surprise is that we would expect cabal to be able to do some unioning here but in fact it just overrides the whole value. Especially in the case where there are two index-states for different repositories (the one with the "unqualified" index-state and the "qualified" one is a bit trickier).

[michaelpj]

Rough diagnosis: the TotalIndexState field in SharedProjectConfig is a Flag TotalIndexState, which has a "first" semigroup instance. Solution:

• Change the field to Maybe TotalIndexState
• Give RepoIndexState a "first" semigroup instance
• Give TotalIndexState a semigroup instance that unions the map of states with <>

If I understand correctly, that will make cabal do the right thing when parsing. I'm giving it a shot.

[andreabedini]

Yes,

index-state: hackage.haskell.org 2022-07-01T00:00:00Z
index-state: cardano-haskell-packages 2022-10-20T00:00:00Z

would use HEAD as index-state for hackage.

[michaelpj]

WIP here: https://github.com/michaelpj/cabal/tree/mpj/merge-index-states

Need to figure out how to write a test.

[ulysses4ever]

It's not exactly clear to me that the proposed semantics is any better than the current one, and the switch will probably break someone's code. It'd be interesting to try to solicit opinions on this on Discourse/Reddit/Cafe.

[mouse07410]

FWIW, the proposed semantics makes more sense to me than the currently-implemented one. I don't expect actual cider to be broken by this proposal.

I support this change, which actually is a fix.

[michaelpj]

It has been this way since it was introduced in #6597, I wonder if @phadej has a drive-by opinion.

[phadej]

The current behavior is so that if there is index-state: in the cabal.project, then the index-state is always completely specified. The type of the field in project config is TotalIndexState.

If you add a PartialIndexState type and change the project config to read that, and add conversion functions where GHC asks for them, then you get what you want.

TL;DR IIRC I did what was simplest at the time.

I doubt it will break much code as multiple repositories and index-states using them are not that common combination.
Maybe someone had written
index-state: hackage.haskell.org 2022-07-01T00:00:00Z today so that any extra repositories are at the latest, but it's not a big thing to fix (and that would work with older cabal-installs).
Also cabal freeze is explicit, writing all repositories in index-state (as well as active-repositories), so freeze files won't be broken, and freeze files are probably the files where those are used the most.

---

FWIW: flags is also a complete overwrite field in cabal.project, but that case is different as there wouldn't be a way to clear the flags if it was additive, where index-state: HEAD is the default setting for index-state.

[michaelpj]

The current behavior is so that if there is index-state: in the cabal.project, then the index-state is always completely specified. The type of the field in project config is TotalIndexState.

If you add a PartialIndexState type and change the project config to read that, and add conversion functions where GHC asks for them, then you get what you want.

It took me a while to see the problem, so let me see if I can say it right. The change I propose is basically just:

- Take the first value when combining
instance Semigroup RepoIndexState where
    s1 <> _ = s1

instance Semigroup TotalIndexState where
    (TIS s1 rs1) <> (TIS s2 rs2) = TIS (s1 <> s2) (Map.unionWith (<>) rs1 rs2)

data ProjectConfigShared =
...
  projectConfigIndexState :: Maybe TotalIndexState,
...

So the first index-state stanza will completely specify the index state, and subsequent ones will just override the bits that they specify. However the issue is with parsing: when we parse an index-state stanza we parse a TotalIndexState so we have to have a value for the first field, which will be HEAD if not supplied. So my change will still do the wrong thing here:

index-state: T1
index-state foo T2 -- will be parsed with HEAD as the first field, so will override T1

So indeed fixing this properly would need the first field to be Maybe RepoIndexState and then indeed it's not total. Probably still doable, but not the very simple fix I thought it was, alas.

[phadej]

No, you understood me wrong.

Nothing have to change about TotalIndexState. As it names implies, it's totally specified the index state.

The ProjectConfigShared have to be changed to

data ProjectConfigShared =
  ...
  projectConfigIndexState :: PartialIndexState,
  ...

where the naive implementation could be

newtype PartialIndexState = PartialIndexState (TotalIndexState -> TotalIndexState)

toTotalIndexState :: PartialIndexState -> TotalIndexState
toTotalIndexState (PartialIndexState f) = f headTotalIndexState

fromTotalIndexState :: TotalIndexState -> PartialIndexState
fromTotalIndexState tis = PartialIndexState (\_ -> tis)

however that representation won't work as you cannot pretty print it.

EDIT: I don't remember how TotalIndexState is represented now, maybe PartialIndexState can be represented very similarly, but please use different types for different things when they are different. That's important for cabal freeze, there the written PartailIndexState should specify the index-state fully.

[Mikolaj]

@andreabedini: do you have a good enough workaround?

@michaelpj: is there a way to cheaply salvage your work and improve the situation, before you completely context switch?

[andreabedini]

@Mikolaj yes, we have a workaround. Thanks for asking.

[michaelpj]

I got stuck thinking about what @phadej said. I think the fundamental change is not that hard, it's just representing it in a way that matches the way the cabal devs want to think about it.

• There are two types
  • One representing a potentially not-fully-specified index-state configuration, which is the sort of thing you get from parsing a project file (PartialIndexState? IndexStateConfig?). This has an obvious Semigroup instance so the cabal parsing setup will combine them in the right way.
  • One represents a fully-specified index-state configuration (TotalIndexState).
• I need to at least try having both.
• I'm not 100% sure if we definitely need TotalIndexState, that was partly what I wanted to investigate.
• I need to figure out how to write a test before I go any further anyway, and I was a bit stuck on that.

I think my previously linked tree provides some value if someone else wants to pick this up, I am planning to come back to this when I get some free cycles but I'm not sure when that will be, especially since we have a workaround.

[phadej]

I'm not 100% sure if we definitely need TotalIndexState, that was partly what I wanted to investigate.

I think we'll need a PartialIndexState if #6101 is done "properly". You need a way to have an "empty" index-state, which would combine monoidally. That if cabal.project files are first parsed, and then interpreted, instead of current CPP-style where cabal.project files are partly parsed first, then conditional are interpreted, and then the actual parsing is done. (I haven't checked, but I assume how it works now, as TotalIndexState is not a monoid).

EDIT: also flags are overriden, IIRC, so the similar change would be needed there as well.

[andreabedini]

I am ok with the current state now that is clear this is the intended behaviour. @michaelpj, can you open a separate PR?

[michaelpj]

You mean a separate issue? I'm unclear that this is desirable behaviour but I also think it's not a huge deal, if we have a way to deal with it.

[andreabedini]

You mean a separate issue? I'm unclear that this is desirable behaviour but I also think it's not a huge deal, if we have a way to deal with it.

Nevermind, I was under the impression you where working on some changes.

[michaelpj]

An example of where the current behaviour would be undesirable. Suppose we have nightly GHC distributions. Users using a particular nightly may also want to use a particular index-state of head.hackage. Using the new support for cabal.project inclusions and ghc version conditionals, GHC could provide a cabal.project fragment that has conditionals for different nightly GHC versions that sets the head.hackage index state. Something like this:

if (ghc >= <some_nightly_version>)
  index-state: head.hackage <some date>

However, the natural way of using this won't work:

import: ghc-nightly-config

index-state: hackage.haskell.org <some other date>

Oops! This clobbers the head.hackage index state. Doing it in the other order will clobber the user's index state.

[gbaz]

imho the right thing here is a monoid with the following semantics -- we start with a fixed total indexed state for all repos. then, in order, every subsequent partial element just overrides the fixed total state for that one repo, and hitting a new total timestamp resets everything that came before.

I'm pretty sure this obeys the monoid laws -- we just need to specify that an partial override with no initial corresponding total element is interpreted as having one anyway, semantically.

i know i wrote it a bit confusingly, but there's a real monoid here, just not a very standard one.

[andreabedini]

I think the use case that @michaelpj presents in #8568 (comment) is compelling.
I would be ok with the behaviour @gbaz describes above (provided I understood it correctly).

Implementation-wise I think we are stuck on #6101 (striclty speaking we are not, but I won't spend time on the legacy parser :P)

[phadej]

@michaelpj use case is compelling, but cabal.project in general doesn't work well for that.

E.g. constraints are accumulating, there is no way to remove/override constraints. So if instead of head.hackage use-case, there is Stackage one, with

constraints: abstract-deque ==0.3,
             abstract-par ==0.3.3,
             accuerr ==0.2.0.2,
...

then people are stuck, if they e.g. want different abstract-par version.

So if you make a breaking but subtle change to index-state treatment, it's really worth making breaking and similarly subtle changes elsewhere too at the same time, so people can adapt their cabal.projects at once.

I agree that constraints being accumulated, but index-state being override (and flags being override) is inconsistent, but I think that override is better default so far, then not having (granular) override at all.

TL;DR, once you "fix" index-state, people will complain of about constraints.

---

Which leads me to another point: cabal.project should be versioned format. E.g. have cabal-project-version field, so cabal-install could warn (or better, refuse to work by default) if it's too old (or too new) to interpret the particular project file. So far updating cabal-install was generally fine, as features had been only added. But if there is non-backward compatible change to cabal.project file treatment, a very annoying update surprise may happen (Again, change from override to accumulate of a field is a subtle breaking change).

[michaelpj]

Okay, let me make a different proposal. If I understand correctly, part of the objection is that index-state is a single field, so it's odd for repeated definitions of the field to have complicated partially-acummulating, partially-overriding behaviour.

But... isn't index-state morally a scoped field, by repository? Compare with e.g. ghc-options:

ghc-options: ...

package foo
  ghc-options: ...

package bar
  ghc-options: ...

The situation today is as if we declared this with ghc-options: "<options>", foo "<options>", bar "<options>". People would rightly be quite unhappy with that!

Moreover, we even have an element for setting options relating to repositories: the repository stanza. So how about this:

-- sets the global index-state
index-state: ...

repository hackage.haskell.org
   -- sets the index-state for hackage
   index-state: ...

This I think has the best of both worlds: individual option settings are overriding, but we have separate option settings for different scopes, which represents what we want to say nicely.

[phadej]

I wouldn't write

package aeson
  constraints: ^>=2.2.0.0

I'd also like to write

flags: aeson +ordered-keymap, vector +boundschecks

or

flags: aeson +ordered-keymap
flags: vector +boundschecks

This is matter of taste (but constraints is so plentiful that terseness is a nice property to have).

---

@michaelpj sure, today I write

constraints: aeson ^>=2.1.0.0
constraints: vector ^>=0.13.0.0

and it would be fine to write

index-state: head.hackage 2023-...
index-state: hackage.haskell.org 2024-...

(and have the semantics of repository stanzas you mention)

and the latter is actually good, as probably something like

index-state: head.hackage 2024-...

or

index-state: head.hackage HEAD

can be written later to override the choice.

I.e. index-state is naturally scoped by repository, and there is a "reset".

but I cannot reset constraints. If I add

constraints: aeson ^>=2.2.0.0

I b0rk by project, ^>=2.1.0.0 && ^>=2.2.0.0 is unsatisfiable constraint. I need to remove the previous aeson ^>=2.1.0.0.

It's good to improve index-state, I'm just trying to say

• another issues will be hit. constraints is known one, there are probably unknown
• I'm no against breaking changes, but as cabal.project is not versioned, I prefer rare or none (and hopefully coherent) breaking changes. If you add versioning first, then I will tolerate breaking change every major cabal-install release.

TL;DR make cabal.project versioned format. Then you can "freely" iterate it.

[phadej]

if monoidal semantics are added, there probably need to be a way to reset to "global" per-repository index-states

index-state: 2024-...
index-state: head.hackage 2023-
index-state: head.hackage clear

would result into index-state: 2024-... total state, as index-state: head.hackage HEAD would 'relax" head.hackage's index state too much.

I don't have a motivation for this, other than countering the @michaelpj if I need to repeat index-state: head.hackage 2024 then I can also repeat the whole index-state.

In repository stanza syntax (which I don't like)

index-state: ...

repository hackage.haskell.org
   -- sets the index-state for hackage
   index-state: ...

one would need a way to reset hackage.haskell.org treatment.

---

This reminds me that flags don't have reset, once a flag foo is set

package aeson
  flags: +foo

there is no AFAIK a way to reset it to the package (versions') default. (EDIT: nor can flags specified in be overriden, constraints, like constraints: aeson +foo not even flipped)

[gbaz]

constraints are accumulating, there is no way to remove/override constraints

This is also something that we want to fix. And in general I agree there should be a full revamp to make virtually anything in cabal.project overridable.

cf. the discussion here #9510

The asks in this ticket and that that PR is trying to resolve are longstanding goals which we have intended to implement ever since conditional and imports were introduced to project files.

Implementation-wise none of these changes require changing the parser. We already accumulate all fields with some form of monoidal semantics -- we just need to pick appropriate monoids compared to the ones we use now.

Finally,

I disagree on blocking these changes on making cabal.project versioned.

We should be careful we don't alter existing common, meaningful syntax. However, there's been a longstanding view that foo.cabal files should be versioned because that syntax is part of the specification of a package, while cabal.project files are not that -- they're auxiliary tools, and since cabal-install is backwards compat with prior ghc versions, its ok, if not awesome, to have users update them to keep pace with new versions of cabal-install they are developing with.

I'm open to someone implementing new cabal project versioning in the future. But there's no reason to prevent us from continuing to improve things now.

tl;dr: oleg's comments are useful but some version of addressing this issue, constraints, and other overrides in cabal.project files is absolutely in keeping with the current direction of travel of cabal.project files.

[andreabedini]

With respec to overriding constraints, see my comment here. IMHO the real use case is to choose a package version and solve for the rest. "overriding constraints" is one possible technical solution but to be honest I am not particularly fan of, due to its unclear semantic.