use "embed" in generated code #2119

vikstrous · 2022-04-23T23:44:58Z

I work in a codebase that uses gqlgen heavily for a large graphql monolith. There are 384k lines of code in the files generated by gqlgen. We check in all these files into git because it takes a long time to generate them (and because it's a lot simpler to work that way).

This PR changes the generated files to use the embed feature of go to reduce the size of the generated files and reduce the chance of having to resolve PR conflicts.

I think that the complexity of the change is fairly low and there is no down side to introducing it. I made sure it has reasonable test coverage. It should be backwards compatible and doesn't change any behaviour.

I have:

Added tests covering the bug / feature (see testing)
[n/a] Updated any relevant documentation (see docs)

StevenACoffman · 2022-04-24T12:20:26Z

This is a great idea, but we definitely need more coverage around it. We support Go 1.16 and newer, and we haven't really taken advantage of embedding up until now.

vikstrous2 · 2022-04-24T18:02:22Z

Alright, here's how we get this mergable:

Schema filenames are currently relative to the place gqlgen is run, not the generated code directory, so we need to change the filenames to make them relative to the correct directory
The filesystem variable and function have to be conditionally included if there are any embeddable sources - so we basically need a list of embeddable and non-embeddable sources
Some of the examples have files that are not embeddable, so that's nice, but maybe we should add an extra test for schema files that are entirely outside the go module to catch any unexpected regressions
We should consider making the sources for "builtins" references to gqlgen library constants rather than including them in the generated code... It would be a nice bonus.

coveralls · 2022-04-25T01:55:21Z

Coverage increased (+0.02%) to 74.706% when pulling 524ec52 on vikstrous:sources-embed into f8e837b on 99designs:master.

StevenACoffman · 2022-04-28T18:34:49Z

Hey, I'm waiting on you to tell me when you think this is mergeable to re-review it

vikstrous · 2022-05-01T18:52:54Z

codegen/data.go

@@ -147,6 +170,30 @@ func BuildData(cfg *config.Config) (*Data, error) {
 		// otherwise show a generic error message
 		return nil, fmt.Errorf("invalid types were encountered while traversing the go source code, this probably means the invalid code generated isnt correct. add try adding -v to debug")
 	}
+	aSources := []AugmentedSource{}
+	for _, s := range cfg.Sources {
+		wd, err := os.Getwd()


I'm not happy with having this here, but it seemed like the most reliable way to build the full path to the sources. Is there a better way?

vikstrous · 2022-05-01T18:55:40Z

The PR is reviewable now. Sorry for the delay. This has been a weekend project :)
I didn't do the extra work to make "builtin" sources references, but someone can add that as a follow up if it's useful for them.

StevenACoffman · 2022-05-05T21:03:32Z

Thanks! I would like to have someone make "builtin" source references, but not holding this up for that.

frederikhors · 2022-05-23T19:40:18Z

I'm having trouble with this. I think we'll have A LOT of issues for this.

@vikstrous can you please PR ASAP for docs instruction for this to work?

For example I'm having coming from 0.17.5:

panic: codegen problem: schema\schema.graphql not availalbe

THIS IS A HUGE PROBLEM.

frederikhors · 2022-05-23T19:44:32Z

Maybe this is a problem with Windows, @vikstrous?

I see path like: //go:embed "schema\\schema.graphql".

frederikhors · 2022-05-23T19:59:18Z

@StevenACoffman this is a huge problem.

I agree with the need to have it - even soon - but it should be tested better and maybe used with a feature flag maybe in the config file.

In projects like the one where I'm applying it (where there are several source of *.graphql files) it has now become a mess and nothing works.

Basically it's a COMPLICATED breaking change that came with a patch.

I propose to release 0.17.7 as soon as possible with this commit turned off and then we can see how to release it properly.

If we don't release 0.17.7 we can't use the other important changes from 0.17.6.

Please.

vikstrous · 2022-05-23T21:15:17Z

Thanks for letting me know about the bug. I created issues for you: #2189 #2190

vikstrous · 2022-05-23T21:19:15Z

@frederikhors Let me know if this fixes the issue: #2191 I don't have windows and I don't have a way to test it.

frederikhors · 2022-05-23T23:44:07Z

I renamed manually all the \\ to / and it works.

So I think we should fix it this way.

mtibben · 2022-05-24T22:49:10Z

Hey @vikstrous @StevenACoffman

I'm a bit surprised by the scale of this change (and potential for issues?) with minimal discussion on its merits. The PR description talks about this as a Proof of Concept, yet it has been merged.

The PR description doesn't explain what the purpose of this change is, who or what it benefits and why.

Have I missed a discussion that happened somewhere else?

vikstrous · 2022-05-24T23:50:19Z

@mtibben You are not missing any other discussion, but note that the PR description hasn't been updated after the proof of concept was turned into a mergable PR.

Sorry for the issues this caused. TBH I didn't expect anyone to be using this on windows, but I see that with popular open source projects, you never know what people will do with your code.

I can add more context about the motivation. I work in a codebase that uses gqlgen heavily for a large graphql monolith. There are 384k lines of code in the files generated by gqlgen. We check in all these files into git be because it takes a long time to generate them (and because it's a lot simpler to work that way). The motivation behind this PR was to reduce the size of the generated files and reduce the chance of having to resolve PR conflicts.

I think that the complexity of the change is fairly low and there is no down side to introducing it. I made sure it has reasonable test coverage. Clearly I hadn't considered windows support, but that's just a bug that could have been introduced by any PR regardless of what it's changing. This didn't seem like a major change because it's backwards compatible and doesn't change any behaviour (I wasn't the one that decided to include it in a patch relaese, but I agree with the decision).

mtibben · 2022-05-25T00:49:50Z

Cool thanks for the explanation @vikstrous. Do you think you could update the PR description with this summary and an accurate description of the changes, so that future devs can understand the change and motivations fully?

frederikhors · 2022-05-25T11:10:07Z

I agree with @vikstrous this is not a big issue (the Windows issue is fixed in master and ready to be released with next patch release).

Perhaps a more conservative approach would have avoided some stress: releasing more often (using minor releases maybe for thing like this) is useful to reduce issues like the one I had on Windows.

Anyway @vikstrous did you get to see #2198?

StevenACoffman · 2022-05-25T12:10:29Z

@mtibben This PR worked flawlessly in our production codebase (on linux and locally on MacOS), and was a backwards compatible change (with a bug on windows that went unnoticed because we weren't running tests there).

Backwards compatible changes seem ok to put in patch releases to me, but I'm open to discussing it. I don't think I've ever articulated my approach, so it's good to talk about it.

While semver implies Major.Minor.Patch => Breaking.Feature.Fix because of how go modules work, we aren't releasing a gqlgen V2, V3, V4, nor do we have the volunteer resources to maintain in parallel versions beyond whatever the latest release is. The V0 implies gqlgen may occasionally (and with great hesitation) break compatibility in minor ways between releases, but not for Patch releases. This effectively means that our releases are tagged according to v0.Breaking.(Feature|Fix)

We can, of course, discuss if this release tagging policy I've been following is ok to continue with.

mtibben · 2022-05-26T00:46:04Z

@StevenACoffman yeah it would be great to get visibility of production systems testing these changes. I'd love for our systems here at 99 to also be on bleeding edge to build confidence (this is what google does with go), this would be something our @99designs/delivery-systems team may be able to think about the practicalities of.

As mentioned above I also think our PR descriptions could just be levelled up to provide accuracy. The PR description here still says this is proof of concept, when it seems to have moved beyond that - I'll try to fix that up now

mtibben · 2022-05-26T00:50:55Z

@vikstrous I have updated the PR description now, can you confirm that it is accurate?

vikstrous · 2022-05-26T11:24:27Z

Thanks! I removed the last part. Looks good now

StevenACoffman · 2022-05-26T13:32:41Z

@vikstrous if you could look into #2204 I would appreciate it.

vikstrous · 2022-10-11T07:00:41Z

Yeah, sorry for the delay. This is more of a weekend project :) I'll clean it up this weekend

…

On Thu, Apr 28, 2022, 14:35 Steve Coffman ***@***.***> wrote: Hey, I'm waiting on you to tell me when you think this is mergeable to re-review it — Reply to this email directly, view it on GitHub <#2119 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADVY6IADLMZQO5XWGP6RJLVHLK5HANCNFSM5UFMCXDA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

vikstrous · 2022-10-11T08:41:12Z

Please don't merge yet! This is not ready. It's still very rough

…

On Tue, Apr 26, 2022, 15:49 Sadegh Ramezani ***@***.***> wrote: ***@***.**** approved this pull request. — Reply to this email directly, view it on GitHub <#2119 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADVY6LQPNIO7UFGWTXYH23VHBCFFANCNFSM5UFMCXDA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

xsadegh approved these changes Apr 26, 2022

View reviewed changes

vikstrous added 6 commits May 1, 2022 18:41

use "embed" in generated code

52774b9

don't use embed for builtins

8420076

working poc

2e13121

handle no embeddable sources

30eb8f9

fix dir

75c9e52

comment

1a1f5ed

vikstrous force-pushed the sources-embed branch from 97ab4d5 to 6ef94bf Compare May 1, 2022 18:44

vikstrous added 2 commits May 1, 2022 18:51

add test for embedding

ec5c463

improve error handling

524ec52

vikstrous force-pushed the sources-embed branch from 6ef94bf to 524ec52 Compare May 1, 2022 18:51

vikstrous commented May 1, 2022

View reviewed changes

StevenACoffman merged commit fb5751a into 99designs:master May 5, 2022

This was referenced May 23, 2022

Add CI for windows #2189

Closed

automatic use of "embed" package generates the wrong path on windows #2190

Closed

bakjos mentioned this pull request May 25, 2022

Embed generation with plugins fails #2204

Closed

vikstrous mentioned this pull request May 26, 2022

fix #2204 - don't try to embed builtin sources #2214

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use "embed" in generated code #2119

use "embed" in generated code #2119

vikstrous commented Apr 23, 2022 •

edited

Loading

StevenACoffman commented Apr 24, 2022

vikstrous2 commented Apr 24, 2022 •

edited

Loading

coveralls commented Apr 25, 2022 •

edited

Loading

StevenACoffman commented Apr 28, 2022

vikstrous May 1, 2022

vikstrous commented May 1, 2022

StevenACoffman commented May 5, 2022

frederikhors commented May 23, 2022

frederikhors commented May 23, 2022

frederikhors commented May 23, 2022

vikstrous commented May 23, 2022

vikstrous commented May 23, 2022

frederikhors commented May 23, 2022

mtibben commented May 24, 2022 •

edited

Loading

vikstrous commented May 24, 2022

mtibben commented May 25, 2022

frederikhors commented May 25, 2022

StevenACoffman commented May 25, 2022 •

edited

Loading

mtibben commented May 26, 2022

mtibben commented May 26, 2022

vikstrous commented May 26, 2022

StevenACoffman commented May 26, 2022

vikstrous commented Oct 11, 2022 via email

vikstrous commented Oct 11, 2022 via email

use "embed" in generated code #2119

use "embed" in generated code #2119

Conversation

vikstrous commented Apr 23, 2022 • edited Loading

StevenACoffman commented Apr 24, 2022

vikstrous2 commented Apr 24, 2022 • edited Loading

coveralls commented Apr 25, 2022 • edited Loading

StevenACoffman commented Apr 28, 2022

vikstrous May 1, 2022

Choose a reason for hiding this comment

vikstrous commented May 1, 2022

StevenACoffman commented May 5, 2022

frederikhors commented May 23, 2022

frederikhors commented May 23, 2022

frederikhors commented May 23, 2022

vikstrous commented May 23, 2022

vikstrous commented May 23, 2022

frederikhors commented May 23, 2022

mtibben commented May 24, 2022 • edited Loading

vikstrous commented May 24, 2022

mtibben commented May 25, 2022

frederikhors commented May 25, 2022

StevenACoffman commented May 25, 2022 • edited Loading

mtibben commented May 26, 2022

mtibben commented May 26, 2022

vikstrous commented May 26, 2022

StevenACoffman commented May 26, 2022

vikstrous commented Oct 11, 2022 via email

vikstrous commented Oct 11, 2022 via email

vikstrous commented Apr 23, 2022 •

edited

Loading

vikstrous2 commented Apr 24, 2022 •

edited

Loading

coveralls commented Apr 25, 2022 •

edited

Loading

mtibben commented May 24, 2022 •

edited

Loading

StevenACoffman commented May 25, 2022 •

edited

Loading