Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

502 proxy error when accessing PR view for jenkinsci/jenkins #3029

Closed
NotMyFault opened this issue Jun 29, 2022 · 8 comments · Fixed by jenkinsci/code-coverage-api-plugin#423
Closed

502 proxy error when accessing PR view for jenkinsci/jenkins #3029

NotMyFault opened this issue Jun 29, 2022 · 8 comments · Fixed by jenkinsci/code-coverage-api-plugin#423
Assignees
@dduportal
Labels
ci.jenkins.io
Milestone
infra-team-sync-2...

Comments

@NotMyFault
Copy link
Member

NotMyFault commented Jun 29, 2022 •
edited
Loading

Service(s)

ci.jenkins.io

Summary

If I try to access the PR view of jenkinsci/jenkins on https://ci.jenkins.io/job/Core/job/jenkins/view/change-requests/, I get a 502 proxy error:
Bildschirmfoto 2022-06-29 um 21 13 02
Does anyone else experience that too? Other views, folders and jobs are unaffected.

Reproduction steps

No response
@NotMyFault NotMyFault added the triage Incoming issues that need review label Jun 29, 2022
@MarkEWaite MarkEWaite removed the triage Incoming issues that need review label Jun 30, 2022
@MarkEWaite
Copy link

Thanks for reporting the issue. I see the same message as well. No idea why Jenkins core has that problem on that page when other projects in the same folder (like acceptance test harness) do not have the problem.

@daniel-beck
Copy link

daniel-beck commented Jun 30, 2022 •
edited
Loading

502 happens when Jenkins takes too long processing the request.

I just accessed the URL linked, and opened a thread dump at the same time, and there were two separate requests from two different IP addresses, both stack traces indicating the culprit is CoverageColumn being too expensive:

"Handling GET /job/Core/job/jenkins/view/change-requests/ from redacted : Jetty (winstone)-272292 View/index.jelly CoverageColumn/column.jelly" Id=272292 Group=main RUNNABLE
	at java.base@11.0.15/java.util.Arrays.copyOfRange(Arrays.java:4030)
	at java.base@11.0.15/java.lang.StringLatin1.newString(StringLatin1.java:715)
	at java.base@11.0.15/java.lang.StringBuffer.toString(StringBuffer.java:720)
	at com.thoughtworks.xstream.io.path.PathTracker.peekElement(PathTracker.java:133)
	at com.thoughtworks.xstream.io.path.PathTracker.getPath(PathTracker.java:169)
	at com.thoughtworks.xstream.core.ReferenceByXPathUnmarshaller.getCurrentReferenceKey(ReferenceByXPathUnmarshaller.java:41)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.readBareItem(AbstractCollectionConverter.java:132)
	at hudson.util.RobustCollectionConverter.populateCollection(RobustCollectionConverter.java:87)
	at com.thoughtworks.xstream.converters.collections.CollectionConverter.unmarshal(CollectionConverter.java:81)
	at hudson.util.RobustCollectionConverter.unmarshal(RobustCollectionConverter.java:78)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at hudson.util.RobustReflectionConverter.unmarshalField(RobustReflectionConverter.java:454)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:350)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1421)
	at hudson.util.XStream2.unmarshal(XStream2.java:189)
	at hudson.util.XStream2.unmarshal(XStream2.java:160)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1399)
	at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1293)
	at hudson.XmlFile.read(XmlFile.java:165)
	at io.jenkins.plugins.util.AbstractXmlStream.readXml(AbstractXmlStream.java:105)
	at io.jenkins.plugins.util.AbstractXmlStream.read(AbstractXmlStream.java:79)
	at io.jenkins.plugins.util.BuildAction.readResult(BuildAction.java:138)
	at io.jenkins.plugins.util.BuildAction.getResult(BuildAction.java:128)
	at io.jenkins.plugins.coverage.model.CoverageBuildAction.hasCoverage(CoverageBuildAction.java:165)
	at io.jenkins.plugins.coverage.model.visualization.dashboard.ProjectCoverage.getCoverage(ProjectCoverage.java:29)
	at io.jenkins.plugins.coverage.model.visualization.dashboard.CoverageColumn.getCoverageValue(CoverageColumn.java:136)
	at io.jenkins.plugins.coverage.model.visualization.dashboard.CoverageColumn.getCoverageText(CoverageColumn.java:118)
	…
	at hudson.model.View.main.run(main.groovy:16)
    …

"Handling GET /job/Core/job/jenkins/view/change-requests/ from redacted : Jetty (winstone)-270501 View/index.jelly CoverageColumn/column.jelly" Id=270501 Group=main RUNNABLE
	at java.base@11.0.15/java.util.ArrayList.spliterator(ArrayList.java:1561)
	at java.base@11.0.15/java.util.Collection.stream(Collection.java:711)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:338)
	at io.jenkins.plugins.coverage.model.CoverageNode.lambda$getCoverage$4(CoverageNode.java:336)
	at io.jenkins.plugins.coverage.model.CoverageNode$$Lambda$2032/0x0000000801f41040.apply(Unknown Source)
	at java.base@11.0.15/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base@11.0.15/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base@11.0.15/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@11.0.15/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:337)
	at io.jenkins.plugins.coverage.model.CoverageNode.lambda$getCoverage$4(CoverageNode.java:336)
	at io.jenkins.plugins.coverage.model.CoverageNode$$Lambda$2032/0x0000000801f41040.apply(Unknown Source)
	at java.base@11.0.15/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base@11.0.15/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base@11.0.15/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@11.0.15/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:337)
	at io.jenkins.plugins.coverage.model.CoverageNode.lambda$getCoverage$4(CoverageNode.java:336)
	at io.jenkins.plugins.coverage.model.CoverageNode$$Lambda$2032/0x0000000801f41040.apply(Unknown Source)
	at java.base@11.0.15/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base@11.0.15/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base@11.0.15/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@11.0.15/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:337)
	at io.jenkins.plugins.coverage.model.CoverageNode.lambda$getCoverage$4(CoverageNode.java:336)
	at io.jenkins.plugins.coverage.model.CoverageNode$$Lambda$2032/0x0000000801f41040.apply(Unknown Source)
	at java.base@11.0.15/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base@11.0.15/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base@11.0.15/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@11.0.15/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:337)
	at io.jenkins.plugins.coverage.model.CoverageNode.lambda$getCoverage$4(CoverageNode.java:336)
	at io.jenkins.plugins.coverage.model.CoverageNode$$Lambda$2032/0x0000000801f41040.apply(Unknown Source)
	at java.base@11.0.15/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base@11.0.15/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base@11.0.15/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base@11.0.15/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base@11.0.15/java.util.stream.ReferencePipeline.reduce(ReferencePipeline.java:553)
	at io.jenkins.plugins.coverage.model.CoverageNode.getCoverage(CoverageNode.java:337)
	at io.jenkins.plugins.coverage.model.CoverageBuildAction.hasCoverage(CoverageBuildAction.java:165)
	at io.jenkins.plugins.coverage.model.visualization.dashboard.ProjectCoverage.getCoverage(ProjectCoverage.java:29)
	at io.jenkins.plugins.coverage.model.visualization.dashboard.CoverageColumn.getCoverageValue(CoverageColumn.java:136)
	…
	at lib.JenkinsTagLib$projectView$1.call(Unknown Source)
	at hudson.model.View.main.run(main.groovy:16)

@dduportal dduportal added this to the infra-team-sync-2022-07-05 milestone Jul 4, 2022
@NotMyFault NotMyFault mentioned this issue Jul 10, 2022
Closed
@dduportal
Copy link
Contributor

I'm not sure what should we do (I don't even know what is "Code Coverage Column"). Could one of you give us a pointer to the next steps so the infra team can do something about it?

@NotMyFault
Copy link
Member Author

NotMyFault commented Jul 10, 2022 •
edited
Loading

Afaik, that's added by https://github.com/jenkinsci/code-coverage-api-plugin and is part of the default pipeline library used for Jenkinsfiles as well.

uhafner added a commit to jenkinsci/code-coverage-api-plugin that referenced this issue Jul 10, 2022
@uhafner
Use the coverages that are persisted within the action if possible.
b785785 
Otherwise, the whole coverage result XML will be loaded just for a single value.
See jenkins-infra/helpdesk#3029 for details.
@uhafner uhafner mentioned this issue Jul 10, 2022
Merged
@uhafner
Copy link

uhafner commented Jul 10, 2022

It would be helpful if someone would report such problems in the plugin, otherwise it's hard to know that we have a problem here 🤷

uhafner added a commit to jenkinsci/code-coverage-api-plugin that referenced this issue Jul 10, 2022
@uhafner
Use the coverages that are persisted within the action if possible.
1db93ae 
Otherwise, the whole coverage result XML will be loaded just for a single value.
See jenkins-infra/helpdesk#3029 for details.
@uhafner
Copy link

uhafner commented Jul 10, 2022

Fixed in https://github.com/jenkinsci/code-coverage-api-plugin/releases/tag/v3.0.3

@medianick
Copy link

This might be what we experienced after the 3.0.0 upgrade to the Code Coverage API plugin (which we hadn't definitively traced to that plugin, so hadn't yet reported it as a bug, but suspected it): namely, a significant performance regression when viewing multibranch pipeline jobs (listing branches or PRs). For us, a display of ~100 PRs now takes over 60 seconds.

@dduportal dduportal self-assigned this Jul 11, 2022
@dduportal
Copy link
Contributor

Wow that was a tricky bug (at least from a Jenkins admin point of view).

Many thanks @daniel-beck for the threaddump and analysis, many thanks @uhafner for the quick fix and many thanks @NotMyFault for the help and raising the issue!

Latest Coverage plugin (and dependencies) had been applied to ci.jenkins.io: the page https://ci.jenkins.io/job/Core/job/jenkins/view/change-requests/ shows in less than 2s on my machine, so closing this issue.

@dduportal dduportal mentioned this issue Jul 26, 2022
Open
uhafner added a commit to jenkinsci/code-coverage-api-plugin that referenced this issue Sep 30, 2022
@uhafner
Use the coverages that are persisted within the action if possible.
e064b33 
Otherwise, the whole coverage result XML will be loaded just for a single value.
See jenkins-infra/helpdesk#3029 for details.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dduportal dduportal

Labels
ci.jenkins.io
Projects
None yet
Milestone
infra-team-sync-2022-07-12
7 participants
@MarkEWaite @medianick @uhafner @dduportal @daniel-beck @NotMyFault @smerle33