Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

findValueForJsonPath always returns 'path can not be null or empty' #115

Closed
1 task
studiovlijmscherp opened this issue Jan 10, 2023 · 9 comments
Closed
1 task

findValueForJsonPath always returns 'path can not be null or empty' #115

studiovlijmscherp opened this issue Jan 10, 2023 · 9 comments

Comments

@studiovlijmscherp
Copy link

Summary

We're crawling all our repo's to find version numbers of certain packages.
Console logs 'processing repo ... xyz' for all repos so we are connected to Github.

When using findPropertyValueInYamlFile (for another property). Parsing works as expected:

    indicatorsToFetchByFile:
      "[composer.lock]":
          - name: content_hash
            type: findPropertyValueInYamlFile
            params:
              propertyName: "content-hash"

When using findValueForJsonPath, an error is returned:

indicatorsToFetchByFile:
      "[composer.lock]":
          - name: content_hash_path
            type: findValueForJsonPath
            params:
              propertyName: "$.content-hash"

error:

2023-01-10 14:05:06.368 ERROR 92041 --- [           main] ication$$EnhancerBySpringCGLIB$$ffb279df : problem while running github crawler
java.lang.IllegalArgumentException: path can not be null or empty

Type of Issue

It is a :

  • [x ] bug
  • request
  • [ x] question regarding the documentation

Your Environment

  • Version used: github-crawler-starter-2.1.0-exec.jar
  • OS and version: MacOs 13.0.1 (22A400)
@vincent-fuchs
Copy link
Contributor

Hi, thanks for reporting the issue. Is there any chance to get a more complete stacktrace ? I can't do much with what you've provided, it doesn't even show from where the exception is thrown.

@vincent-fuchs
Copy link
Contributor

vincent-fuchs commented Jan 14, 2023 •
edited
Loading

one strong possibility though :

for the findPropertyValueInYamlFile parser, the expected parameter is indeed propertyName : https://github.com/societe-generale/github-crawler/blob/master/github-crawler-core/src/main/kotlin/com/societegenerale/githubcrawler/parsers/YamlParserForPropertyValue.kt#L12

However, for findValueForJsonPath parser, the expected parameter is jsonPath : https://github.com/societe-generale/github-crawler/blob/master/github-crawler-core/src/main/kotlin/com/societegenerale/githubcrawler/parsers/JsonPathParser.kt#L18

please give it a try and let me know if that helps

@studiovlijmscherp
Copy link
Author

Thanks!

Changing propertyName to jsonPath did get met one step further. I'm able to find some basic values.
Searching for a key/value still returns an error.

Trying to search for these vales:

    indicatorsToFetchByFile:
      "[composer.lock]":
          - name: package_version
            type: findValueForJsonPath
            params:
              jsonPath: "$.packages[?(@.name == 'sulu/sulu')].name"

Returns the following error:

2023-01-16 15:02:30.110 ERROR 31371 --- [main] lication$$EnhancerBySpringCGLIB$$d72a6fb : problem while running github crawler
java.lang.ClassCastException: net.minidev.json.JSONArray cannot be cast to java.lang.String at com.societegenerale.githubcrawler.parsers.JsonPathParser.internalParseFileContentForIndicator(JsonPathParser.kt:37) ~[github-crawler-core-2.1.0.jar!/:2.1.0]

Attached is an example of the file I'm crawling
composer.lock.zip

When evaluating the jsonPath @ https://jsonpath.com/ - the correct values are being returned.

@vincent-fuchs
Copy link
Contributor

I think the problem is that your jsonpath expression returns an array of 1 String, not a String

image

see https://stackoverflow.com/questions/23608050/getting-a-single-value-from-a-json-object-using-jsonpath : it seems to be a common issue with JsonPath !

I've committed a fix, you can build the latest version locally to try without waiting for the next release

@studiovlijmscherp
Copy link
Author

Trying to build, but it seems some dependencies are missing?

[ERROR] Failed to execute goal on project github-crawler-starter: Could not resolve dependencies for project com.societegenerale.github-crawler:github-crawler-starter:jar:2.1.1-SNAPSHOT: Could not find artifact com.societegenerale.github-crawler:github-crawler-autoconfigure:jar:2.1.1-SNAPSHOT -> [Help 1]

Trying to build github-crawler/github-crawler-starter using:
mvn package

When changing the versions in github-crawler-starter/pom.xml from 2.1.1-SNAPSHOT to 2.1.0 the build completes succesful. But - of course - without the latest update.

@vincent-fuchs
Copy link
Contributor

you need to build it from the root (github-crawler, not github-crawler/github-crawler-starter), so that it builds all the modules, including the github-crawler-starter.jar you need that will be in github-crawler-starter/target folder

@studiovlijmscherp
Copy link
Author

Unfortunately no succes:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for github-crawler 2.1.1-SNAPSHOT:
[INFO]
[INFO] github-crawler ..................................... SUCCESS [  1.233 s]
[INFO] github-crawler-core ................................ FAILURE [ 42.796 s]
[INFO] github-crawler-autoconfigure ....................... SKIPPED
[INFO] github-crawler-starter ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  44.964 s
[INFO] Finished at: 2023-01-23T09:52:49+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (java-test-compile) on project github-crawler-core: Fatal error compiling: invalid target release: 11 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.10.1:testCompile (java-test-compile) on project github-crawler-core: Fatal error compiling
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:375)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error compiling
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:1220)
    at org.apache.maven.plugin.compiler.TestCompilerMojo.execute (TestCompilerMojo.java:183)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.codehaus.plexus.compiler.CompilerException: invalid target release: 11
    at org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess (JavaxToolsCompiler.java:197)
    at org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile (JavacCompiler.java:182)
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:1209)
    at org.apache.maven.plugin.compiler.TestCompilerMojo.execute (TestCompilerMojo.java:183)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.IllegalArgumentException: invalid target release: 11
    at com.sun.tools.javac.main.OptionHelper$GrumpyHelper.error (OptionHelper.java:103)
    at com.sun.tools.javac.main.Option$12.process (Option.java:216)
    at com.sun.tools.javac.api.JavacTool.processOptions (JavacTool.java:217)
    at com.sun.tools.javac.api.JavacTool.getTask (JavacTool.java:156)
    at com.sun.tools.javac.api.JavacTool.getTask (JavacTool.java:107)
    at com.sun.tools.javac.api.JavacTool.getTask (JavacTool.java:64)
    at org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess (JavaxToolsCompiler.java:135)
    at org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile (JavacCompiler.java:182)
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:1209)
    at org.apache.maven.plugin.compiler.TestCompilerMojo.execute (TestCompilerMojo.java:183)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:370)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:351)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:171)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:163)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:298)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:960)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :github-crawler-core

@vincent-fuchs
Copy link
Contributor

looks like you're not building it with JDK 11, while you should

@studiovlijmscherp
Copy link
Author

Thanks! Fixed. findValueForJsonPath returns correct values. Awesome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@studiovlijmscherp @vincent-fuchs