Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures when scanning pom.xml files with a <parent> tag #1321

Open
LironJit opened this issue Oct 13, 2024 · 6 comments
Open

Failures when scanning pom.xml files with a <parent> tag #1321

LironJit opened this issue Oct 13, 2024 · 6 comments
Assignees
@cuixq
Labels
bug Something isn't working

Comments

@LironJit
Copy link

Hey osv team, I've been running into some issues with scanning pom.xml files which have a <parent> tag:

osv-scanner v1.7.3 and latest (v1.9.0) are unable to scan this repository: https://github.com/javaee/javamail but after deleting the <parent> tags from pom.xml files (for example javamail/taglib/pom.xml) the files are scanned successfully.

Attempted to scan lockfile but failed: javamail/android/activation/pom.xml
Attempted to scan lockfile but failed: javamail/android/mail/pom.xml
Attempted to scan lockfile but failed: javamail/android/pom.xml
Attempted to scan lockfile but failed: javamail/client/pom.xml
Attempted to scan lockfile but failed: javamail/demo/pom.xml
Attempted to scan lockfile but failed: javamail/dsn/pom.xml
Attempted to scan lockfile but failed: javamail/gimap/pom.xml
Attempted to scan lockfile but failed: javamail/imap/pom.xml
Attempted to scan lockfile but failed: javamail/javadoc/pom.xml
Attempted to scan lockfile but failed: javamail/logging/pom.xml
Attempted to scan lockfile but failed: javamail/mail/pom.xml
Attempted to scan lockfile but failed: javamail/mailapi/pom.xml
Attempted to scan lockfile but failed: javamail/mailapijar/pom.xml
Attempted to scan lockfile but failed: javamail/mailhandler/pom.xml
Attempted to scan lockfile but failed: javamail/mbox/native/pom.xml
Attempted to scan lockfile but failed: javamail/mbox/pom.xml
Attempted to scan lockfile but failed: javamail/outlook/pom.xml
Attempted to scan lockfile but failed: javamail/parent-distrib/pom.xml
Attempted to scan lockfile but failed: javamail/pom.xml
Attempted to scan lockfile but failed: javamail/pop3/pom.xml
Attempted to scan lockfile but failed: javamail/publish/pom.xml
Attempted to scan lockfile but failed: javamail/servlet/pom.xml
Attempted to scan lockfile but failed: javamail/smtp/pom.xml
Attempted to scan lockfile but failed: javamail/taglib/pom.xml
Attempted to scan lockfile but failed: javamail/webapp/pom.xml

Also notice when I'm using v1.7.3 I'm able to scan the following (which has a <parent> tag):

✗ docker run --rm -v $(pwd):/code ghcr.io/google/osv-scanner:v1.7.3 scan -r /code/
Scanning dir /code/
Scanned /code/pom.xml file and found 1 package
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
| OSV URL                             | CVSS | ECOSYSTEM | PACKAGE                 | VERSION | SOURCE       |
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
| https://osv.dev/GHSA-5gj6-62g7-vmgf | 4.3  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-8h4x-xvjp-vf99 | 6.5  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-c5hg-mr8r-f6jp | 9.1  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-c5vj-wp4v-mmvx | 7.6  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
| https://osv.dev/GHSA-xh6m-7cr7-xx66 | 7.6  | Maven     | com.hazelcast:hazelcast | 5.1     | code/pom.xml |
+-------------------------------------+------+-----------+-------------------------+---------+--------------+
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.apache.seatunnel</groupId>
        <artifactId>seatunnel-e2e</artifactId>
        <version>${revision}</version>
    </parent>
    
    <properties>
        <hazelcast.version>5.1</hazelcast.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>${hazelcast.version}</version>
        </dependency>
    </dependencies>
</project>

However, running v1.9.0 will not work unless I remove the <parent> tag:

✗ docker run --rm -v $(pwd):/code ghcr.io/google/osv-scanner:v1.9.0 scan -r /code/
Scanning dir /code/
Attempted to scan lockfile but failed: /code/pom.xml
<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    
    <properties>
        <hazelcast.version>5.1</hazelcast.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.hazelcast</groupId>
            <artifactId>hazelcast</artifactId>
            <version>${hazelcast.version}</version>
        </dependency>
    </dependencies>
</project>

Thanks in advance!
@oliverchang
Copy link
Collaborator

Thanks for the report! @cuixq can you please take a look at this?

@cuixq
Copy link
Contributor

cuixq commented Oct 14, 2024

@LironJit we launched transitive scanning for Maven pom.xml in version 1.8.1 which makes network requests to fetch parent POMs.

When I investigated this issue I noticed the following error:

xml: encoding "iso-8859-1" declared but Decoder.CharsetReader is nil

and I am going to make a fix for this.

However this issue still stops the scanner working for pom.xml without <parent> tag.

Could you give us more information on the error you see when running the scanner? Thanks.

@cuixq cuixq added the bug Something isn't working label Oct 14, 2024
@cuixq cuixq mentioned this issue Oct 14, 2024
Merged
cuixq added a commit that referenced this issue Oct 14, 2024
@cuixq
fix: set CharsetReader and Entity when reading pom.xml (#1325)
d3b75df 
#1321

If non UTF-8 encoding is declared in pom.xml, we need to set
`CharsetReader` to avoid the error.
@desolatorxxl
Copy link

I have the same problem and would like to help, however I'm not sure how.

For starters I added the parent error to Attempted to scan lockfile but failed here:

r.Errorf("Attempted to scan lockfile but failed: %s: %s\n", err, path)

Now I get:

$ go run . --verbosity verbose -skip-git -r myservice
Scanning dir myservice
Attempted to scan lockfile but failed: failed extracting myservice/api-mock/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/api-mock/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/myservice/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/myservice/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/myservice-st/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/myservice-st/pom.xml
Attempted to scan lockfile but failed: failed extracting myservice/pom.xml: failed to merge parents: failed to get Maven project com.company.commons:maven.artifactory:2.0: failed to fetch Maven project com.company.commons:maven.artifactory@2.0: myservice/pom.xml
No package sources found, --help for usage information.
exit status 128

(Just cloned osv-scanner from master at e054385)

Which seems to be related to osv-scanner not resolving artifacts from private artifact registries.

However if I try the javamail project, I get some different errors:

$ go run . --verbosity verbose -skip-git -r ./javamail
Scanning dir ./javamail
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/activation/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/mail/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/android/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/client/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/demo/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/dsn/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/gimap/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/imap/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/javadoc/pom.xml file and found 4 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/logging/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mail/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailapi/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailapijar/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mailhandler/pom.xml file and found 1 package
Attempted to scan lockfile but failed: failed extracting /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/native/pom.xml: failed resolving {Maven:com.sun.mail:libmbox[Concrete:1.6.2] {}}: version Maven:com.sun.mail:mbox[Concrete:1.6.2]: not found: /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/native/pom.xml
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/mbox/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/outlook/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/parent-distrib/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/pom.xml file and found 0 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/pop3/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/publish/pom.xml file and found 2 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/servlet/pom.xml file and found 3 packages
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/smtp/pom.xml file and found 1 package
Scanned /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/taglib/pom.xml file and found 4 packages
Attempted to scan lockfile but failed: failed extracting /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/webapp/pom.xml: failed resolving {Maven:com.sun.mail:webapp[Concrete:1.6.2] {}}:
version Maven:com.sun.mail:taglib[Concrete:1.6.2]: not found: /home/user/scratch/osv-scanner/cmd/osv-scanner/javamail/webapp/pom.xml
exit status 127

@cuixq
Copy link
Contributor

cuixq commented Oct 23, 2024 •
edited
Loading

@desolatorxxl thanks for the feedback!

May I ask where do you place your parent pom.xml? On local file system or remote private registry?
OSV-Scanner at master should be able to track the registries defined in <repositories/> - for your case, where do you specify the registry URL?

@desolatorxxl
Copy link

@cuixq all pom.xml files are on a local file system. For clarity the directory structure looks like this:

$ find -name pom.xml
./pom.xml
./api-mock/pom.xml
./myservice/pom.xml
./myservice-st/pom.xml

But the ./pom.xml has, what I believe is, a remote parent, see further below.

I specified the registry URL in my ~/.m2/settings.xml, it looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <servers>
    <server>
      <username>...</username>
      <password>...</password>
      <id>central</id>
    </server>
    <server>
      <username>...</username>
      <password>...</password>
      <id>snapshots</id>
    </server>
  </servers>
  <profiles>
    <profile>
      <repositories>
        <repository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>libs-release</name>
          <url>https://artifactory.org.com:443/artifactory/libs-release</url>
        </repository>
        <repository>
          <snapshots />
          <id>snapshots</id>
          <name>libs-snapshot</name>
          <url>https://artifactory.org.com:443/artifactory/libs-snapshot</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>libs-release</name>
          <url>https://artifactory.org.com:443/artifactory/libs-release</url>
        </pluginRepository>
        <pluginRepository>
          <snapshots />
          <id>snapshots</id>
          <name>libs-snapshot</name>
          <url>https://artifactory.org.com:443/artifactory/libs-snapshot</url>
        </pluginRepository>
      </pluginRepositories>
      <id>artifactory</id>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>artifactory</activeProfile>
  </activeProfiles>
</settings>

I digged around a bit in all the pom.xml files and found out that the top level pom (./pom.xml) references a remote parent, like this:

    <parent>
        <groupId>com.org.common</groupId>
        <artifactId>maven.artifactory</artifactId>
        <version>1.0</version>
    </parent>

The remote pom.xml contains a slightly different incarnation of my settings.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.org.common</groupId>
    <artifactId>maven.artifactory</artifactId>
    <version>1.0</version>
    <packaging>pom</packaging>
    <description>ORG POM for Artifactory</description>
    <organization>
        <name>Private ORG</name>
        <url>https://org.com</url>
    </organization>
    <distributionManagement>
        <repository>
            <id>central</id>
            <name>libs-release</name>
            <url>https://artifactory.org.com:443/artifactory/libs-release-local</url>
        </repository>
        <snapshotRepository>
            <id>snapshots</id>
            <name>libs-snapshot</name>
            <url>https://artifactory.org.com:443/artifactory/libs-snapshot-local</url>
        </snapshotRepository>
    </distributionManagement>
</project>

If I remove that parent, the scan runs successfully.

It seems that is a really odd way of centrally managing repository configuration?

Anyways, today that artifact is not required for the application to run or build, so I will try to get it removed.

@cuixq
Copy link
Contributor

cuixq commented Oct 29, 2024

@desolatorxxl we currently do not support reading repositories from settings.xml and we have a issue opened for this #1269.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cuixq cuixq

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@oliverchang @desolatorxxl @cuixq @LironJit