About result IDs and structure

[baruchiro]

The current results structure includes two identification fields: ID and Source:

- id: example.txt
  source: git show 797f277509117a61b81eb97e1c6f509b5addf0c2:testdata/example.txt
  description: Private Key
  startline: 2
  endline: 2
  startcolumn: 3
  endcolumn: 0
  value: |-
    -----BEGIN RSA PRIVATE KEY-----
    MIICWw
    -----END RSA PRIVATE KEY----

The source is the identifier of an Item, before finding one or more secrets inside it.
The id is the source considered as a Full Path of a file, and cutting it to the file name. (I think it is a legacy code).

SimilarityID

In Checkmarx SAST and Kics, we are calculating a "SimilarityID", which is considering multiple properties of a result, and hashing it to a string. (Kics implementation)

Questions

• Do we want to remove the ID field?
• Do we want a unique identification field?
• Can we find enough properties for each Item/Secret to make them unique? (*We should avoid properties like "line number" because they might be changed by the content).

[baruchiro]

BTW, I used the Source to give some extra information (and it's still unique), for example: git show 797f277509117a61b81eb97e1c6f509b5addf0c2:testdata/example.txt.

I think we should change the Secret.Description to this information, and move the description: Private Key to be under a new field called SecretType.

[baruchiro]

See #124

[baruchiro]

I will research to check if we can create a unique ID for every secret in each plugin even between scans (examples: forked git repo).

The advantages are differentiating between multiple secrets on the same page, and the same secrets throughout history. The ID should be something that will not be changed on page edit or so.
Ideally, also to avoid duplicates (#41), the best approach is the Git approach- reading the changes of each version, instead of reading all the versions.

To differentiate between two secrets in the same item, I suggest adding the secret itself (or part of it) to the ID (hashed, of course).

Base ID structure: Result Id = sha1(plugin name + rule id + item ID [+ version ID] + secret)

Confluence

Since Confluence is retrieving the same secret over all the versions, I think the ID should not include the version ID. Yes, it will create duplicated IDs, but it is expected since they are representing the same secret.

ID: sha1("Confluence" + rule id + page.ID + secret)

If the content on the page will be changed, the ID will stay the same.

Discord, Slack

In Discord we are not reading the message history.

ID: sha1("Discord"/"Slack" + rule id + guildId/channel.Name + message.ChannelID + message.ID/message.Timestamp + secret)

FileSystem, Paligo

There is no history for files, and one file may contain multiple secrets:

ID: sha1("FileSystem"/"Paligo" + rule id + filePath/document.ID + secret)

If the file will be changed, the ID will stay the same (or the secret will not be there anymore.)

Git

In Git, there is a commit, file, and chunks. We can add the chunk number to the ID, but assuming that even the chunk may contain more than one secret, I think it is not needed.

ID: sha1("Git" + rule id + SHA + file.NewName + secret)

The Git history will not change (unless the user will recreate its history).