Overhaul of type propagation #1268

oxisto · 2023-07-29T09:23:35Z

This PR is a major overhaul of the existing type listeners. This should greatly improve type parsing. The PR is actually slightly too big, but unfortunately a lot of things were dependent on each other and could not be fixed in smaller steps.

Changes

Renamed TypeListener into TypeObserver and made its documentation a lot clearer
Looked carefully at every Node, to see whether we really need a TypeObserver and (hopefully) removed all loops in type propagation
Smaller changes and fixes to the type handling of the CXXLanguageFrontend
Replaced all usages of a BinaryOperator using the = operator with an AssignExpression. Creating a binary operator with = or a compound assignment will now throw a TranslationException
Added support for compound assignments using AssignExpression in the ValueEvaluator and MultiValueEvaluator
Removed the type cache that was used in parallel parsing. Now, the type system is active even if parallel parsing is used. This probably needs a bit more testing, if this does not lead to any concurrent modification errors, but for now it seems to be fine
With the removal of the type caching, I also got rid of the slightly weird _type variables and the setType function. Now only a type variable exists and a Kolin setter is used to inform the type observers.
Moved the creation of DFG edges of an InitializerListExpression back to the DFGPass instead of the class itself. It was moved there because of performance issues, which seem to be resolved now.
Introduced a new shortcut that allows to "invoke" a list of nodes using a Kotlin class and an index (which defaults to 0). This can be used to retrieve a certain index in a node list and cast it to a specific class - or return null if its not castable. For example: assign.lhs<DeclaredReferenceExpression>() will retrieve the first element in lhs of an AssignExpression and cast it to DeclaredReferenceExpression?.

Fixes

Fixes #204
Fixes #705
Fixes #1071
Fixes #1260
Fixes #1262
Fixes #1263
Fixes #1269
Fixes #1270

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/types/HasType.kt

cpg-neo4j/build.gradle.kts

cpg-language-typescript/src/main/nodejs/package-lock.json

cpg-language-java/src/test/kotlin/de/fraunhofer/aisec/cpg/graph/types/TypeTests.kt

cpg-language-java/src/main/kotlin/de/fraunhofer/aisec/cpg/passes/JavaCallResolverHelper.kt

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/passes/ControlFlowSensitiveDFGPass.kt

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/types/HasType.kt

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/statements/expressions/BinaryOperator.kt

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/ExpressionBuilder.kt

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/TranslationContext.kt

maximiliankaul

Python: LGTM.
Other changes as far as I understand them 👍

cpg-language-python/src/main/python/CPGPython/_statements.py

cpg-analysis/src/main/kotlin/de/fraunhofer/aisec/cpg/analysis/MultiValueEvaluator.kt

Removed almost all type listeners for now. This will break anything and we need to rebuild.

…asType.kt Co-authored-by: KuechA <31155350+KuechA@users.noreply.github.com>

Co-authored-by: Maximilian Kaul <maximilian.kaul@aisec.fraunhofer.de>

sonarcloud · 2023-08-09T08:04:08Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
7 Code Smells

75.7% Coverage
1.0% Duplication

Co-authored-by: Alexander Kuechler <alexander.kuechler@aisec.fraunhofer.de> Co-authored-by: Maximilian Kaul <maximilian.kaul@aisec.fraunhofer.de>

KuechA reviewed Jul 31, 2023

View reviewed changes

cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/types/HasType.kt Show resolved Hide resolved

oxisto force-pushed the new-type-propagation branch 3 times, most recently from e2a6c51 to a119c23 Compare July 31, 2023 19:39

oxisto mentioned this pull request Aug 3, 2023

Using AssignExpression in C/C++ frontend to model assignments #1264

Closed

oxisto force-pushed the new-type-propagation branch from b447307 to 23eb330 Compare August 5, 2023 13:41

oxisto marked this pull request as ready for review August 5, 2023 18:12

oxisto requested review from peckto, maximiliankaul and konradweiss as code owners August 5, 2023 18:12

oxisto force-pushed the new-type-propagation branch 3 times, most recently from 45fd258 to af60ff3 Compare August 7, 2023 09:13

KuechA reviewed Aug 7, 2023

View reviewed changes

oxisto force-pushed the new-type-propagation branch 2 times, most recently from 6d6045b to 4c28ecb Compare August 8, 2023 07:36

maximiliankaul approved these changes Aug 8, 2023

View reviewed changes

KuechA mentioned this pull request Aug 8, 2023

StackOverflowError in subtype listening system #1194

Closed

oxisto and others added 12 commits August 9, 2023 09:55

Overhaul of type propagation

c9feecc

Removed almost all type listeners for now. This will break anything and we need to rebuild.

Removed weird type cache

8224561

Split up type and assigned type change events

95c2a99

More assigned types

c501749

Auto type

94eef13

Fix one problem with dfg test

25ef1b4

Fixed some C++ tests

2d5be9c

Fixed more tests

e399211

auto propagation

bda4f28

assign instead of binop

a1b1657

MAJOR hack to support some function pointers

fc32802

Made function pointers work with a very very very hacky approach

8008ed9

oxisto and others added 21 commits August 9, 2023 09:55

Using assign expression in python

f93b449

Removing any assignments from binary operator

b35d6ea

More type propagation tests

67e31ed

Added more complex type propagation example and test

fe44c11

Trying to improve test coverage slightly

1a88461

Truncate numeric types when propagating

79bced3

Added some more primitive types

93b7d43

Improved access to typemanager / ctx in C++ frontend

e049899

Slightly less warnings and convience type

3e216e5

Adding more fancy c++ types

e5fda0b

Trying to catch C++ conversion functions

f28ff31

Addressed first round of review feedback

4abff03

Update cpg-core/src/main/kotlin/de/fraunhofer/aisec/cpg/graph/types/H…

31cf293

…asType.kt Co-authored-by: KuechA <31155350+KuechA@users.noreply.github.com>

Addressed second round of review comments

85477cb

Moved SecondaryTypeEdge to upper class

bb4ec5e

Added documentation in HasType

abd0c21

Apply suggestions from code review

cb9c649

Co-authored-by: Maximilian Kaul <maximilian.kaul@aisec.fraunhofer.de>

Added comment to BinaryOperator propagation

c0f89d3

Removed warning when truncating types

46c7353

Add comment

74efaa7

Format

ec24e91

oxisto force-pushed the new-type-propagation branch from 7f2fb44 to ec24e91 Compare August 9, 2023 07:55

KuechA approved these changes Aug 9, 2023

View reviewed changes

oxisto enabled auto-merge (squash) August 9, 2023 07:57

oxisto merged commit 8b9b297 into main Aug 9, 2023

oxisto deleted the new-type-propagation branch August 9, 2023 08:04

oxisto added a commit that referenced this pull request Aug 9, 2023

Overhaul of type propagation (#1268)

490562d

Co-authored-by: Alexander Kuechler <alexander.kuechler@aisec.fraunhofer.de> Co-authored-by: Maximilian Kaul <maximilian.kaul@aisec.fraunhofer.de>

oxisto mentioned this pull request Aug 9, 2023

Improvements of Go language frontend #1276

Merged

oxisto added a commit that referenced this pull request Aug 9, 2023

Overhaul of type propagation (#1268)

9638fff

Co-authored-by: Alexander Kuechler <alexander.kuechler@aisec.fraunhofer.de> Co-authored-by: Maximilian Kaul <maximilian.kaul@aisec.fraunhofer.de>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul of type propagation #1268

Overhaul of type propagation #1268

oxisto commented Jul 29, 2023 •

edited

Loading

maximiliankaul left a comment

sonarcloud bot commented Aug 9, 2023

Overhaul of type propagation #1268

Overhaul of type propagation #1268

Conversation

oxisto commented Jul 29, 2023 • edited Loading

Changes

Fixes

maximiliankaul left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Aug 9, 2023

oxisto commented Jul 29, 2023 •

edited

Loading