-
Notifications
You must be signed in to change notification settings - Fork 316
5. End to End Testing
The basic structure of the end-to-end testing module is discussed in the corresponding readme file.
To be able to create the test data, some examples from science, which have addressed the concealment of plagiarism, were used to ensure the greatest possible coverage of the JPlag functionality. Here, the changes were split as finely as possible and applied to the levies. The following elaborations were used for this:
- Detecting source code plagiarism on introductory programming course assignments using a bytecode approach
- Detecting Disguised Plagiarism
These elaborations provide basic ideas on how a modification of the plagiarized source code can look like or be adapted. These code adaptations refer to a wide range of changes starting from adding/removing comments to architectural changes in the deliverables.
- (1) Inserting comments or empty lines
- (2) Changing variable names or function names
- (3) Insertion of unnecessary or changed code lines
- (4) Changing the program flow (statements and functions must be independent from each other)
- (1) Variable declaration at the beginning of the program
- (2) Combining declarations of variables
- (3) Reuse of the same variable for other functions
- (5) Changing control structures
- (1) for(...) to while(...)
- (2) if(...) to switch-case
- (6) Modification of expressions
- (1) (X < Y) to !(X >= Y) and ++x to x = x + 1
- (7) Splitting and merging statements
- (1) x = getSomeValue(); y = x- z; to y = (getSomeValue() - Z;
- (8) Inserting unnecessary casts
These changes were now transferred to a base class and thus the plagiarism was created. The named base class was provided with the individual changes. The numbers in the list shown above are intended for the traceability of the test data. Here the test data filenames were named with the respective changes. Example: SortAlgo4d1 contains the changes "Variable declaration at the beginning of the program". If several points are combined, this is separated by "_" e.g.: SortAlgo1_3 contains "(1) Inserting comments or empty lines" and "(3) Insertion of unnecessary or changed code lines".
The following code examples show how these changes affect the program code and also how the detection of JPLag behaves. All the code examples shown and more can be found at testdata-resources-SortAlgo.
Adding empty lines or comments affects the normalization of the output. If the End-To-End tests fail with these changes, it means that something has changed in the normalization, e.g. removing empty lines or recognizing comments no longer works.
In the following, the modified base class looks like this:
Original:
public void BubbleSortWithoutRecursion(Integer arr[]) {
for(int i = arr.length; i > 1 ; i--) {Plagiarized:
/*
Unnecessary comment
*/
public void BubbleSortWithoutRecursion(Integer arr[]) {
//Unnecessary comment
for(int i = arr.length; i > 1 ; i--) {As expected, the resulting outputs have a match of 100% (JPLag result):
"SortAlgo-SortAlgo1" : {
"minimal_similarity" : 100.0,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
},Changing variable names and function names has, like point 1, also the goal of detecting adjustments in the normalization level. If the End-To-End tests fail with these changes, it means that something has changed in the normalization, e.g. creating constants function and variable names.
Orginal:
private final <T> void swap(T[] arr, int i, int j) {
T t = arr[i];
arr[i] = arr[j];
arr[j] = t;
}Plagiarized:
private final <T> void paws(T[] otherArr, int i, int j) {
T t = otherArr[i];
otherArr[i] = otherArr[j];
otherArr[j] = t;
}As expected, the resulting outputs have a match of 100% (JPLag result):
"SortAlgo-SortAlgo2" : {
"minimal_similarity" : 100.0,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
},In contrast to points 1 and 2, adding unnecessary code lines reduces the token generation. This has the consequence that the recognition can no longer be 100% sure whether plagiarism is present or not. The failure of the end-to-end tests in these cases means that either the tokens have been adjusted, the normalization has changed the function separation or something has changed in the minimum token numbers. This can be easily seen by running the end-to-end tests in different options. this will be shown in the next result examples.
Original:
private final <T> void swap(T[] arr, int i, int j) {
T t = arr[i];
arr[i] = arr[j];
arr[j] = t;
}Plagiarized:
private final <T> void swap(T[] arr, int i, int j) {
var tempVar1 = 0;
if (true) {
T t = arr[i];
arr[i] = arr[j];
arr[j] = t;
var tempVar2 = 0;
tempVar2++;
tempVar2 = tempVar2 + 1;
}
}The results for the recognition already allow first recognition changes. Here the change of the minimum_token_match also has an effect on the result, which was not the case with (1) and (2).
[{"options" : {
"minimum_token_match" : 1
},
"SortAlgo-SortAlgo3" : {
"minimal_similarity" : 81.159424,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
},
}][{"options" : {
"minimum_token_match" : 15
},
"SortAlgo-SortAlgo3" : {
"minimal_similarity" : 57.971016,
"maximum_similarity" : 71.42857,
"matched_token_number" : 40
},
}]This subitem breaks down into three more change methods to maintain fine granularity:
- (1) Variable declaration at the beginning of the program
public class SortAlgo4d1 {
private int firstCounter;
private int arrayLenght;
private int swapVarI;
private int swapVarJ;- (2) Combining declarations of variables
public class SortAlgo4d2 {
private int firstCounter,swapVarJ,arrayLenght ,swapVarI;- (3) Reuse of the same variable for other functions
public class SortAlgo4d3 {
private int firstCounterAndArrayLenghtAndswapVarJ ,swapVarI;The adjustments to the program flow with the previous instantiation of the variables were also made: Original:
if (n == 1)
{
return;
}Plagiarized:
firstCounter = n;
if (firstCounter == 1) {
return;
}The results of the individual adjustment are as follows:
"SortAlgo-SortAlgo4d1" : {
"minimal_similarity" : 87.30159,
"maximum_similarity" : 98.21429,
"matched_token_number" : 55
},
"SortAlgo-SortAlgo4d2" : {
"minimal_similarity" : 87.5,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
},
"SortAlgo-SortAlgo4d3" : {
"minimal_similarity" : 90.32258,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
},The change of the control structure in the program also indicates a change of the token generation in case of faulty tests. In contrast to (4), however, these are specially designed for other tokens that are made for if, else, ... structures.
These changes were made to the SortAlgo test data in a plagiarized form.
Original:
public void BubbleSortRecursion(Integer arr[], int n) {
if (n == 1)
{
return;
}
for (int i = 0; i < n - 1; i++)
{
if (arr[i] > arr[i + 1])
{
swap(arr, i , i+1);
}
}
BubbleSortRecursion(arr, n - 1);
}Plagiarized:
public void BubbleSortRecursion(Integer arr[], int n) {
switch (n) {
case 1:
return;
}
int i = 0;
while(i < n-1)
{
var tempBool = arr[i] > arr[i + 1];
if (tempBool) {
swap(arr, i, i + 1);
}
i++;
}
BubbleSortRecursion(arr, n - 1);
}Here it is remarkable which affects the adjustment of the minimum_token_match has on the recognition of the plagiarism.
Changes of the token generation as well as the minimum_token_match have an effect on this kind of End-To-End test.
"options" : {
"minimum_token_match" : 1
},
"tests" : {
"SortAlgo-SortAlgo5" : {
"minimal_similarity" : 82.14286,
"maximum_similarity" : 82.14286,
"matched_token_number" : 46
}, "options" : {
"minimum_token_match" : 15
},
"tests" : {
"SortAlgo-SortAlgo5" : {
"minimal_similarity" : 0.0,
"maximum_similarity" : 0.0,
"matched_token_number" : 0
},Changing the order of compare also changes the order of the program flow which is difficult to determine the exact effect of plagiarism. Here the statements (X < Y) to !(X >= Y) and ++x to x = x + 1 are changed. Since the syntax should be recognized however as expression, the pure change of the expression has little effect on their plagiarism recognition.
Orginal:
public void BubbleSortRecursion(Integer arr[], int n) {
if (n == 1)
{
return;
}
for (int i = 0; i < n - 1; i++)
{
if (arr[i] > arr[i + 1])
{
swap(arr, i , i+1);
}
}
BubbleSortRecursion(arr, n - 1);
}Plagiarized:
public void BubbleSortRecursion(Integer arr[], int n) {
if (n != 1)
{
for (int i = 0; !(i >= (n - 1));)
{
if (!(arr[i] <= arr[i + 1]))
{
swap(arr, i , i+1);
}
i = i + 1;
}
BubbleSortRecursion(arr, n - 1);
}
else
{
return;
}
}Results:
{
"options" : {
"minimum_token_match" : 1
},
"SortAlgo-SortAlgo6" : {
"minimal_similarity" : 83.58209,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
}, "options" : {
"minimum_token_match" : 15
},
"SortAlgo-SortAlgo6" : {
"minimal_similarity" : 43.28358,
"maximum_similarity" : 51.785713,
"matched_token_number" : 29
},The merging or splitting of statements results in changing the token for the respective plagiarism detection.
Here code lines are either fetched from functions or stored in functions like x = getSomeValue(); y = x- z; to y = (getSomeValue() - Z.
Original:
[...]
swap(arr, i , i+1);
[...]
if (arr[innerCounter] > arr[innerCounter + 1]) {
[...]Plagiarized:
[...]
swap(arr, i, add(i , 1));
[...]
if (arr[innerCounter] > arr[add(innerCounter , 1)]) {
[...]
private int add(int value1, int value2)
{
return value1 + value2;
}
private int subtract(int value1, int value2)
{
return value1 - value2;
}Results:
"options" : {
"minimum_token_match" : 1
},
"tests" : {
"SortAlgo-SortAlgo7" : {
"minimal_similarity" : 76.712326,
"maximum_similarity" : 100.0,
"matched_token_number" : 56
}, "options" : {
"minimum_token_match" : 15
},
"tests" : {
"SortAlgo-SortAlgo7" : {
"minimal_similarity" : 49.315067,
"maximum_similarity" : 64.28571,
"matched_token_number" : 36
},The results and the test coverage of the end-to-end tests strongly depend on the tested plagiarisms. It is also important to use and test many different options of the API offered by JPlag, as these have a direct influence on the detection and are therefore also important for the change detection.
To summarize
- (1) and (2) test normalization level
- (3) to (7) the token generation level
If a result differs only in the options, it is very likely that the change is in the configuration of the minimum_token_match.
This means that if Option1 does not change in the result of the detection, but the result in Option2 does, this is the basis of the minimum_token_match.
java: (1)SortAlgo-SortAlgo5 --> passed
java: (15)SortAlgo-SortAlgo5 --> failed