Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bulkupdateresults #894

Merged
merged 61 commits into from
Aug 17, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
0410e12
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
521f161
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
a56c5c1
UpdateResults API optimized.
msvinaykumar May 23, 2023
3ab5e1f
resolved conflicts
msvinaykumar Jun 6, 2023
b152858
Moved logic to select results into perfProfiles.
msvinaykumar Jun 12, 2023
a5c528c
Date range update recommendations are handled
msvinaykumar Jun 19, 2023
86ac1d1
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
7a0b5a7
endDate is not mandatory
msvinaykumar Jul 20, 2023
3be9c1b
resolved conflicts
msvinaykumar Jul 21, 2023
886b435
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
6581ea0
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
79fac3c
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
52ffa53
add changes for the performance profile table addition in the DB
khansaad May 22, 2023
1c58657
update service methods for performance profiles
khansaad Jun 1, 2023
40a7099
fix issue with the sloData not getting populated and other minor changes
khansaad Jun 6, 2023
e55c51a
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
42477dd
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
a897c68
Added exception conditions.
msvinaykumar May 26, 2023
eb1e941
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
e59e21d
Date range update recommendations are handled
msvinaykumar Jun 19, 2023
8018c25
Bulk upload results
msvinaykumar Jul 14, 2023
38f0c71
resolved conflicts
msvinaykumar Jul 17, 2023
f1f7131
incorporated review comments
msvinaykumar Jul 6, 2023
4cbf5f6
resolved conflicts
msvinaykumar Jul 17, 2023
ad541f0
incorporated review comments
msvinaykumar Jul 25, 2023
a7de1f1
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
4e7ad66
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
e9b0dee
Added exception conditions.
msvinaykumar May 26, 2023
04b3a9f
Merging confilts
msvinaykumar Jun 1, 2023
dec3c9d
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
b1d1eed
resolving conflicts
msvinaykumar Jun 6, 2023
e4a0db9
Moved logic to select results into perfProfiles.
msvinaykumar Jun 12, 2023
9689644
Date range update recommendations are handled
msvinaykumar Jun 19, 2023
dd69e0e
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
2e70475
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
26af430
Added exception conditions.
msvinaykumar May 26, 2023
e8b7c3c
To restrict the number of rows in the result set, the Load results op…
msvinaykumar May 27, 2023
2dfc303
endDate is not mandatory
msvinaykumar Jul 20, 2023
57b33cd
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
eee8dbc
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
0f91741
Added exception conditions.
msvinaykumar May 26, 2023
c053761
To restrict the number of rows in the result set, the Load results op…
msvinaykumar May 27, 2023
801835c
Merging confilts
msvinaykumar Jun 1, 2023
6b787d2
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
b4ca2b6
resolving conflicts
msvinaykumar Jun 6, 2023
198417f
Moved logic to select results into perfProfiles.
msvinaykumar Jun 12, 2023
7ab4fee
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
e9873e9
Added exception conditions.
msvinaykumar May 26, 2023
209e8f1
To restrict the number of rows in the result set, the Load results op…
msvinaykumar May 27, 2023
9dbb7da
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
59d6cd5
UpdateResults API optimized.
msvinaykumar May 23, 2023
0da8d1f
In progress code checked in for updateRecommendation API.
msvinaykumar May 18, 2023
a9785e1
UpdateRecommendation API E2E working code is ready.
msvinaykumar May 20, 2023
0b8e39b
To restrict the number of rows in the result set, the Load results op…
msvinaykumar May 27, 2023
9be2841
Merging confilts
msvinaykumar Jun 1, 2023
676de21
Taking care of duplicate entry to recommendation table
msvinaykumar Jun 1, 2023
d707612
restricted http request thread to 6
msvinaykumar Aug 11, 2023
c78abeb
resolved conflicts
msvinaykumar Aug 11, 2023
f5090a9
tested hole in results for already updated recommendations fix
msvinaykumar Aug 12, 2023
c5fb5e2
merged Conflicts
msvinaykumar Aug 12, 2023
0d82ace
resolved conflicts
msvinaykumar Aug 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
<postgresql-version>42.3.1</postgresql-version>
<hibernatecore-version>6.1.7.Final</hibernatecore-version>
<hibernatecp30-version>6.1.7.Final</hibernatecp30-version>
<hibernate-Validator>8.0.1.Final</hibernate-Validator>
<micrometer-version>1.9.9</micrometer-version>
</properties>
<dependencies>
Expand Down Expand Up @@ -154,6 +155,12 @@
<artifactId>hibernate-c3p0</artifactId>
<version>${hibernatecp30-version}</version>
</dependency>
<dependency>
<groupId>org.hibernate.validator</groupId>
<artifactId>hibernate-validator</artifactId>
<version>${hibernate-Validator}</version>
</dependency>

<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
Expand Down
17 changes: 14 additions & 3 deletions src/main/java/com/autotune/Autotune.java
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import com.autotune.analyzer.exceptions.MonitoringAgentNotFoundException;
import com.autotune.analyzer.exceptions.MonitoringAgentNotSupportedException;
import com.autotune.database.init.KruizeHibernateUtil;
import com.autotune.database.service.ExperimentDBService;
import com.autotune.experimentManager.core.ExperimentManager;
import com.autotune.operator.InitializeDeployment;
import com.autotune.operator.KruizeDeploymentInfo;
Expand All @@ -34,8 +33,10 @@
import io.prometheus.client.exporter.MetricsServlet;
import io.prometheus.client.hotspot.DefaultExports;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.ServerConnector;
import org.eclipse.jetty.servlet.ServletContextHandler;
import org.eclipse.jetty.servlet.ServletHolder;
import org.eclipse.jetty.util.thread.QueuedThreadPool;
import org.hibernate.Session;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand All @@ -53,8 +54,16 @@ public static void main(String[] args) {
ServletContextHandler context = null;

disableServerLogging();
// Create a thread pool with the desired number of threads
QueuedThreadPool threadPool = new QueuedThreadPool();
threadPool.setMaxThreads(KRUIZE_HTTP_THREAD_POOL_COUNT); // Set the maximum number of threads in the pool
Server server = new Server(threadPool);
// Create a connector (e.g., HTTP)
ServerConnector connector = new ServerConnector(server);
connector.setPort(KRUIZE_SERVER_PORT);
// Set the connector to the server
server.addConnector(connector);

Server server = new Server(KRUIZE_SERVER_PORT);
context = new ServletContextHandler();
context.setContextPath(ServerContext.ROOT_CONTEXT);
context.setErrorHandler(new KruizeErrorHandler());
Expand All @@ -74,7 +83,8 @@ public static void main(String[] args) {

try {
InitializeDeployment.setup_deployment_info();
} catch (Exception | K8sTypeNotSupportedException | MonitoringAgentNotSupportedException | MonitoringAgentNotFoundException e) {
} catch (Exception | K8sTypeNotSupportedException | MonitoringAgentNotSupportedException |
MonitoringAgentNotFoundException e) {
e.printStackTrace();
System.exit(1);
}
Expand Down Expand Up @@ -105,6 +115,7 @@ public static void main(String[] args) {

try {
server.start();
server.join();
} catch (Exception e) {
LOGGER.error("Could not start the server!");
e.printStackTrace();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
*******************************************************************************/
package com.autotune.analyzer.exceptions;

import com.autotune.analyzer.serviceObjects.UpdateResultsAPIObject;
import com.google.gson.Gson;
import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.servlet.ErrorPageErrorHandler;
Expand All @@ -23,6 +24,7 @@
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
import java.util.List;

import static com.autotune.analyzer.utils.AnalyzerConstants.ServiceConstants.CHARACTER_ENCODING;
import static com.autotune.analyzer.utils.AnalyzerConstants.ServiceConstants.JSON_CONTENT_TYPE;
Expand All @@ -41,9 +43,12 @@ public void handle(String target, Request baseRequest, HttpServletRequest reques
response.setCharacterEncoding(CHARACTER_ENCODING);
String origMessage = (String) request.getAttribute("javax.servlet.error.message");
int errorCode = response.getStatus();
List<UpdateResultsAPIObject> myList = (List<UpdateResultsAPIObject>) request.getAttribute("data");


PrintWriter out = response.getWriter();
out.append(
new Gson().toJson(new KruizeResponse(origMessage, errorCode, "", "ERROR")));
new Gson().toJson(new KruizeResponse(origMessage, errorCode, "", "ERROR", myList)));
out.flush();
}
}
15 changes: 15 additions & 0 deletions src/main/java/com/autotune/analyzer/exceptions/KruizeResponse.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,19 +15,34 @@
*******************************************************************************/
package com.autotune.analyzer.exceptions;

import com.autotune.analyzer.serviceObjects.UpdateResultsAPIObject;

import java.util.List;

public class KruizeResponse {
private String message;
private int httpcode;
private String documentationLink;
private String status;

private List<UpdateResultsAPIObject> data;


public KruizeResponse(String message, int httpcode, String documentationLink, String status) {
this.message = message;
this.httpcode = httpcode;
this.documentationLink = documentationLink;
this.status = status;
}

public KruizeResponse(String message, int httpcode, String documentationLink, String status, List<UpdateResultsAPIObject> data) {
this.message = message;
this.httpcode = httpcode;
this.documentationLink = documentationLink;
this.status = status;
this.data = data;
}

public String getMessage() {
return message;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,28 @@
package com.autotune.analyzer.experiment;

import com.autotune.analyzer.kruizeObject.KruizeObject;
import com.autotune.analyzer.performanceProfiles.PerformanceProfile;
import com.autotune.analyzer.performanceProfiles.PerformanceProfileInterface.PerfProfileInterface;
import com.autotune.analyzer.serviceObjects.Converters;
import com.autotune.analyzer.serviceObjects.UpdateResultsAPIObject;
import com.autotune.analyzer.utils.AnalyzerConstants;
import com.autotune.common.data.ValidationOutputData;
import com.autotune.common.data.result.ExperimentResultData;
import com.autotune.database.service.ExperimentDBService;
import com.google.gson.annotations.SerializedName;
import jakarta.validation.ConstraintViolation;
import jakarta.validation.Validation;
import jakarta.validation.Validator;
import org.hibernate.validator.HibernateValidator;
import org.hibernate.validator.messageinterpolation.ParameterMessageInterpolator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import javax.servlet.http.HttpServletResponse;
import java.lang.reflect.Field;
import java.sql.Timestamp;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;

/**
* Initiates new experiment data validations and push into queue for worker to
Expand All @@ -36,9 +46,10 @@
public class ExperimentInitiator {
private static final long serialVersionUID = 1L;
private static final Logger LOGGER = LoggerFactory.getLogger(ExperimentInitiator.class);
List<UpdateResultsAPIObject> successUpdateResultsAPIObjects = new ArrayList<>();
List<UpdateResultsAPIObject> failedUpdateResultsAPIObjects = new ArrayList<>();
private ValidationOutputData validationOutputData;


/**
* Initiate Experiment validation
*
Expand Down Expand Up @@ -68,36 +79,6 @@ public void validateAndAddNewExperiments(
}
}

/**
* @param mainKruizeExperimentMap
* @param experimentResultDataList
* @param performanceProfilesMap
*/
public ValidationOutputData validateAndUpdateResults(
Map<String, KruizeObject> mainKruizeExperimentMap,
List<ExperimentResultData> experimentResultDataList,
Map<String, PerformanceProfile> performanceProfilesMap) {
ValidationOutputData validationOutputData = new ValidationOutputData(false, null, null);
try {
ExperimentResultValidation experimentResultValidation = new ExperimentResultValidation(mainKruizeExperimentMap, performanceProfilesMap);
experimentResultValidation.validate(experimentResultDataList, performanceProfilesMap);
if (experimentResultValidation.isSuccess()) {
ExperimentInterface experimentInterface = new ExperimentInterfaceImpl();
experimentInterface.addResultsToLocalStorage(mainKruizeExperimentMap, experimentResultDataList);
validationOutputData.setSuccess(true);
} else {
validationOutputData.setSuccess(false);
validationOutputData.setMessage("Validation failed: " + experimentResultValidation.getErrorMessage());
}
} catch (Exception e) {
LOGGER.error("Validate and push experiment falied: " + e.getMessage());
validationOutputData.setSuccess(false);
validationOutputData.setMessage("Exception occurred while validating the result data: " + e.getMessage());
validationOutputData.setErrorCode(HttpServletResponse.SC_INTERNAL_SERVER_ERROR);
}
return validationOutputData;
}

// Generate recommendations and add it to the kruize object
public void generateAndAddRecommendations(KruizeObject kruizeObject, List<ExperimentResultData> experimentResultDataList, Timestamp interval_start_time, Timestamp interval_end_time) throws Exception {
if (AnalyzerConstants.PerformanceProfileConstants.perfProfileInstances.containsKey(kruizeObject.getPerformanceProfile())) {
Expand All @@ -113,4 +94,71 @@ public void generateAndAddRecommendations(KruizeObject kruizeObject, List<Experi
}


public void validateAndAddExperimentResults(List<UpdateResultsAPIObject> updateResultsAPIObjects) {
List<UpdateResultsAPIObject> failedDBObjects = new ArrayList<>();
Validator validator = Validation.byProvider(HibernateValidator.class)
.configure()
.messageInterpolator(new ParameterMessageInterpolator())
.failFast(false)
.buildValidatorFactory()
.getValidator();

for (UpdateResultsAPIObject object : updateResultsAPIObjects) {
Set<ConstraintViolation<UpdateResultsAPIObject>> violations = validator.validate(object, UpdateResultsAPIObject.FullValidationSequence.class);
if (violations.isEmpty()) {
successUpdateResultsAPIObjects.add(object);
} else {
List<String> errorReasons = new ArrayList<>();
for (ConstraintViolation<UpdateResultsAPIObject> violation : violations) {
String propertyPath = violation.getPropertyPath().toString();
if (null != propertyPath && propertyPath.length() != 0) {
errorReasons.add(getSerializedName(propertyPath, UpdateResultsAPIObject.class) + ": " + violation.getMessage());
} else {
errorReasons.add(violation.getMessage());
}
}
object.setErrorReasons(errorReasons);
failedUpdateResultsAPIObjects.add(object);
}
}
List<ExperimentResultData> resultDataList = new ArrayList<>();
successUpdateResultsAPIObjects.forEach(
(successObj) -> {
resultDataList.add(Converters.KruizeObjectConverters.convertUpdateResultsAPIObjToExperimentResultData(successObj));
}
);

if (successUpdateResultsAPIObjects.size() > 0) {
failedDBObjects = new ExperimentDBService().addResultsToDB(resultDataList);
failedUpdateResultsAPIObjects.addAll(failedDBObjects);
}
}

public String getSerializedName(String fieldName, Class<?> targetClass) {
Class<?> currentClass = targetClass;
while (currentClass != null) {
try {
Field field = currentClass.getDeclaredField(fieldName);
SerializedName annotation = field.getAnnotation(SerializedName.class);
if (annotation != null) {
fieldName = annotation.value();
}
} catch (NoSuchFieldException e) {
// Field not found in the current class
// Move up to the superclass
currentClass = currentClass.getSuperclass();
}
}
return fieldName;
}

public List<UpdateResultsAPIObject> getSuccessUpdateResultsAPIObjects() {
return successUpdateResultsAPIObjects;
}

public List<UpdateResultsAPIObject> getFailedUpdateResultsAPIObjects() {
return failedUpdateResultsAPIObjects;
}


}
Loading
Loading