accomodated review comments from JHutar #106

sarathbrp · 2024-01-11T14:10:26Z

No description provided.

added verification false for https

jhutar · 2024-01-12T07:42:18Z

opl/investigator/config.py

@@ -75,6 +75,10 @@ def load_config(conf, fp):
        assert not conf.history_es_server.endswith("/")
        conf.history_es_index = data["history"]["es_index"]
        conf.history_es_query = data["history"]["es_query"]
+        if "es_server_user" in data["history"]:
+            conf.es_server_user = data["history"]["es_server_user"]
+            conf.es_server_pass = data["history"]["es_server_pass"]


Suggested change

conf.es_server_pass = data["history"]["es_server_pass"]

conf.es_server_pass_env_var = data["history"]["es_server_pass_env_var"]

jhutar · 2024-01-12T07:42:32Z

opl/investigator/config.py

@@ -88,6 +92,10 @@ def load_config(conf, fp):
        conf.decisions_es_server = data["decisions"]["es_server"]
        assert not conf.decisions_es_server.endswith("/")
        conf.decisions_es_index = data["decisions"]["es_index"]
+        if "es_server_user" in data["decisions"]:
+            conf.decisions_es_server_user = data["decisions"]["es_server_user"]
+            conf.decisions_es_server_pass = data["decisions"]["es_server_pass"]


Again, just env var

jhutar · 2024-01-12T07:49:00Z

opl/http.py

@@ -12,6 +12,7 @@ def disable_insecure_request_warnings(disable_it):
    if disable_it:
        logging.debug("Disabling insecure request warnings")
        urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+        session.verify = False


I would not put it even here. I would leave it on user to explicitly do http.session.verify = False or so. I admit this function name in unfortunate. Also this http module should have been a class or so, this sucks.

Or maybe just create function like this:

def insecure(): session.verify = False disable_insecure_request_warnings(True)

jhutar · 2024-01-12T08:20:50Z

opl/investigator/elasticsearch_loader.py

+    else:
+        response = opl.http.get(url, headers=headers, json=data)
+
+    if response:


Why is this if response: required?

jhutar · 2024-01-12T08:21:26Z

opl/pass_or_fail.py

@@ -114,12 +114,18 @@ def main():
    if args.history_type == "csv":
        history = opl.investigator.csv_loader.load(args.history_file, args.sets)
    elif args.history_type == "elasticsearch":
+        if hasattr(args, "es_server_user"):


Suggested change

if hasattr(args, "es_server_user"):

if hasattr(args, "es_server_verify"):

jhutar · 2024-01-12T08:22:50Z

opl/pass_or_fail.py

@@ -200,8 +206,17 @@ def main():
    if not args.dry_run:
        for d_type in args.decisions_type:
            if d_type == "elasticsearch":
+                if hasattr(args, "es_server_user"):


Suggested change

if hasattr(args, "es_server_user"):

if hasattr(args, "es_server_verify"):

jhutar · 2024-01-12T13:22:36Z

opl/http.py

@@ -8,6 +8,12 @@
 session = requests.Session()


+def insecure():
+    session.verify = False
+    logging.debug("Disabling insecure request warnings")


This is not needed, it will be printed in that disable_insecure_request_warnings. Or maybe you meant:

Suggested change

logging.debug("Disabling insecure request warnings")

logging.debug("Disabling SSL verifications for this session")

jhutar · 2024-01-12T13:25:52Z

opl/pass_or_fail.py

        history = opl.investigator.elasticsearch_loader.load(
            args.history_es_server,
            args.history_es_index,
            args.history_es_query,
            args.sets,
+            es_server_user=getattr(args, "es_server_user", None),
+            es_server_pass=getattr(args, "es_server_pass_env_var", None),


Pls rename es_server_pass as well to be consistent.

jhutar · 2024-01-12T13:27:00Z

opl/pass_or_fail.py

+                    args.decisions_es_index,
+                    info_all,
+                    es_server_user=getattr(args, "decisions_es_server_user", None),
+                    es_server_pass=getattr(args, "decisions_es_server_pass_env_var", None),


Same here. Pls either use es_server_pass_env_var or call os.environ.get(...) here.

jhutar · 2024-01-15T07:54:39Z

opl/pass_or_fail.py

@@ -114,12 +114,18 @@ def main():
    if args.history_type == "csv":
        history = opl.investigator.csv_loader.load(args.history_file, args.sets)
    elif args.history_type == "elasticsearch":
+        if hasattr(args, "es_server_verify"):


I assume you need this, or am I missing something?

Suggested change

if hasattr(args, "es_server_verify"):

if hasattr(args, "es_server_verify") and args.es_server_verify == False:

jhutar · 2024-01-15T07:54:57Z

opl/pass_or_fail.py

@@ -200,8 +206,17 @@ def main():
    if not args.dry_run:
        for d_type in args.decisions_type:
            if d_type == "elasticsearch":
+                if hasattr(args, "es_server_verify"):


Same as above.

jhutar · 2024-01-15T07:57:01Z

opl/pass_or_fail.py

+                    args.decisions_es_index,
+                    info_all,
+                    es_server_user=getattr(args, "decisions_es_server_user", None),
+                    decisions_es_server_pass_env_var=getattr(


Tihs is not a requirement, but to align with es_server_user I would rename decisions_es_server_pass_env_var to es_server_pass_env_var. Or rename es_server_user to decisions_es_server_user

jhutar · 2024-01-15T08:03:20Z

opl/investigator/config.py

+        if "es_server_user" in data["history"]:
+            conf.es_server_user = data["history"]["es_server_user"]
+            conf.es_server_pass_env_var = data["history"]["es_server_pass_env_var"]
+            conf.es_server_verify = data["history"]["es_server_verify"]


Pls make this option optional, defaulting to True. Also this option is valid for setups without auth, so should be outside of this if.

If I have removed If condition then most of services fail at this line because investigator.conf file is not updated with username, password and verify flag. The plan is to start with front-end service and slowly propagate changes to other services, once it is complete then If condition is not longer required.

I will keep it for now until all investigator.conf files are updated across the services.

opl/investigator/config.py

jhutar · 2024-01-15T12:52:31Z

Pls also add these new fields (commented out) to the sample config, maybe with some explanation.

jhutar · 2024-01-15T13:55:55Z

opl/investigator/config.py

@@ -75,6 +75,9 @@ def load_config(conf, fp):
        assert not conf.history_es_server.endswith("/")
        conf.history_es_index = data["history"]["es_index"]
        conf.history_es_query = data["history"]["es_query"]
+        conf.history_es_server_user = data["history"]["es_server_user"]


Hello! conf.history_es_server_user and pass needs to stay in that if because they will not be configured every time. Just conf.history_es_server_verify should be excluded from it because it can be used for unauthenticated ES.

Maybe something like this:

if conf.history_type == "elasticsearch": conf.history_es_server = data["history"]["es_server"] assert not conf.history_es_server.endswith("/") conf.history_es_index = data["history"]["es_index"] conf.history_es_query = data["history"]["es_query"] if "es_server_user" in data["history"]: conf.history_es_server_user = data["history"]["es_server_user"] conf.history_es_server_pass_env_var = data["history"]["es_server_pass_env_var"] if "es_server_verify" in data["history"]: conf.history_es_server_verify = data["history"]["es_server_verify"] else: conf.history_es_server_verify = True

jhutar · 2024-01-15T13:56:51Z

opl/investigator/config.py

@@ -75,6 +75,9 @@ def load_config(conf, fp):
        assert not conf.history_es_server.endswith("/")
        conf.history_es_index = data["history"]["es_index"]
        conf.history_es_query = data["history"]["es_query"]
+        conf.history_es_server_user = data["history"]["es_server_user"]
+        conf.history_es_server_pass_env_var = data["history"]["es_server_pass_env_var"]
+        conf.history_es_server_verify = data["history"]["es_server_verify"]


Make it so that if es_server_verify is not configured in the config, conf.history_es_server_verify is set to true.

jhutar · 2024-01-15T13:58:38Z

opl/investigator/config.py

+        conf.decisions_es_server_pass_env_var = data["decisions"][
+            "decisions_es_server_pass_env_var"
+        ]
+        conf.decisions_es_server_verify = data["decisions"]["es_server_verify"]


jhutar · 2024-01-15T14:00:42Z

opl/pass_or_fail.py

+                    decisions_es_server_user=getattr(args, "decisions_es_server_user", None),
+                    es_server_pass_env_var=getattr(


Either call both with decisions_* or without.

Probably without, as you are doing that bit above in call to history = opl.investigator.elasticsearch_loader.load(...)

jhutar · 2024-01-15T14:02:59Z

Pls also fix linter complaints and add commented example to https://github.com/redhat-performance/opl/blob/main/opl/investigator/sample_config.yaml

jhutar

Thank you.

jhutar · 2024-01-16T08:28:15Z

core/opl/shovel.py

@@ -1,22 +1,42 @@
 import argparse


Hello. This file should not be in this PR. Pls remove it.

sarathbrp added 6 commits December 18, 2023 07:00

added opensearch keys to a secret

e63d975

fixing lint errors

cf5069d

fixing black formatter issues

2739348

added new elastic search authentication

f6d34e4

Update http.py

f0fa934

added verification false for https

fixing elastic search update using investigator conf

b5dea9d

sarathbrp requested a review from jhutar January 11, 2024 14:10

sarathbrp self-assigned this Jan 11, 2024

jhutar requested changes Jan 12, 2024

View reviewed changes

updated after JHutar comments

c24965d

sarathbrp closed this Jan 12, 2024

sarathbrp reopened this Jan 12, 2024

Merge branch 'main' into sp-perf-opl

9b5f208

jhutar requested changes Jan 12, 2024

View reviewed changes

Part3 comments updated

bc212ec

jhutar reviewed Jan 15, 2024

View reviewed changes

opl/investigator/config.py Show resolved Hide resolved

jhutar reviewed Jan 15, 2024

View reviewed changes

opl/investigator/config.py Show resolved Hide resolved

sarathbrp added 2 commits January 15, 2024 07:34

Merge branch 'main' into sp-perf-opl

a0358ca

updated files with more comments

e0f20be

jhutar reviewed Jan 15, 2024

View reviewed changes

opl/investigator/config.py Show resolved Hide resolved

updated another round of comments

65dcb1e

jhutar reviewed Jan 15, 2024

View reviewed changes

updated files after comments

830882a

jhutar approved these changes Jan 15, 2024

View reviewed changes

shubham-html-css-js and others added 7 commits January 15, 2024 17:26

Convert prow-to-es functions to python

163206a

Applied black

e254257

Fixc python flake errors

0e34499

Review comment changes

ae68787

Removed unnecessary module

29355dc

fix pylint errors

e67aa08

fixing lint error

f8dd214

jhutar reviewed Jan 16, 2024

View reviewed changes

core/opl/shovel.py Outdated

@@ -1,22 +1,42 @@

import argparse

Copy link

Contributor

jhutar Jan 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello. This file should not be in this PR. Pls remove it.

Delete core/opl/shovel.py

610dfe4

sarathbrp closed this Jan 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

accomodated review comments from JHutar #106

accomodated review comments from JHutar #106

sarathbrp commented Jan 11, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 12, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

sarathbrp Jan 15, 2024

sarathbrp Jan 15, 2024

jhutar commented Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar Jan 15, 2024

jhutar commented Jan 15, 2024

jhutar left a comment

jhutar Jan 16, 2024

	conf.es_server_pass = data["history"]["es_server_pass"]
	conf.es_server_pass_env_var = data["history"]["es_server_pass_env_var"]

	if hasattr(args, "es_server_user"):
	if hasattr(args, "es_server_verify"):

	logging.debug("Disabling insecure request warnings")
	logging.debug("Disabling SSL verifications for this session")

	if hasattr(args, "es_server_verify"):
	if hasattr(args, "es_server_verify") and args.es_server_verify == False:

		decisions_es_server_user=getattr(args, "decisions_es_server_user", None),
		es_server_pass_env_var=getattr(

accomodated review comments from JHutar #106

accomodated review comments from JHutar #106

Conversation

sarathbrp commented Jan 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhutar commented Jan 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhutar commented Jan 15, 2024

jhutar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment