#992: added design doc for script options #477

tomuben · 2024-11-15T10:58:57Z

related to exasol/script-languages-release#992

tkilias · 2024-11-17T23:54:50Z

exaudfclient/docs/diagrams/CTPGParserHandler.png

How did you create these? If it is drawio, please use the extension drawio.png otherwise it is not clear how to change them and the tooling won't recognize them.

tkilias · 2024-11-18T00:01:03Z

exaudfclient/docs/script_options_design.md

+
+#### Legacy Parser
+
+The legacy parser (V1) parser searches for one specific script option. The parser starts from beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the 


In a narrow interpretation of arc42, the removal would be part of the runtime view and not the building blocks.

tkilias · 2024-11-18T00:15:57Z

exaudfclient/docs/script_options_design.md

+
+Tags: V2
+
+### General Parser Integration


Same for the DB

tkilias · 2024-12-04T23:38:43Z

.github/workflows/scripts/run_oft.sh

@@ -0,0 +1,27 @@
+#!/bin/bash


We should at some Point replace this with a nox session, please create an issue

tkilias · 2024-12-04T23:40:14Z

exaudfclient/docs/script_options_design.md

+## Constraints
+
+- The parser implementation must be in C++.
+- The chosen parser implementation is [ctpg](https://github.com/peter-winter/ctpg), which supports the definition of Lexer and Parser Rules in C++ code.


Not a constraint, but a decision that follows from the constraints

tkilias · 2024-12-04T23:41:34Z

exaudfclient/docs/script_options_design.md

+
+![Components](diagrams/OveralScriptOptionalsBuildingBlocks.drawio.png)
+
+At the very high level there can be distinguished between the generic "Script Options Parser" module which parses a UDF script code and returns the found script options, and the "Script Options Parser Handler" which converts the Java UDF specific script options. In both modules there are specific implementation for the legacy parser and the new CTPG based parser.


Add reason, why we need to implementation

tkilias · 2024-12-04T23:42:39Z

exaudfclient/docs/script_options_design.md

+
+### Script Options Parser
+
+The parser component can be used to parse any script code (Java, Python, R) for any script options. It provides simplistic interfaces, which are different between the two versions, which accept the script code as input and return the found script option(s). 


Add reason, why the interfaces are different. Here, they work inherently different

tkilias · 2024-12-04T23:44:10Z

exaudfclient/docs/script_options_design.md

+
+As the parser needs to find script options in any given script code, the generated parser must accept any strings which are not script options and ignore those. In order to achieve this, the lexer rules need to be as simple as possible, in order to avoid collisions.
+
+It is important to emphasize that in contrast to the legacy parser, the caller is responsible for removing the script options from the script code.


Do we have a good reason?

tkilias · 2024-12-04T23:46:08Z

exaudfclient/docs/script_options_design.md

+
+![LegacyParserHandler](diagrams/LegacyParserHandler.drawio.png)
+
+The `ScriptOptionsLinesParserLegacy` class uses the Parser to search for Java specific script options and forwards the found options to class `ConverterLegacy`, which uses a common implementation for the conversion of the options.


Suggested change

The `ScriptOptionsLinesParserLegacy` class uses the Parser to search for Java specific script options and forwards the found options to class `ConverterLegacy`, which uses a common implementation for the conversion of the options.

The `ScriptOptionsLinesParserLegacy` class uses the Parser to search for Java specific script options and forwards the found options to the class `ConverterLegacy`, which uses a common implementation for the conversion of the options.

common implementation is misleading, because it suggests both handler use it

tkilias · 2024-12-04T23:48:38Z

exaudfclient/docs/script_options_design.md

+
+![CTPGParserHandler](diagrams/CTPGParserHandler.drawio.png)
+
+The `ScriptOptionsLinesParserCTPG` class uses the new CTPG basedParser to search for **all** Java specific script options at once. Then it forwards the found options to class `ConverterV2`, which uses a common implementation for the conversion of the options. `ConverterV2` also implements the functions to convert Jvm otions and JAR options.   


Suggested change

The `ScriptOptionsLinesParserCTPG` class uses the new CTPG basedParser to search for **all** Java specific script options at once. Then it forwards the found options to class `ConverterV2`, which uses a common implementation for the conversion of the options. `ConverterV2` also implements the functions to convert Jvm otions and JAR options.

The `ScriptOptionsLinesParserCTPG` class uses the new CTPG based Parser to search for **all** Java specific script options at once. Then it forwards the found options to class `ConverterV2`, which uses a common implementation for the conversion of the options. `ConverterV2` also implements the functions to convert Jvm otions and JAR options.

common again

tkilias · 2024-12-04T23:50:42Z

exaudfclient/docs/script_options_design.md

+`ScriptOptionsLinesParserCTPG` uses an instance of `ScriptImporter` to import foreign scripts. Because the new parser collects all script options at once, but backwards compatibility with existing UDF scripts must be ensured, there is an additional level of complexity in the import script algorithm. The algorithm is described in the following pseudocode snippet:
+```
+function import(script_code, options_map)
+ import_option = options_map.find("import")


Suggested change

import_option = options_map.find("import")

import_options = options_map.find("import")

Should this be plural?

tkilias · 2024-12-04T23:53:04Z

exaudfclient/docs/script_options_design.md

+    static void doSomething() {}
+}
+```
+


Maybe add the expected results

tkilias · 2024-12-04T23:53:49Z

exaudfclient/docs/script_options_design.md

+
+### Parser Implementation V1
+
+The legacy parser (V1) parser searches for one specific script option. The parser starts from beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the 


Suggested change

The legacy parser (V1) parser searches for one specific script option. The parser starts from beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the

The legacy parser (V1) parser searches for one specific script option. The parser starts from the beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the

The rest of the sentence is missing

tkilias · 2024-12-04T23:56:47Z

exaudfclient/docs/script_options_design.md

+### Lexer and Parser Rules Option
+`dsn~lexer-parser-rules~1`
+
+Lexer and Parser rules to recognize `%optionKey`, `optionValue`, with whitespace characters as separator. The Parser rules will define the grammar to correctly identify Script Options, manage multiple options with the same key, and handle duplicates.


Mention simple lexer rules to deal with whitespace and escaping correct

tkilias · 2024-12-04T23:57:37Z

exaudfclient/docs/script_options_design.md

+### Ignore lines without script options
+`dsn~ignore-lines-without-script-options~1`
+
+In order to avoid lower performance compared to the old implementation, the parser must run only on lines which contain a `%` character.


Suggested change

In order to avoid lower performance compared to the old implementation, the parser must run only on lines which contain a `%` character.

In order to avoid lower performance compared to the old implementation, the parser must run only on lines which contain a `%` character after only whitespaces.

tkilias · 2024-12-04T23:58:57Z

exaudfclient/docs/script_options_design.md

+
+Tags: V2
+
+### Lexer and Parser Rules Whitespace Sequences


Need to continue here

tomuben force-pushed the doc/992_add_design_doc_for_script_options branch from 3522853 to 0a2684d Compare November 15, 2024 14:59

tkilias reviewed Nov 17, 2024

View reviewed changes

tkilias reviewed Nov 18, 2024

View reviewed changes

exaudfclient/docs/script_options_design.md

Tags: V2

### General Parser Integration

Copy link

Collaborator

tkilias Nov 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for the DB

tomuben added 17 commits November 22, 2024 12:31

#991: Added Design Doc and Requirements Doc for Script Options parser

d2c2f6e

Updated design document

9385004

Updated design document

6c4c049

Updated design document

b9a7904

Updated design document

2b1e472

Updated design document & requirements document

9671883

Added a GH workflow to run

93dca60

Fixed oft.yaml

9e35421

Fixed oft.yaml

3911606

Fixed oft.yaml

d0fee61

Fixed oft.yaml

a2fcff1

Fixed oft.yaml

7910608

Fixed oft.yaml

24caddc

Removed Needs: req in script_options_design.md

14425fa

Fixed findings from OFT

6800239

Fixed findings from OFT

1f08f8d

Fixed findings from review

207e27f

tomuben force-pushed the doc/992_add_design_doc_for_script_options branch from b6dca79 to 207e27f Compare November 22, 2024 16:15

Merge branch 'master' into doc/992_add_design_doc_for_script_options

af0d8ee

tkilias reviewed Dec 4, 2024

View reviewed changes

exaudfclient/docs/script_options_design.md

static void doSomething() {}

}

```

Copy link

Collaborator

tkilias Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add the expected results

tkilias reviewed Dec 4, 2024

View reviewed changes

exaudfclient/docs/script_options_design.md

Tags: V2

### Lexer and Parser Rules Whitespace Sequences

Copy link

Collaborator

tkilias Dec 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to continue here

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#992: added design doc for script options #477

#992: added design doc for script options #477

tomuben commented Nov 15, 2024

tkilias Nov 17, 2024

tkilias Nov 18, 2024 •

edited

Loading

tkilias Nov 18, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024


		#### Legacy Parser

		The legacy parser (V1) parser searches for one specific script option. The parser starts from beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the


		![Components](diagrams/OveralScriptOptionalsBuildingBlocks.drawio.png)

		At the very high level there can be distinguished between the generic "Script Options Parser" module which parses a UDF script code and returns the found script options, and the "Script Options Parser Handler" which converts the Java UDF specific script options. In both modules there are specific implementation for the legacy parser and the new CTPG based parser.


		### Script Options Parser

		The parser component can be used to parse any script code (Java, Python, R) for any script options. It provides simplistic interfaces, which are different between the two versions, which accept the script code as input and return the found script option(s).


		As the parser needs to find script options in any given script code, the generated parser must accept any strings which are not script options and ignore those. In order to achieve this, the lexer rules need to be as simple as possible, in order to avoid collisions.

		It is important to emphasize that in contrast to the legacy parser, the caller is responsible for removing the script options from the script code.


		![LegacyParserHandler](diagrams/LegacyParserHandler.drawio.png)

		The `ScriptOptionsLinesParserLegacy` class uses the Parser to search for Java specific script options and forwards the found options to class `ConverterLegacy`, which uses a common implementation for the conversion of the options.


		![CTPGParserHandler](diagrams/CTPGParserHandler.drawio.png)

		The `ScriptOptionsLinesParserCTPG` class uses the new CTPG basedParser to search for all Java specific script options at once. Then it forwards the found options to class `ConverterV2`, which uses a common implementation for the conversion of the options. `ConverterV2` also implements the functions to convert Jvm otions and JAR options.

	import_option = options_map.find("import")
	import_options = options_map.find("import")


		### Parser Implementation V1

		The legacy parser (V1) parser searches for one specific script option. The parser starts from beginning of the script code. If found, the parser immediately removes the script option from the script code and returns the option value. It validates the

	In order to avoid lower performance compared to the old implementation, the parser must run only on lines which contain a `%` character.
	In order to avoid lower performance compared to the old implementation, the parser must run only on lines which contain a `%` character after only whitespaces.

#992: added design doc for script options #477

Are you sure you want to change the base?

#992: added design doc for script options #477

Conversation

tomuben commented Nov 15, 2024

Choose a reason for hiding this comment

tkilias Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkilias Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

tkilias Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

tkilias Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkilias Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkilias Nov 18, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading

tkilias Dec 4, 2024 •

edited

Loading