Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add validations from appsec #562

Merged
merged 10 commits into from
Jan 31, 2024
Merged

Add validations from appsec #562

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
ed8e581
add validations from appsec
HenryL27 Jan 26, 2024
9a6fe41
update changelog
HenryL27 Jan 26, 2024
66d2efd
add validation on number of characters in query context path (1024)
HenryL27 Jan 29, 2024
a8c4c9e
drop max strlen to 1000
HenryL27 Jan 29, 2024
e53c0a1
separate query validation errors; also log them
HenryL27 Jan 29, 2024
4b17c4a
remove a superfluous return lol
HenryL27 Jan 29, 2024
838755f
Apply suggestions from code review
HenryL27 Jan 30, 2024
c03d0b0
use clusterService to get settings rather than environment
HenryL27 Jan 30, 2024
dba691d
import sneakythrows
HenryL27 Jan 30, 2024
75adcac
apply spotless and fix embarassing syntax error
HenryL27 Jan 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .../java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@
public static final String QUERY_TEXT_FIELD = "query_text";
public static final String QUERY_TEXT_PATH_FIELD = "query_text_path";

public static final Integer MAX_QUERY_PATH_STRLEN = 1024;

private final Environment environment;

@Override
Expand Down Expand Up @@ -76,9 +78,10 @@
throw new IllegalArgumentException(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d",
"%s exceeded the maximum path length of %d nested fields or %d characters",
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings()),
MAX_QUERY_PATH_STRLEN
)
);
}
Expand Down Expand Up @@ -127,8 +130,11 @@

private boolean validatePath(final String path) {
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
if (path == null || path.isEmpty()) {
return true;

Check warning on line 133 in src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/neuralsearch/processor/rerank/context/QueryContextSourceFetcher.java#L133 

Added line #L133 was not covered by tests
}
if (path.length() > MAX_QUERY_PATH_STRLEN) {
return false;
}
return path.split("\\.").length <= MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings());
}
}
30 changes: 27 additions & 3 deletions ...t/java/org/opensearch/neuralsearch/processor/rerank/MLOpenSearchRerankProcessorTests.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,30 @@ public void testRerankContext_whenQueryTextPathIsBadPointer_thenFail() throws IO
.equals(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD + " must point to a string field"));
}

public void testRerankContext_whenQueryTextPathIsExceeedinglyLong_thenFail() throws IOException {
public void testRerankContext_whenQueryTextPathIsExceeedinglyManyCharacters_thenFail() throws IOException {
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
// "eighteencharacters" * 60 = 1080 character string > max len of 1024
setupParams(Map.of(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD, "eighteencharacters".repeat(60)));
setupSearchResults();
@SuppressWarnings("unchecked")
ActionListener<Map<String, Object>> listener = mock(ActionListener.class);
processor.generateRerankingContext(request, response, listener);
ArgumentCaptor<Exception> argCaptor = ArgumentCaptor.forClass(Exception.class);
verify(listener, times(1)).onFailure(argCaptor.capture());
assert (argCaptor.getValue() instanceof IllegalArgumentException);
assert (argCaptor.getValue()
.getMessage()
.equals(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d nested fields or %d characters",
QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings()),
QueryContextSourceFetcher.MAX_QUERY_PATH_STRLEN
)
));
}

public void textRerankContext_whenQueryTextPathIsExceeedinglyDeeplyNested_ThenFail() throws IOException {
HenryL27 marked this conversation as resolved.
Show resolved Hide resolved
setupParams(Map.of(QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD, "a.b.c.d.e.f.g.h.i.j.k.l.m.n.o.p.q.r.s.t.w.x.y.z"));
setupSearchResults();
@SuppressWarnings("unchecked")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this annotation? I don't see ActionListener<Map<String, Object>> listener = mock(ActionListener.class); is causing any warning in my local.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

¯_(ツ)_/¯. I get a warning when I remove it. Makes sense since it's a typecast from generic action listener to action listener for that map thingy.

Expand All @@ -244,9 +267,10 @@ public void testRerankContext_whenQueryTextPathIsExceeedinglyLong_thenFail() thr
.equals(
String.format(
Locale.ROOT,
"%s exceeded the maximum path length of %d",
"%s exceeded the maximum path length of %d nested fields or %d characters",
QueryContextSourceFetcher.QUERY_TEXT_PATH_FIELD,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings())
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING.get(environment.settings()),
QueryContextSourceFetcher.MAX_QUERY_PATH_STRLEN
)
));
}
Expand Down
Loading