Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature](stream-load)(config) implement streamload via http stream #43536

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

[Feature](stream-load)(config) implement streamload via http stream #43536

wants to merge 7 commits into from
+601 −10

Conversation

Yufei-YAO
Copy link

What problem does this PR solve?

Issue Number: #36773

Problem Summary:

Currently we can put data to doris via http using streamload or http stream. httpstream transforms data via sql in a header while streamload via special header. Sql is more easier to use.

In future, we would suggest http stream for users, and we also need keep only one in internal implementation. So we can add a config option to enable implement streamload via ways of httpstream.

We can transform streamload headers to a sql internally and implement streamload in ways of httstream.

Problem Solution:

  1. Add a new BE config enable_streamload_by_httpstream
  2. Perform streamload via httpstream if enable_streamload_by_httpstream=true
  3. Manual test the implementation by previous regression test streamLoad_action

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

github-actions[bot]
github-actions bot reviewed Nov 8, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/stream_load.cpp Outdated
return _exec_env->stream_load_executor()->execute_plan_fragment(ctx);
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' has cognitive complexity of 102 (threshold 50) [readability-function-cognitive-complexity]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1205: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1205: +1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
                                           ^

be/src/http/action/stream_load.cpp:1207: +2, including nesting penalty of 1, nesting level increased to 2

        if (enclose_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1212: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1212: +1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
                                          ^

be/src/http/action/stream_load.cpp:1214: +2, including nesting penalty of 1, nesting level increased to 2

        if (escape_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1219: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1220: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1225: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1226: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1232: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_SEND_BATCH_PARALLELISM).empty()) {
    ^

be/src/http/action/stream_load.cpp:1235: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::invalid_argument& e) {
          ^

be/src/http/action/stream_load.cpp:1238: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::out_of_range& e) {
          ^

be/src/http/action/stream_load.cpp:1247: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_MERGE_TYPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1250: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != merge_type_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1252: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1255: +2, including nesting penalty of 1, nesting level increased to 2

        if (merge_type == TMergeType::MERGE && req->header(HTTP_DELETE_CONDITION).empty()) {
        ^

be/src/http/action/stream_load.cpp:1257: +1, nesting level increased to 2

        } else if (merge_type != TMergeType::MERGE && !req->header(HTTP_DELETE_CONDITION).empty()) {
               ^

be/src/http/action/stream_load.cpp:1263: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_UNIQUE_KEY_UPDATE_MODE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1270: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != unique_key_update_mode_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1272: +3, including nesting penalty of 2, nesting level increased to 3

            if (unique_key_update_mode == TUniqueKeyUpdateMode::UPDATE_FLEXIBLE_COLUMNS) {
            ^

be/src/http/action/stream_load.cpp:1274: +4, including nesting penalty of 3, nesting level increased to 4

                if (ctx->format != TFileFormatType::FORMAT_JSON) {
                ^

be/src/http/action/stream_load.cpp:1279: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                ^

be/src/http/action/stream_load.cpp:1279: +1

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                                                           ^

be/src/http/action/stream_load.cpp:1284: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1288: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_JSONPATHS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1292: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_HIDDEN_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1297: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL).empty()) {
                ^

be/src/http/action/stream_load.cpp:1302: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_MERGE_TYPE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1307: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_WHERE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1313: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1321: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_FORMAT_KEY).empty()) {
    ^

be/src/http/action/stream_load.cpp:1323: +1, nesting level increased to 1

    } else {
      ^

be/src/http/action/stream_load.cpp:1332: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1333: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1334: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1335: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1336: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1337: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1338: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1339: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1340: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1341: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1342: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1343: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1344: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1345: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1346: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1347: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1348: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1349: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1350: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1351: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1352: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1353: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1354: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1355: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1356: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1357: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1358: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1359: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1360: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1361: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1362: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1363: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1364: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1328: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1369: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_WHERE).empty()) {
    ^

be/src/http/action/stream_load.cpp Outdated
return _exec_env->stream_load_executor()->execute_plan_fragment(ctx);
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' exceeds recommended size/complexity thresholds [readability-function-size]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1188: 185 lines including whitespace and comments (threshold 80)

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^

@Yufei-YAO Yufei-YAO changed the title [feature](stream-load)(config) implement streamload via http stream (#36773) [feature](stream-load)(config) implement streamload via http stream Nov 8, 2024
@Yufei-YAO Yufei-YAO changed the title [feature](stream-load)(config) implement streamload via http stream [Enhancement](stream-load)(config) implement streamload via http stream Nov 8, 2024
@Yufei-YAO Yufei-YAO changed the title [Enhancement](stream-load)(config) implement streamload via http stream [Feature](stream-load)(config) implement streamload via http stream Nov 8, 2024
github-actions[bot]
github-actions bot reviewed Nov 8, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/stream_load.cpp Outdated
return ss.str();
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' has cognitive complexity of 102 (threshold 50) [readability-function-cognitive-complexity]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1246: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1246: +1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
                                           ^

be/src/http/action/stream_load.cpp:1248: +2, including nesting penalty of 1, nesting level increased to 2

        if (enclose_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1253: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1253: +1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
                                          ^

be/src/http/action/stream_load.cpp:1255: +2, including nesting penalty of 1, nesting level increased to 2

        if (escape_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1260: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1261: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1266: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1267: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1273: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_SEND_BATCH_PARALLELISM).empty()) {
    ^

be/src/http/action/stream_load.cpp:1276: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::invalid_argument& e) {
          ^

be/src/http/action/stream_load.cpp:1279: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::out_of_range& e) {
          ^

be/src/http/action/stream_load.cpp:1288: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_MERGE_TYPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1291: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != merge_type_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1293: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1296: +2, including nesting penalty of 1, nesting level increased to 2

        if (merge_type == TMergeType::MERGE && req->header(HTTP_DELETE_CONDITION).empty()) {
        ^

be/src/http/action/stream_load.cpp:1298: +1, nesting level increased to 2

        } else if (merge_type != TMergeType::MERGE && !req->header(HTTP_DELETE_CONDITION).empty()) {
               ^

be/src/http/action/stream_load.cpp:1304: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_UNIQUE_KEY_UPDATE_MODE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1311: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != unique_key_update_mode_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1313: +3, including nesting penalty of 2, nesting level increased to 3

            if (unique_key_update_mode == TUniqueKeyUpdateMode::UPDATE_FLEXIBLE_COLUMNS) {
            ^

be/src/http/action/stream_load.cpp:1315: +4, including nesting penalty of 3, nesting level increased to 4

                if (ctx->format != TFileFormatType::FORMAT_JSON) {
                ^

be/src/http/action/stream_load.cpp:1320: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                ^

be/src/http/action/stream_load.cpp:1320: +1

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                                                           ^

be/src/http/action/stream_load.cpp:1325: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1329: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_JSONPATHS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1333: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_HIDDEN_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1338: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL).empty()) {
                ^

be/src/http/action/stream_load.cpp:1343: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_MERGE_TYPE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1348: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_WHERE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1354: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1362: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_FORMAT_KEY).empty()) {
    ^

be/src/http/action/stream_load.cpp:1364: +1, nesting level increased to 1

    } else {
      ^

be/src/http/action/stream_load.cpp:1373: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1374: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1375: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1376: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1377: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1378: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1379: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1380: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1381: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1382: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1383: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1384: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1385: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1386: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1387: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1388: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1389: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1390: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1391: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1392: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1393: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1394: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1395: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1396: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1397: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1398: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1399: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1400: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1401: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1402: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1403: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1404: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1405: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1369: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                       \
    ^

be/src/http/action/stream_load.cpp:1410: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_WHERE).empty()) {
    ^

be/src/http/action/stream_load.cpp Outdated
return ss.str();
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' exceeds recommended size/complexity thresholds [readability-function-size]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1229: 185 lines including whitespace and comments (threshold 80)

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^

@Yufei-YAO Yufei-YAO force-pushed the yyf.dev branch 2 times, most recently from 62ded03 to 59b0045 Compare November 8, 2024 17:10
github-actions[bot]
github-actions bot reviewed Nov 8, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/stream_load.cpp
if (!req->header(HTTP_PARTITIONS).empty()) {
return Status::InvalidArgument(
"Can not specify both partitions and temporary partitions");
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' has cognitive complexity of 102 (threshold 50) [readability-function-cognitive-complexity]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1246: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1246: +1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
                                           ^

be/src/http/action/stream_load.cpp:1248: +2, including nesting penalty of 1, nesting level increased to 2

        if (enclose_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1253: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1253: +1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
                                          ^

be/src/http/action/stream_load.cpp:1255: +2, including nesting penalty of 1, nesting level increased to 2

        if (escape_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1260: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1261: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1266: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1267: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1273: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_SEND_BATCH_PARALLELISM).empty()) {
    ^

be/src/http/action/stream_load.cpp:1276: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::invalid_argument& e) {
          ^

be/src/http/action/stream_load.cpp:1279: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::out_of_range& e) {
          ^

be/src/http/action/stream_load.cpp:1288: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_MERGE_TYPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1291: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != merge_type_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1293: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1296: +2, including nesting penalty of 1, nesting level increased to 2

        if (merge_type == TMergeType::MERGE && req->header(HTTP_DELETE_CONDITION).empty()) {
        ^

be/src/http/action/stream_load.cpp:1298: +1, nesting level increased to 2

        } else if (merge_type != TMergeType::MERGE && !req->header(HTTP_DELETE_CONDITION).empty()) {
               ^

be/src/http/action/stream_load.cpp:1304: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_UNIQUE_KEY_UPDATE_MODE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1311: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != unique_key_update_mode_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1313: +3, including nesting penalty of 2, nesting level increased to 3

            if (unique_key_update_mode == TUniqueKeyUpdateMode::UPDATE_FLEXIBLE_COLUMNS) {
            ^

be/src/http/action/stream_load.cpp:1315: +4, including nesting penalty of 3, nesting level increased to 4

                if (ctx->format != TFileFormatType::FORMAT_JSON) {
                ^

be/src/http/action/stream_load.cpp:1320: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                ^

be/src/http/action/stream_load.cpp:1320: +1

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                                                           ^

be/src/http/action/stream_load.cpp:1325: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1329: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_JSONPATHS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1333: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_HIDDEN_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1338: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL).empty()) {
                ^

be/src/http/action/stream_load.cpp:1343: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_MERGE_TYPE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1348: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_WHERE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1354: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1362: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_FORMAT_KEY).empty()) {
    ^

be/src/http/action/stream_load.cpp:1365: +1, nesting level increased to 1

    } else {
      ^

be/src/http/action/stream_load.cpp:1374: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1375: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1376: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1377: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1378: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1379: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1380: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1381: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1382: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1383: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1384: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1385: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1386: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1387: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1388: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1389: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1390: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1391: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1392: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1393: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1394: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1395: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1396: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1397: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1398: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1399: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1400: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1401: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1402: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1403: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1404: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1405: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1406: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1411: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_WHERE).empty()) {
    ^

be/src/http/action/stream_load.cpp
if (!req->header(HTTP_PARTITIONS).empty()) {
return Status::InvalidArgument(
"Can not specify both partitions and temporary partitions");
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' exceeds recommended size/complexity thresholds [readability-function-size]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1229: 186 lines including whitespace and comments (threshold 80)

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^

@Yufei-YAO Yufei-YAO force-pushed the yyf.dev branch 3 times, most recently from 112d382 to 4fcf985 Compare November 9, 2024 03:46
github-actions[bot]
github-actions bot reviewed Nov 9, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/stream_load.cpp Outdated
return ss.str();
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' has cognitive complexity of 102 (threshold 50) [readability-function-cognitive-complexity]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1246: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1246: +1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
                                           ^

be/src/http/action/stream_load.cpp:1248: +2, including nesting penalty of 1, nesting level increased to 2

        if (enclose_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1253: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1253: +1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
                                          ^

be/src/http/action/stream_load.cpp:1255: +2, including nesting penalty of 1, nesting level increased to 2

        if (escape_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1260: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1261: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1266: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1267: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1273: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_SEND_BATCH_PARALLELISM).empty()) {
    ^

be/src/http/action/stream_load.cpp:1276: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::invalid_argument& e) {
          ^

be/src/http/action/stream_load.cpp:1279: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::out_of_range& e) {
          ^

be/src/http/action/stream_load.cpp:1288: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_MERGE_TYPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1291: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != merge_type_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1293: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1296: +2, including nesting penalty of 1, nesting level increased to 2

        if (merge_type == TMergeType::MERGE && req->header(HTTP_DELETE_CONDITION).empty()) {
        ^

be/src/http/action/stream_load.cpp:1298: +1, nesting level increased to 2

        } else if (merge_type != TMergeType::MERGE && !req->header(HTTP_DELETE_CONDITION).empty()) {
               ^

be/src/http/action/stream_load.cpp:1304: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_UNIQUE_KEY_UPDATE_MODE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1311: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != unique_key_update_mode_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1313: +3, including nesting penalty of 2, nesting level increased to 3

            if (unique_key_update_mode == TUniqueKeyUpdateMode::UPDATE_FLEXIBLE_COLUMNS) {
            ^

be/src/http/action/stream_load.cpp:1315: +4, including nesting penalty of 3, nesting level increased to 4

                if (ctx->format != TFileFormatType::FORMAT_JSON) {
                ^

be/src/http/action/stream_load.cpp:1320: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                ^

be/src/http/action/stream_load.cpp:1320: +1

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                                                           ^

be/src/http/action/stream_load.cpp:1325: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1329: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_JSONPATHS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1333: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_HIDDEN_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1338: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL).empty()) {
                ^

be/src/http/action/stream_load.cpp:1343: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_MERGE_TYPE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1348: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_WHERE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1354: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1362: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_FORMAT_KEY).empty()) {
    ^

be/src/http/action/stream_load.cpp:1365: +1, nesting level increased to 1

    } else {
      ^

be/src/http/action/stream_load.cpp:1374: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1375: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1376: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1377: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1378: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1379: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1380: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1381: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1382: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1383: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1384: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1385: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1386: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1387: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1388: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1389: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1390: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1391: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1392: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1393: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1394: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1395: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1396: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1397: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1398: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1399: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1400: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1401: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1402: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1403: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1404: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1405: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1406: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1370: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1411: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_WHERE).empty()) {
    ^

be/src/http/action/stream_load.cpp Outdated
return ss.str();
}

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_construct_sql_from_req' exceeds recommended size/complexity thresholds [readability-function-size]

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^
Additional context

be/src/http/action/stream_load.cpp:1229: 186 lines including whitespace and comments (threshold 80)

Status StreamLoadAction::_construct_sql_from_req(HttpRequest* req,
                         ^

@sollhui
Copy link
Contributor

sollhui commented Nov 10, 2024

run buildall

@dataroaring
Copy link
Contributor

You'd better set default to true to run all tests.

@Yufei-YAO
Copy link
Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.90% (9867/26035)
Line Coverage: 29.05% (82287/283248)
Region Coverage: 28.18% (42350/150281)
Branch Coverage: 24.72% (21442/86752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dd3ac6d1e9385329df6eb1006c4f0c18629bd27d_dd3ac6d1e9385329df6eb1006c4f0c18629bd27d/report/index.html

@Yufei-YAO
Copy link
Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.89% (9865/26035)
Line Coverage: 29.03% (82235/283248)
Region Coverage: 28.17% (42332/150281)
Branch Coverage: 24.71% (21437/86752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9b86d642f78cd76a902692580a7ece241f1e9fa4_9b86d642f78cd76a902692580a7ece241f1e9fa4/report/index.html

@Yufei-YAO
[Feature](stream-load)(config) implement streamload via http stream (a…
655a00d 
…pache#36773)

1. Add a new BE config `enable_streamload_by_httpstream`
2. Implement streamload via httpstream
@Yufei-YAO
Copy link
Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.90% (9866/26035)
Line Coverage: 29.04% (82245/283248)
Region Coverage: 28.18% (42348/150281)
Branch Coverage: 24.71% (21440/86752)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fbe3896d5a8e1c98d3a1982b0d25a4f5478da87f_fbe3896d5a8e1c98d3a1982b0d25a4f5478da87f/report/index.html

github-actions[bot]
github-actions bot reviewed Nov 12, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/http/action/stream_load.cpp
return ss.str();
}

Status StreamLoadAction::_parse_header(HttpRequest* req, std::shared_ptr<StreamLoadContext> ctx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_parse_header' has cognitive complexity of 171 (threshold 50) [readability-function-cognitive-complexity]

Status StreamLoadAction::_parse_header(HttpRequest* req, std::shared_ptr<StreamLoadContext> ctx) {
                         ^
Additional context

be/src/http/action/stream_load.cpp:1275: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1275: +1

    if (!req->header(HTTP_ENCLOSE).empty() && !req->header(HTTP_ENCLOSE).empty()) {
                                           ^

be/src/http/action/stream_load.cpp:1277: +2, including nesting penalty of 1, nesting level increased to 2

        if (enclose_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1282: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1282: +1

    if (!req->header(HTTP_ESCAPE).empty() && !req->header(HTTP_ESCAPE).empty()) {
                                          ^

be/src/http/action/stream_load.cpp:1284: +2, including nesting penalty of 1, nesting level increased to 2

        if (escape_str.length() != 1) {
        ^

be/src/http/action/stream_load.cpp:1289: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1290: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1295: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_TEMP_PARTITIONS).empty()) {
    ^

be/src/http/action/stream_load.cpp:1296: +2, including nesting penalty of 1, nesting level increased to 2

        if (!req->header(HTTP_PARTITIONS).empty()) {
        ^

be/src/http/action/stream_load.cpp:1302: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_SEND_BATCH_PARALLELISM).empty()) {
    ^

be/src/http/action/stream_load.cpp:1305: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::invalid_argument& e) {
          ^

be/src/http/action/stream_load.cpp:1308: +2, including nesting penalty of 1, nesting level increased to 2

        } catch (const std::out_of_range& e) {
          ^

be/src/http/action/stream_load.cpp:1317: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_MERGE_TYPE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1320: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != merge_type_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1322: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1325: +2, including nesting penalty of 1, nesting level increased to 2

        if (merge_type == TMergeType::MERGE && req->header(HTTP_DELETE_CONDITION).empty()) {
        ^

be/src/http/action/stream_load.cpp:1327: +1, nesting level increased to 2

        } else if (merge_type != TMergeType::MERGE && !req->header(HTTP_DELETE_CONDITION).empty()) {
               ^

be/src/http/action/stream_load.cpp:1333: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_UNIQUE_KEY_UPDATE_MODE).empty()) {
    ^

be/src/http/action/stream_load.cpp:1340: +2, including nesting penalty of 1, nesting level increased to 2

        if (iter != unique_key_update_mode_map.end()) {
        ^

be/src/http/action/stream_load.cpp:1342: +3, including nesting penalty of 2, nesting level increased to 3

            if (unique_key_update_mode == TUniqueKeyUpdateMode::UPDATE_FLEXIBLE_COLUMNS) {
            ^

be/src/http/action/stream_load.cpp:1344: +4, including nesting penalty of 3, nesting level increased to 4

                if (ctx->format != TFileFormatType::FORMAT_JSON) {
                ^

be/src/http/action/stream_load.cpp:1349: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                ^

be/src/http/action/stream_load.cpp:1349: +1

                if (!req->header(HTTP_FUZZY_PARSE).empty() &&
                                                           ^

be/src/http/action/stream_load.cpp:1354: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1358: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_JSONPATHS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1362: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_HIDDEN_COLUMNS).empty()) {
                ^

be/src/http/action/stream_load.cpp:1367: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL).empty()) {
                ^

be/src/http/action/stream_load.cpp:1372: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_MERGE_TYPE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1377: +4, including nesting penalty of 3, nesting level increased to 4

                if (!req->header(HTTP_WHERE).empty()) {
                ^

be/src/http/action/stream_load.cpp:1383: +1, nesting level increased to 2

        } else {
          ^

be/src/http/action/stream_load.cpp:1391: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_FORMAT_KEY).empty()) {
    ^

be/src/http/action/stream_load.cpp:1394: +1, nesting level increased to 1

    } else {
      ^

be/src/http/action/stream_load.cpp:1407: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1407: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1408: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1408: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_COLUMN_SEPARATOR)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1409: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_COMPRESS_TYPE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1409: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_COMPRESS_TYPE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1410: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1410: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_LINE_DELIMITER)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1411: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1411: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_ENCLOSE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1412: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1412: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_ESCAPE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1413: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1413: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1414: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1414: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_TEMP_PARTITIONS)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1415: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1415: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_NEGATIVE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1416: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1416: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_STRICT_MODE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1417: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1417: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_TIME_ZONE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1418: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1418: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_EXEC_MEM_LIMIT)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1419: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1419: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_JSONPATHS)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1420: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1420: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_JSONROOT)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1421: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1421: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_STRIP_OUTER_ARRAY)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1422: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1422: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_NUM_AS_STRING)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1423: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1423: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_FUZZY_PARSE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1424: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1424: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_READ_JSON_BY_LINE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1425: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1425: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_FUNCTION_COLUMN + "." + HTTP_SEQUENCE_COL)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1426: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1426: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_SEND_BATCH_PARALLELISM)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1427: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1427: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_LOAD_TO_SINGLE_TABLET)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1428: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1428: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_MERGE_TYPE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1429: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1429: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_DELETE_CONDITION)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1430: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1430: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_MAX_FILTER_RATIO)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1431: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1431: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_HIDDEN_COLUMNS)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1432: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1432: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_TRIM_DOUBLE_QUOTES)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1433: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1433: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_SKIP_LINES)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1434: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1434: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_ENABLE_PROFILE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1435: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1435: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1436: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1436: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_MEMTABLE_ON_SINKNODE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1437: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1437: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_LOAD_STREAM_PER_NODE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1438: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1438: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_GROUP_COMMIT)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1439: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1439: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_CLOUD_CLUSTER)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1440: +1, including nesting penalty of 0, nesting level increased to 1

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1400: expanded from macro '__ADD_IF_EXIST'

    if (!req->header(PROPERTIY_KEY).empty()) {                                                     \
    ^

be/src/http/action/stream_load.cpp:1440: +2, including nesting penalty of 1, nesting level increased to 2

    __ADD_IF_EXIST(HTTP_UNIQUE_KEY_UPDATE_MODE)
    ^

be/src/http/action/stream_load.cpp:1401: expanded from macro '__ADD_IF_EXIST'

        if (pri) {                                                                                 \
        ^

be/src/http/action/stream_load.cpp:1445: +1, including nesting penalty of 0, nesting level increased to 1

    if (!req->header(HTTP_WHERE).empty()) {
    ^

be/src/http/action/stream_load.cpp
return ss.str();
}

Status StreamLoadAction::_parse_header(HttpRequest* req, std::shared_ptr<StreamLoadContext> ctx) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_parse_header' exceeds recommended size/complexity thresholds [readability-function-size]

Status StreamLoadAction::_parse_header(HttpRequest* req, std::shared_ptr<StreamLoadContext> ctx) {
                         ^
Additional context

be/src/http/action/stream_load.cpp:1233: 218 lines including whitespace and comments (threshold 80)

Status StreamLoadAction::_parse_header(HttpRequest* req, std::shared_ptr<StreamLoadContext> ctx) {
                         ^

@Yufei-YAO
Copy link
Author

run buildall

@Yufei-YAO
Copy link
Author

run buildall

1 similar comment
@Yufei-YAO
Copy link
Author

run buildall

@Yufei-YAO
Copy link
Author

run buildall

@Yufei-YAO
Copy link
Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Yufei-YAO @doris-robot @sollhui @dataroaring