Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: row protocol support for opentsdb #2468

Closed
wants to merge 36 commits into from
Closed

feat: row protocol support for opentsdb #2468

wants to merge 36 commits into from

Conversation

Lilit0x
Copy link
Contributor

@Lilit0x Lilit0x commented Sep 21, 2023 •
edited by MichaelScofield
Loading

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Support for row inserts in opentsdb

Please explain IN DETAIL what the changes are in this PR and why they are needed:
OpenTSDB now uses the row insert protocol as against column inserts

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

close #2444

@Lilit0x Lilit0x marked this pull request as draft September 21, 2023 20:06
@Lilit0x Lilit0x marked this pull request as ready for review September 25, 2023 21:19
@MichaelScofield
Copy link
Collaborator

@Lilit0x can you fix the failed tests?

@codecov
Copy link

codecov bot commented Sep 26, 2023 •
edited
Loading

Codecov Report

Merging #2468 (fcf65a1) into develop (201acd1) will decrease coverage by 0.57%.
The diff coverage is 100.00%.

❗ Current head fcf65a1 differs from pull request most recent head ff300d2. Consider uploading reports for the commit ff300d2 to get more accurate results

Additional details and impacted files 
@@             Coverage Diff             @@
##           develop    #2468      +/-   ##
===========================================
- Coverage    85.28%   84.72%   -0.57%     
===========================================
  Files          730      724       -6     
  Lines       116093   115218     -875     
===========================================
- Hits         99009    97615    -1394     
- Misses       17084    17603     +519

@fengjiachun
Copy link
Collaborator

Hi, @Lilit0x , can we make OpentsdbProtocolHandler accept a collection of DataPoints? In this way, we can convert the collection of DataPoints into a RowInsertRequests at once. I think this will improve efficiency. cc @MichaelScofield

@Lilit0x
Copy link
Contributor Author

Lilit0x commented Sep 27, 2023

Hi, @Lilit0x , can we make OpentsdbProtocolHandler accept a collection of DataPoints? In this way, we can convert the collection of DataPoints into a RowInsertRequests at once. I think this will improve efficiency. cc @MichaelScofield

Hi, that makes sense. I will work on it.

waynexia and others added 22 commits October 2, 2023 11:39
@waynexia @Lilit0x
refactor: remove SqlStatementExecutor (#2464)
67651e2 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@TheWaWaR @Lilit0x
feat: limit grpc message size (#2459)
2292c41 
* feat: add two grpc config options

Those options are for:
* Limit receiving(decoding) message size
* Limit sending(enoding) message size

* test: add integration tests for message size limit
@fengjiachun @Lilit0x
chore: remove unused region_stats method form table (#2458)
f9315bd 
chore: remove unused region_status method form table
@Lilit0x
feat: support for show full tables (#2410)
37156fc 
* feat: added show tables command

* fix(tests): fixed parser and statement unit tests

* chore: implemeted display trait for table type

* fix: handled no tabletype and error for usopprted command in show databse

* chore: removed full as a show kind, instead as a show option

* chore(tests): fixed failing test and added more tests for show full

* chore: refactored table types to use filters

* fix: changed table_type to tables
@evenyag @Lilit0x
feat: update proto and remove create_if_not_exists (#2467)
8a12762
@evenyag @Lilit0x
fix(mito): compaction scheduler schedules more tasks than expected (#…
e2a8c0c 
…2466)

* test: test on_compaction_finished

* fix: avoid submit same region to compact

* feat: persist and recover compaction time window

* test: fix test

* test: sort like result
@QuenKar @killme2008 @Lilit0x
refactor: not allowed int64 type as time index (#2460)
5e1322e 
* refactor: remove is_timestamp_compatible.

* chore: fmt

* refactor: remove int64 to timestamp match

* chore

* chore: apply suggestions from code review

Co-authored-by: dennis zhuang <killme2008@gmail.com>

* chore: fmt

---------

Co-authored-by: dennis zhuang <killme2008@gmail.com>
@fengjiachun @Lilit0x
chore: typo (#2470)
e7c1d91
@sunng87 @Lilit0x
feat: implement storage for OTLP histogram (#2282)
ff2b44e 
* feat: implement new histogram data model

* feat:  use prometheus table format for histogram

* refactor: remove duplicated code

* fix: histogram tag column

* fix: use accumulated count in buckets

* refactor: using row based protocol for otlp WIP

* refactor: use row based writer for otlp.

Also updated row writer for owned keys

* refactor: use row writers for otlp

* test: add integration tests for histogram

* refactor: change le column name
@evenyag @Lilit0x
fix(mito): reset is_sorted to true after the merger finishing one ser…
2d22d60 
…ies (#2474)

fix: reset is_sorted flag to true after the merger finishing one series
@evenyag @Lilit0x
fix(mito): normalize region dir in RegionOpener (#2475)
a3ff532 
fix: normalize region dir in RegionOpener
@fengjiachun @Lilit0x
chore: refactor options (#2476)
e4cbe5f
@zyy17 @Lilit0x
chore: fix typo (#2477)
e051841
@NiwakaDev @Lilit0x
chore: add integration test for issue2437 (#2481)
198fcba
@Lilit0x
fix: changed tags to string type
c6c8c2a
@MichaelScofield @Lilit0x
fix: print root cause error message to user facing interface (#2486)
87589e7
@evenyag @Lilit0x
refactor(mito): remove #[allow(dead_code)] (#2479)
958adf7
@QuenKar @evenyag @Lilit0x
refactor: datetime time unit (#2469)
6efc972 
* refactor: datetime time unit

* Update src/common/time/src/datetime.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: cr.

---------

Co-authored-by: Yingwen <realevenyag@gmail.com>
@WenyXu @Lilit0x
fix: dn doesn't have chance to send a heartbeat to the new leader (#2471
013a3fb 
)

* refactor: set meta leader lease secs to 3s

* fix: correct default heartbeat interval

* refactor: ask meta leader in parallel

* feat: configure heartbeat client timeout to 500ms

* fix: trigger to send heartbeat immediately after fail

* fix: fix clippy
@Lilit0x
fix(tests): fixed open tsdb integration test
212c490
@Lilit0x
fix(tests): aligned top row in table for failing test
6ede1be
@Lilit0x
fix(tests): aligned top row in table for failing test
662e9bb
zhongzc and others added 13 commits October 2, 2023 11:39
@zhongzc @Lilit0x
feat: re-support query engine execute dml (#2484)
13d22ad 
* feat: re-support query engine execute dml

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: remove region_number in InsertRequest

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: add doc comments

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@zhongzc @Lilit0x
fix: convert datetime to chrono datetime (#2497)
a2fcb27 
* fix: convert datetime to chrono datetime

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* chore: typo

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

* fix the bad fix

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>

---------

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@DevilExileSu @Lilit0x
feat: distribute truncate table in region server (#2414)
d80977a 
* feat: distribute truncate table

* chore: add metrics for truncate table

* test: add sqlness test

* chore: cr

* test: add multi truncate

* chore: add trace id to the header
@waynexia @Lilit0x
feat: stack trace style debug print for error (#2489)
c53f366 
* impl macro stack_trace_debug

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* manually mark external error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* ignore warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warnings

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use debug print

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplify the error and warn macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add docs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* replace snafu backtrace with location

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@evenyag @Lilit0x
feat(mito): Add cache manager (#2488)
090e100 
* feat: add cache manager

* feat: add cache to reader builder

* feat: add AsyncFileReaderCache

* feat: Impl AsyncFileReaderCache

* chore: move moka dep to workspace

* feat: add moka cache to the manager

* feat: implement parquet meta cache

* test: test cache manager

* feat: consider vec size

* style: fix clippy

* test: fix config api test

* feat: divide cache

* test: test disabling meta cache

* test: fix config api test

* feat: remove meta cache if file is purged
@waynexia @Lilit0x
fix: check for table scan before expanding (#2491)
c20bade 
* fix: check for table scan before expanding

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* change assert_ok to unwrap

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix clippy warning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't skip dml

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* uncomment ignored tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@evenyag @Lilit0x
fix(mito): fix region drop task runs multiple times but never clean t…
987eefb 
…he dir (#2504)

fix: fix region drop task runs multiple times but never clean the directory
@waynexia @Lilit0x
refactor: simplify warn! and error! macros (#2503)
efbd857 
* refactor: simplify the error! and warn! macros

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* support display format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* err.msg to err

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@WenyXu @Lilit0x
fix: re-create heartbeat stream ASAP (#2499)
5d8f46f 
* chore: set default connect_timeout_millis to 1000

* fix: re-create heartbeat stream ASAP

* chore: apply suggestions
@shuiyisong @waynexia @Lilit0x
chore: tune return msg (#2506)
cfbee03 
* chore: test return msg

* fix: test_child_error

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* chore: fix test

* chore: minor fix grpc return value

* chore: format return msg

* chore: use root error as return value

* chore: fix empty err display

* chore: iter through external error

* chore: remove err msg

* chore: remove unused field

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
@MichaelScofield @Lilit0x
fix: allow .(dot) literal in table name (#2483)
a71bf83 
* fix: allow `.`(dot) literal in table name

* fix: resolve PR comments
@zhongzc @Lilit0x
feat: push down filters to region engine (#2513)
6eb49b5 
feat: pushdown filters to region engine

Signed-off-by: Zhenchi <zhongzc_arch@outlook.com>
@Lilit0x
chore: changed opentsd handler to accept collection of datapoints
9d72117
@Lilit0x Lilit0x closed this Oct 2, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fengjiachun fengjiachun Awaiting requested review from fengjiachun

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use row protocol to support opentsdb