plugin: introduce plugin framework #8788

lysu · 2018-12-24T12:07:44Z

What problem does this PR solve?

introduce plugin framework to tidb

What is changed and how it works?

How to use

here just a simple describe how to write a new plugin, later will give a full developer guide.

choose a plugin kind or no suitable need add new plugin kind
create a normal go package, add manifest.toml like example one
implement validate, init, destory method which is needed for all plugin
implement kind special method to implement plugin logic
use cmd/pluginpkg to build plugin binary, and put plugin binary into plugin deploy folder
start TiDB with -plugin-dir and -plugin-load parameter
show plugin to check it's load status

What change

add a new plugin pkg, the design detail described in a proposal doc

Check List

Tests

Unit test
Manual test (add detailed scripts or steps below)
WIP

Code changes

Add new package
Add new cmd tool

Side effects

Increased code complexity

Related changes

Need to update the documentation
Need to be included in the release note

Code Review Guide

proposal: docs/design: add proposal for plugin framework #8802
framework: plugin: introduce plugin framework #8788 (current PR)
tidb-integration: *: integrate plugin framework with TiDB #9006
package tool: cmd: add plugin package tool #9007

This change is

shenli · 2018-12-24T13:35:41Z

@lysu Good job!
This is a huge PR. Could you split it into multiple PRs? At least you can send the proposal first.

tiancaiamao

Would you split this PR into small parts?
For example, document and example can be a separate one;
show plugins and config related things could comes after the plugin package core.

Smaller PR is more friendly to reviewers.

cmd/pluginpkg/pluginpkg.go

plugin/conn_ip_example/conn_ip_example.go

plugin/const.go

tiancaiamao · 2019-01-09T15:45:07Z

plugin/plugin.go

+				return
+			}
+			p := tiPlugins.plugins[kind][i]
+			if err = p.OnInit(ctx, p.Manifest); err != nil {


Should we defer panic for plugin init, thus plugin panic will not make TiDB panic ?

IMHO, it's better let framework user to decide panic or do other things - -

@tiancaiamao PR has be splitted.

plugin/plugin.go

plugin/spi.go

codecov-io · 2019-01-10T11:36:59Z

Codecov Report

Merging #8788 into master will decrease coverage by 0.21%.
The diff coverage is 7.27%.

@@            Coverage Diff             @@
##           master    #8788      +/-   ##
==========================================
- Coverage   67.63%   67.41%   -0.22%     
==========================================
  Files         364      370       +6     
  Lines       75471    75732     +261     
==========================================
+ Hits        51043    51057      +14     
- Misses      19929    20176     +247     
  Partials     4499     4499

Impacted Files	Coverage Δ
plugin/const.go	`0% <0%> (ø)`
plugin/conn_ip_example/conn_ip_example.go	`0% <0%> (ø)`
plugin/plugin.go	`0% <0%> (ø)`
plugin/spi.go	`100% <100%> (ø)`
plugin/errors.go	`100% <100%> (ø)`
plugin/helper.go	`12.5% <12.5%> (ø)`
ddl/session_pool.go	`82.75% <0%> (-10.35%)`	⬇️
executor/join.go	`77.92% <0%> (-0.52%)`	⬇️
executor/distsql.go	`72.53% <0%> (-0.47%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ac79f3...cf8a45c. Read the comment docs.

plugin/plugin.go

jackysp · 2019-01-11T03:28:14Z

plugin/plugin.go

+			return errors.Errorf("plugin %s isn't loaded so can not reload", p.Name)
+		}
+		if len(p.SysVars) != len(oldPlugin.SysVars) {
+			return errors.Errorf("reload plugin with different sysVar is unsupported")


It's best to define a plugin error category that we will need it sooner or later.

it's little confuse, so we should make plugin rely on parser's predefine error?

@tiancaiamao @jackysp message move to parser, please review parser first pingcap/parser#173

plugin/conn_ip_example/conn_ip_example.go

plugin/conn_ip_example/conn_ip_example_test.go

plugin/const.go

plugin/helper.go

tiancaiamao · 2019-01-12T13:24:59Z

plugin/helper.go

+type ID string
+
+// Decode decodes a plugin id into name, version parts.
+func (n ID) Decode() (name string, version string, err error) {


// Plugin name should not contain '-'

Decode has check a42f8e9#diff-51012a4c21599aa07955e2f374958792R40, refine msg in new commit, and pluginpkg also add a check 5bbc246

plugin/plugin.go

tiancaiamao · 2019-01-12T15:47:06Z

plugin/spi.go

+// ExportManifest exports a manifest to TiDB as a known format.
+// it just casts sub-manifest to manifest.
+func ExportManifest(m interface{}) *Manifest {
+	return (*Manifest)((*(*iface)(unsafe.Pointer(&m))).Data)


If Manifest is not the first field in the extended manifest struct, this code broke.

Yes, but only tidb developer can add new kind of manifest, this can be restricted by code review(and it also has be documented); for plugin develop only touch toml configuration with predefine manifest

plugin/spi.go

plugin/spi_test.go

lysu · 2019-01-13T13:58:57Z

/run-all-tests

tiancaiamao · 2019-01-13T14:24:51Z

LGTM
Please fix CI

lysu · 2019-01-13T14:30:05Z

ok~ @tiancaiamao now this PR use my parser branch so CI fail on tidy check, after parser PR shipped this will pass

jackysp

LGTM

lysu · 2019-01-14T02:00:22Z

/run-all-tests

lysu added the type/new-feature label Dec 24, 2018

lysu changed the title ~~plugin: introduce plugin framework to TiDB~~ plugin: introduce plugin framework Dec 24, 2018

lysu added the status/WIP label Dec 24, 2018

jackysp added the type/TEP label Dec 25, 2018

tiancaiamao reviewed Jan 9, 2019

View reviewed changes

This was referenced Jan 10, 2019

*: integrate plugin framework with TiDB #9006

Merged

cmd: add plugin package tool #9007

Merged

lysu requested a review from jackysp January 10, 2019 11:22

lysu added component/plugin and removed status/WIP labels Jan 10, 2019

jackysp reviewed Jan 11, 2019

View reviewed changes

plugin/plugin.go Show resolved Hide resolved

jackysp reviewed Jan 11, 2019

View reviewed changes

tiancaiamao reviewed Jan 12, 2019

View reviewed changes

lysu added 7 commits January 13, 2019 16:55

plugin: plugin framework core

c9105ff

fix ci

148b2f3

fix ci

8b9d4b6

address comment

a42f8e9

address comment, check plugin name

5bbc246

address comment, use reflection

27e6502

add schema/daemon plugin type.

04cd85f

lysu mentioned this pull request Jan 13, 2019

mysql,terror: add plugin error msg pingcap/parser#173

Merged

use parser error msg

cf351e7

tiancaiamao added the status/LGT1 Indicates that a PR has LGTM 1. label Jan 13, 2019

jackysp previously approved these changes Jan 14, 2019

View reviewed changes

use parser master

cf8a45c

lysu dismissed jackysp’s stale review via cf8a45c January 14, 2019 02:00

lysu added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Jan 14, 2019

jackysp approved these changes Jan 14, 2019

View reviewed changes

jackysp added the status/all tests passed label Jan 14, 2019

lysu merged commit e1381b4 into pingcap:master Jan 14, 2019

lysu mentioned this pull request Mar 25, 2019

plugin: introduce plugin framework (#8788) #9880

Merged

zz-jason pushed a commit that referenced this pull request Mar 25, 2019

plugin: introduce plugin framework (#8788) (#9880)

49c5972

lysu deleted the dev-plugin-fw branch March 25, 2019 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

plugin: introduce plugin framework #8788

plugin: introduce plugin framework #8788

lysu commented Dec 24, 2018 •

edited

Loading

shenli commented Dec 24, 2018

tiancaiamao left a comment

tiancaiamao Jan 9, 2019

lysu Jan 10, 2019

lysu Jan 10, 2019

codecov-io commented Jan 10, 2019 •

edited

Loading

jackysp Jan 11, 2019

lysu Jan 13, 2019

lysu Jan 13, 2019

tiancaiamao Jan 12, 2019

lysu Jan 13, 2019 •

edited

Loading

tiancaiamao Jan 12, 2019

lysu Jan 13, 2019

lysu commented Jan 13, 2019

tiancaiamao commented Jan 13, 2019

lysu commented Jan 13, 2019

jackysp left a comment

lysu commented Jan 14, 2019

plugin: introduce plugin framework #8788

plugin: introduce plugin framework #8788

Conversation

lysu commented Dec 24, 2018 • edited Loading

What problem does this PR solve?

What is changed and how it works?

How to use

What change

Check List

Code Review Guide

shenli commented Dec 24, 2018

tiancaiamao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Jan 10, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu Jan 13, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lysu commented Jan 13, 2019

tiancaiamao commented Jan 13, 2019

lysu commented Jan 13, 2019

jackysp left a comment

Choose a reason for hiding this comment

lysu commented Jan 14, 2019

lysu commented Dec 24, 2018 •

edited

Loading

codecov-io commented Jan 10, 2019 •

edited

Loading

lysu Jan 13, 2019 •

edited

Loading