Feature/pipeline queue manager #2334
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #2334 +/- ##
==========================================
- Coverage 15.90% 15.72% -0.19%
==========================================
Files 1195 1214 +19
Lines 118228 122682 +4454
==========================================
+ Hits 18810 19286 +476
- Misses 97500 101411 +3911
- Partials 1918 1985 +67
|
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
2 times, most recently
from
fad96fc to
4790747
Compare
|
|
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
from
4790747 to
0751721
Compare
|
Add at least 1000 pipeline to queue and tested it. |
|
2000 queue item consume about 86kb, one etcd key limit 1.5m value, so, one queue may save 35000 queue item |
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
4 times, most recently
from
cd628b5 to
5ff713c
Compare
sfwn
requested changes
Oct 15, 2021
| r.continueBackupQueueUsage(ctx) | ||
| }() | ||
| go func() { | ||
| r.QueueManager.Listen(ctx) |
| costTime = end.Sub(begin) | ||
| queueSnapshot := manager.SnapshotObj{} | ||
| if err := json.Unmarshal(backup, &queueSnapshot); err != nil { | ||
| errDone <- err |
| }) | ||
| } | ||
|
|
||
| func (mgr *defaultManager) SendPipelineIDS(queueID uint64, pipelineIDS []uint64) { |
| ) | ||
|
|
||
| const ( | ||
| etcdQueueWatchPrefix = "/devops/pipeline/queue/action/update/" |
| mgr.qLock.Lock() | ||
| defer mgr.qLock.Unlock() | ||
|
|
||
| // construct newQueue first for later use |
There was a problem hiding this comment.
Abstract these code to a function. And I remember already have a function?
| return b | ||
| } | ||
|
|
||
| // Import default queue manager execute in memory, don't need import |
modules/pipeline/pipengine/reconciler/queuemanage/types/manager.go
Outdated
Show resolved
Hide resolved
sfwn
requested changes
Oct 15, 2021
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
2 times, most recently
from
17d4b5d to
bf43a39
Compare
sfwn
requested changes
Oct 16, 2021
modules/pipeline/pipengine/reconciler/queuemanage/manager/usage.go
Outdated
Show resolved
Hide resolved
sfwn
requested changes
Oct 16, 2021
modules/pipeline/pipengine/reconciler/queuemanage/types/manager.go
Outdated
Show resolved
Hide resolved
modules/pipeline/pipengine/reconciler/queuemanage/types/manager.go
Outdated
Show resolved
Hide resolved
modules/pipeline/pipengine/reconciler/queuemanage/types/manager.go
Outdated
Show resolved
Hide resolved
modules/pipeline/pipengine/reconciler/queuemanage/types/manager.go
Outdated
Show resolved
Hide resolved
sfwn
requested changes
Oct 16, 2021
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
from
bf43a39 to
aa96fdd
Compare
sfwn
requested changes
Oct 18, 2021
| @@ -27,4 +30,9 @@ type QueueManager interface { | |||
| PopOutPipelineFromQueue(pipelineID uint64) | |||
| BatchUpdatePipelinePriorityInQueue(pq *apistructs.PipelineQueue, pipelineIDs []uint64) error | |||
| Stop() | |||
| SendQueueToETCD(queueID uint64) | |||
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
from
aa96fdd to
7ef61d5
Compare
sfwn
previously approved these changes
Oct 18, 2021
chengjoey
force-pushed
the
feature/pipeline-queue-manager
branch
from
7ef61d5 to
b8a6bf7
Compare
sfwn
approved these changes
Oct 18, 2021
|
/approve |
erda-bot
approved these changes
Oct 18, 2021
|
/cherry-pick release/1.3 |
erda-bot
pushed a commit
to erda-bot/erda
that referenced
this pull request
Oct 18, 2021
* annotate backup throttler func * query, update queue usage to etcd * use protobuf marshal unmarshal queue usage * watch etcd to update queue instead of in memory
johnlanni
pushed a commit
that referenced
this pull request
Oct 19, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of this PR
Add one of the following kinds:
/kind feature
What this PR does / why we need it:
update queue usage to etcd for multiple pipeline
Which issue(s) this PR fixes:
Specified Reviewers:
/assign @your-reviewer
ChangeLog
Need cherry-pick to release versions?
Add comment like
/cherry-pick release/1.0when this PR is merged.