Skip to content

Commit

Permalink
cherry pick #7740 "Remove the usage of TmpFromAccountId and TmpToAcco…
Browse files Browse the repository at this point in the history
…untId in Jira issue related data" to v0.21. (#7741)

* fix: github deployment status active and inactive should be treated as success (#6873) (#6874)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: github run api page<35 limit (#6876) (#6878)

* fix: github run api page<35 limit

* fix: some tips

* fix: update note

Co-authored-by: abeizn <zikuan.an@merico.dev>

* feat: support hide custom type toast in operator (#6879) (#6880)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: some bugs (#6882) (#6883)

* fix: operator toast control error

* fix: adapt to small screen margins

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: adjust the style for page header (#6886) (#6887)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* feat: auto cherry pick support multiple versions (#6894)

* fix: the layout error (#6895) (#6896)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: icon missed in data scope select (#6897) (#6898)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: triggering blueprint concurrently might lead to deadlock (#6901) (#6904)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: triggering blueprint concurrently might lead to deadlock (#6902) (#6903)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: asfheader format (#6900) (#6906)

* fix: asf header format

* fix: asf header format

* fix: asf header format

* fix: asf header format

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: adjust the layout for page (#6909) (#6910)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: adjust the margin for page layout (#6913) (#6914)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: copywriting (#6917)

* cherry pick #6881 #6871 update dashboards, remove warnings to release v0.21 (#6919)

* fix(dashboard): update circleci dashboard

* fix(dashboard): update AzureDevops

* fix(dashboards): remove warnings

* fix(dashboard): fix selected_path filter

* feat: upgrade dora benchmarks to 2023 version (#6893) (#6920)

* feat: upgrade dora benchmarks to 2023 version

* feat: add 2021/2023 benchmark flag

* feat: add 2021/2023 benchmark flag

Co-authored-by: abeizn <zikuan.an@merico.dev>

* chore: upgrade antd to 5.14.0 (#6922) (#6924)

* chore: upgrade antd to 5.14.0

* fix: lint error

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: default value error in jira transformation (#6925) (#6926)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: missed redirect path prefix in connection detail (#6928) (#6929)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: rerun task deadlock (#6932) (#6933)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: new dshelper does not set raw data origin properly (#6935) (#6936)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* feat: add default name for scope config (#6937) (#6938)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: rerun pipeline deadlock (#6939) (#6940)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: deployment dora dashboard (#6931) (#6942)

* fix: deployment dora dashboard

* fix: deployment dashbord add dora debug update

* fix: some note

* fix: adjust sql

* fix: mysql id lint

* fix: some bugs

* fix: some note

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: update cfr metric (#6945) (#6946)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* feat(dashboard): revert changes that try to remove wanings (#6948)

* chore: adds support for IPv6 in nginx (#6955) (#6960)

(cherry picked from commit 89f1461)

* fix: /grafana should be off (#6966) (#6967)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix(migration): fix rename operations in tapd plugin (#6970) (#6972)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* feat: extends DNS variable assignment to additionally handle IPv6 (#6976) (#6979)

(cherry picked from commit 716d1e4)

* fix(zentao): update the type of `PriOrder` (#6986) (#6987)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* refactor(middlewares): add `CheckAuthorizationHeader` (#6993) (#6994)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* refactor: optimize bp cronjob scheduling (#6995) (#6996)

* refactor: optimize bp cronjob scheduling

* fix: linting

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* feat: updates backend to listen on both IPv4 and IPv6 addresses (#6992)

(cherry picked from commit d15956e)

* cherry pick #7001 fix(azuredevops): fix some fields' value to v0.21 (#7003)

* fix(azuredevops): fix some fields' value

* fix(azuredevops): fix updated_date in cicd_scopes

* fix(azuredevops): fix test

* fix(azuredevops): fix e2e test errors

* refactor(azuredevops): update mysqlclien to v2.2.4, make it compatible with mysql8.3 on macOS (#7004) (#7012)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* [AzureDevops ] fix: some fields' values (#7013) (#7015)

* fix(auzredevops): remove update_date from repos and cicd_scopes

* fix(azuredevops): fix environment field in cicd_tasks and cicd_pipelins

* fix(azuredevops): fix enviroment field in cicd_tasks and cicd_pipelines

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: invalid url detection not working correctly (#7016) (#7017)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix(gitextractor): cover errors that may leak sensitive information (#7018) (#7019)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(azuredevops): fix unexpected environment field in cicd_tasks and cicd_pipelines (#7022) (#7023)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: unset scopeConfigId for scope not working on azdo/bamboo (#7024) (#7025)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* feat(cicd): generate deployment in cicd plugin (#6965) (#7028)

(cherry picked from commit 3254310)

* feat: missed circleci transformation (#7011) (#7029)

* fix: jira test connection error message for wrong user/pass not working (#7031) (#7034)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: cfr dashboard value (#7035)

* fix(dockerfile): try to fix build errors (#7036) (#7037)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(ui): Add /health url with basicAuth disabled (#7009) (#7038)

* fix(zentao): fix `PatchConnection` (#7039) (#7040)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix `updated_at` field in cicd_tasks and cicd_pipelines (#7032) (#7041)

* fix(gitextractor): cover errors that may leak sensitive information

* fix(azuredevops): fix updated_at field in cicd_tasks and cicd_pipelines

* fix(azuredevops): fix test errors

* fix(azuredevops): remove domain layer tables when collecting data fully

* fix(azuredevops): remove debug codes

* fix(azuredevops): fix testing errors

* fix(azuredevops): fix test errors

* fix(azuredevops): fix comments(most of them are about format)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(tapd): sanitize connection's respo (#7042) (#7043)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: cicd deployment dora delete when data nil (#7033) (#7044)

* fix: cicd deployment dora delete when data nil

* fix: lint

* fix: dora delete data where the data after filtering conditions is empty

* fix: comments

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(security): fix cwe-276, (parts of)cwe-22 (#7045) (#7047)

* fix(security): fix cwe-276, (parts of)cwe-22

* fix(test): fix errors

* fix(test): fix errors

* fix(test): fix errors

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(zentao): fix parsing errors for date related fields on zentao v18.10 (#7055)

* feat: add docs about v0.21 (#7030) (#7057)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* chore: adjust the doc link to v0.21 (#7058)

* fix: pr cycle time doesn't respect time interval and is not null replace of != null (#7056)

* fix(azuredevops): fix environment field in cicd_tasks and cicd_pipelines (#7061)

* fix(azuredevops): remove unneccessary collectors when re-transformating project (#7062) (#7063)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(azuredevops): fix environment field (#7064) (#7065)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* Fetch AzureDevops Builds by `finishTimeDescending` (#7068) (#7069)

* fix(azuredevops): fix environment field

* fix(azuredevops): fetch build with queryOrder = finishTimeDescending

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: jenkins should be using `fullName` instead of `scopeId` when generating plan (#7067) (#7071)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: empty dora data when run the tasks (#7070) (#7072)

* fix: empty dora data when run the tasks

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: jenkins task name error (#7066) (#7077)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* feat(plugins): add WRAP_RESPONSE_ERROR in config to avoid SSRF when testing connections (#7080) (#7081)

* feat(plugins): add WRAP_RESPONSE_ERROR in config to avoid SSRF when testing connections

* fix(plugins): update error messages

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: show badge in connection name (#7085) (#7087)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: gitlab delate data by raw data table (#7095) (#7102)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: sonarqube connection delete on no scope config (#7093) (#7103)

* fix: sonarqube connection delete on no scope config

Co-authored-by: abeizn <zikuan.an@merico.dev>

* chore: upgrade miller-columns-select to 1.3.1 (#7109)

* fix: modify deployable_commit_title from varchar255 to text (#7106) (#7110)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(azuredevops): fix some bugs (#7108) (#7111)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: sonarqube test connection can not effective (#7112) (#7113)

* fix: sonarqube test connection can not effective

* fix: testConnection add check

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: cronjob won't work for a new instance (#7119) (#7120)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix(dora): remove calculated `started_date` when converting pipelines to deployments (#7122) (#7123)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* chore: upgrade miller-columns-select to 1.3.2 (#7126)

* fix: revert dora empty (#7128) (#7129)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(dora): add started_date when generating cicd_deployment_commits (#7130) (#7131)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: missed path prefix (#7127) (#7133)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: project path error (#7134) (#7136)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix(plugins): fix wrong resp when testing connections, fix a typo in bitbucket (#7138)

* fix(customize): issue_repo_commit.host column ignores port, consistant with devinsights change. (#7094) (#7140)

* [bug][customize] Import csv issues API handle issues in multiple boards (#6978) (#7142)

* fix(customize): csv import should handle issues belong to multiple boards

* fix(customize): use more practical csv file for update test

* fix: add flag for data convert to dora (#7143)

* fix: add flag for data convert to dora

* fix: some minor optimizations

* fix: add subtask_name filter

* fix: blueprint gets disabled when disabling dora for project (#7146) (#7147)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: missed plugin opsgenie doc (#7144) (#7145)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: add connectionId params on swag (#7152)

* fix: test connection swag doc (#7151)

* fix: unable to trigger blueprints with webhooks only (#7148) (#7149)

Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>

* fix: jira epic add time after (#7161) (#7162)

* fix: jira epic add time after

* fix: description

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: some logs (#7163) (#7164)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: jira server test connection bug (#7165) (#7166)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: limit layout content width to 1280 (#7169) (#7170)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix(helper): add more log (#7171)

Co-authored-by: houlinwei <timeror@gmail.com>

* Morelog (#7173)

* fix(helper): add more log

* fix(helper): add more log

---------

Co-authored-by: houlinwei <timeror@gmail.com>

* fix(zentao): update error message when testing connection fails (#7175) (#7176)

Co-authored-by: Lynwee <linwei.hou@merico.dev>
Co-authored-by: houlinwei <timeror@gmail.com>

* fix(helper): remove debug logs (#7179)

* fix(helper): remove debug logs

* fix(helper): remove logs

---------

Co-authored-by: houlinwei <timeror@gmail.com>

* fix: unnecessary loading (#7183) (#7186)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: no correct reset state (#7191) (#7192)

Co-authored-by: 青湛 <0x1304570@gmail.com>

* fix: jira jql checker add debug log (#7215) (#7216)

* fix: jira jql checker add debug log

* fix: jira jql checker add debug log

* fix: jira jql checker add debug log

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: soc2 upgrade  (#7271)

* fix: upgrade go git

* fix: upgrade dgrijalva/jwt-go

* fix(python): update urllib3 to latest version (#7267)

* fix: upgrade grafana version

* fix: upgrade grafana

* fix: upgrade grafana

---------

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* feat: add devops file to v0.21 (#7296)

* fix(azuredevops): fix _tool_azuredevops_gitrepositories.id field (#7301) (#7302)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: db timeafter adjust (#7314) (#7316)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(tapd): change iteration_id's type from uint64 to int64 (#7334) (#7336)

* fix(tapd): change iteration_id's type from uint64 to int64

* fix(tapd): fix test

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix: add jira jql checker debug log (#7376) (#7377)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: customize plugin can not update old data (#7389) (#7392)

* fix: customize plugin can not update old data

* fix: recover pipeline_plan

* fix: e2e test

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(zentao): update storyID field in tasks reponse, make it compatiable with oss 18.11 (#7403) (#7404)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* feat: fetch execution information from the built-in page interface (#7513) (#7514)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: execution id not exist (#7520) (#7522)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* feat: add assignees and reviewers to pull_request_x table (#7556)

* feat: add assignees and reviewers to pull_request_x table

* fix: unit test and e2e test

* fix: unit test

* fix: some tips

* fix: rename assignee reviewer id name

* feat: starrocks add STARROCKS_REPLICAS_NUM env (#7565)

* fix: add primary key to _tool_gitlab_assignees and _tool_gitlab_reviewers tables (#7573) (#7574)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: gitlab pr reviewers and assignees id (#7588) (#7594)

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix: pr reviewers and assignees script version error (#7645) (#7647)

* fix: pr reviewers and assignees script version error

* fix: adapt pg test

Co-authored-by: abeizn <zikuan.an@merico.dev>

* fix(zentao): make sure close_date must be before opened_date (#7699) (#7700)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* cherry pick #7710 feat(jira): collector accounts from issue changelog items as much as possible to v0.21 (#7724)

* feat(jira): collector account from issue changelog items as much as possible

* fix(jira): fix migration scripts

* feat(jira): add `_tool_jira_issue_fields`, collect account field from the new table

* fix(jira): fix testes

* fix(jira): fix `reporter` field in issue change log items (#7726) (#7728)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(jira): fix account id when there is no from/to values (#7734) (#7735)

Co-authored-by: Lynwee <linwei.hou@merico.dev>

* fix(jira): fix OriginalToValue in issue_changelogs (#7737) (#7738)

Co-authored-by: Lynwee <1507509064@qq.com>

* fix(jira): don't use `tmpFromAccountId` and `tmpToAccountId` in issue changelogs

* fix(jira): remove all usage of `tmpFromAccountId` and `tmpToAccountId`

* refactor(jira): remove some codes

* fix(jira): fix e2e test

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Klesh Wong <zhenmian.huang@merico.dev>
Co-authored-by: abeizn <zikuan.an@merico.dev>
Co-authored-by: 青湛 <0x1304570@gmail.com>
Co-authored-by: Wanis Fahmy <51237742+wanisfahmyDE@users.noreply.github.com>
Co-authored-by: Leric Zhang <lianghui.zhang@merico.dev>
Co-authored-by: Sayeed M Hussain <sayeedhussain19@gmail.com>
Co-authored-by: houlinwei <timeror@gmail.com>
Co-authored-by: long2ice <jinlong.peng@merico.dev>
  • Loading branch information
10 people authored Jul 16, 2024
1 parent 9dbff0d commit 76cfd3d
Show file tree
Hide file tree
Showing 7 changed files with 68 additions and 54 deletions.
1 change: 1 addition & 0 deletions backend/plugins/jira/e2e/changelog_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ func TestIssueChangelogDataFlow(t *testing.T) {
}
// import raw data table
dataflowTester.ImportCsvIntoRawTable("./raw_tables/_raw_jira_api_issue_changelogs.csv", "_raw_jira_api_issue_changelogs")
dataflowTester.ImportCsvIntoTabler("./snapshot_tables/_tool_jira_issue_fields.csv", &models.JiraIssueField{})
dataflowTester.FlushTabler(&models.JiraIssueChangelogs{})
dataflowTester.FlushTabler(&models.JiraIssueChangelogItems{})
dataflowTester.FlushTabler(&models.JiraAccount{})
Expand Down
14 changes: 3 additions & 11 deletions backend/plugins/jira/tasks/apiv2models/changelog.go
Original file line number Diff line number Diff line change
Expand Up @@ -76,23 +76,15 @@ func (c ChangelogItem) ToToolLayer(connectionId, changelogId uint64) *models.Jir
return item
}

func (c ChangelogItem) ExtractUser(connectionId uint64) []*models.JiraAccount {
func (c ChangelogItem) ExtractUser(connectionId uint64, userFieldMaps map[string]struct{}) []*models.JiraAccount {
var result []*models.JiraAccount
// if `tmpFromAccountId` or `tmpToAccountId` is not empty, then this change log item stands for changes about accounts.
if c.TmpFromAccountId != "" {
// User `from` firstly
_, ok := userFieldMaps[c.Field]
if c.Field == "assignee" || c.Field == "reporter" || ok {
if c.FromValue != "" {
result = append(result, &models.JiraAccount{ConnectionId: connectionId, AccountId: c.FromValue})
} else {
result = append(result, &models.JiraAccount{ConnectionId: connectionId, AccountId: c.TmpFromAccountId})
}
}
if c.TmpToAccountId != "" {
// User `to` firstly
if c.ToValue != "" {
result = append(result, &models.JiraAccount{ConnectionId: connectionId, AccountId: c.ToValue})
} else {
result = append(result, &models.JiraAccount{ConnectionId: connectionId, AccountId: c.TmpToAccountId})
}
}
return result
Expand Down
4 changes: 2 additions & 2 deletions backend/plugins/jira/tasks/apiv2models/issue.go
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ func (i *Issue) SetAllFields(raw json.RawMessage) errors.Error {
return nil
}

func (i Issue) ExtractEntities(connectionId uint64) ([]uint64, *models.JiraIssue, []*models.JiraIssueComment, []*models.JiraWorklog, []*models.JiraIssueChangelogs, []*models.JiraIssueChangelogItems, []*models.JiraAccount) {
func (i Issue) ExtractEntities(connectionId uint64, userFieldMaps map[string]struct{}) ([]uint64, *models.JiraIssue, []*models.JiraIssueComment, []*models.JiraWorklog, []*models.JiraIssueChangelogs, []*models.JiraIssueChangelogItems, []*models.JiraAccount) {
issue := i.toToolLayer(connectionId)
var comments []*models.JiraIssueComment
var worklogs []*models.JiraWorklog
Expand Down Expand Up @@ -338,7 +338,7 @@ func (i Issue) ExtractEntities(connectionId uint64) ([]uint64, *models.JiraIssue
}
for _, item := range changelog.Items {
changelogItems = append(changelogItems, item.ToToolLayer(connectionId, changelog.ID))
users = append(users, item.ExtractUser(connectionId)...)
users = append(users, item.ExtractUser(connectionId, userFieldMaps)...)
}
}
}
Expand Down
42 changes: 41 additions & 1 deletion backend/plugins/jira/tasks/epic_extractor.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,12 @@ limitations under the License.
package tasks

import (
"github.com/apache/incubator-devlake/core/dal"
"github.com/apache/incubator-devlake/core/errors"
"github.com/apache/incubator-devlake/core/log"
"github.com/apache/incubator-devlake/core/plugin"
"github.com/apache/incubator-devlake/helpers/pluginhelper/api"
"github.com/apache/incubator-devlake/plugins/jira/models"
)

var _ plugin.SubTaskEntryPoint = ExtractEpics
Expand All @@ -44,6 +47,10 @@ func ExtractEpics(taskCtx plugin.SubTaskContext) errors.Error {
if err != nil {
return err
}
userFieldMap, err := getUserFieldMap(db, connectionId, logger)
if err != nil {
return err
}
extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
RawDataSubTaskArgs: api.RawDataSubTaskArgs{
Ctx: taskCtx,
Expand All @@ -54,11 +61,44 @@ func ExtractEpics(taskCtx plugin.SubTaskContext) errors.Error {
Table: RAW_EPIC_TABLE,
},
Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
return extractIssues(data, mappings, row)
return extractIssues(data, mappings, row, userFieldMap)
},
})
if err != nil {
return err
}
return extractor.Execute()
}

func getIssueFieldMap(db dal.Dal, connectionId uint64, logger log.Logger) (map[string]models.JiraIssueField, errors.Error) {
var allIssueFields []models.JiraIssueField
if err := db.All(&allIssueFields, dal.Where("connection_id = ?", connectionId)); err != nil {
return nil, err
}
issueFieldMap := make(map[string]models.JiraIssueField)
for _, v := range allIssueFields {
if _, ok := issueFieldMap[v.Name]; ok {
logger.Warn(nil, "filed name %s is duplicated", v.Name)
if v.SchemaType == "user" {
issueFieldMap[v.Name] = v
}
} else {
issueFieldMap[v.Name] = v
}
}
return issueFieldMap, nil
}

func getUserFieldMap(db dal.Dal, connectionId uint64, logger log.Logger) (map[string]struct{}, errors.Error) {
userFieldMap := make(map[string]struct{})
issueFieldMap, err := getIssueFieldMap(db, connectionId, logger)
if err != nil {
return nil, err
}
for filedName, issueField := range issueFieldMap {
if issueField.SchemaType == "user" {
userFieldMap[filedName] = struct{}{}
}
}
return userFieldMap, nil
}
45 changes: 9 additions & 36 deletions backend/plugins/jira/tasks/issue_changelog_convertor.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,17 +89,10 @@ func ConvertIssueChangelogs(taskCtx plugin.SubTaskContext) errors.Error {
}
defer cursor.Close()

var allIssueFields []models.JiraIssueField
if err := db.All(&allIssueFields, dal.Where("connection_id = ?", connectionId)); err != nil {
issueFieldMap, err := getIssueFieldMap(db, connectionId, logger)
if err != nil {
return err
}
issueFieldMap := make(map[string]models.JiraIssueField)
for _, v := range allIssueFields {
if _, ok := issueFieldMap[v.Name]; ok {
logger.Warn(nil, "filed name %s is duplicated", v.Name)
}
issueFieldMap[v.Name] = v
}

issueIdGenerator := didgen.NewDomainIdGenerator(&models.JiraIssue{})
sprintIdGenerator := didgen.NewDomainIdGenerator(&models.JiraSprint{})
Expand Down Expand Up @@ -160,16 +153,15 @@ func ConvertIssueChangelogs(taskCtx plugin.SubTaskContext) errors.Error {
changelog.ToValue = getStdStatus(toStatus.StatusCategory)
}
default:
fromAccountId := tryToResolveAccountIdFromAccountLikeField(row.Field, row.TmpFromAccountId, row.FromValue, issueFieldMap)
if fromAccountId != "" {
changelog.OriginalFromValue = accountIdGen.Generate(connectionId, fromAccountId)
}
toAccountId := tryToResolveAccountIdFromAccountLikeField(row.Field, row.TmpToAccountId, row.ToValue, issueFieldMap)
if toAccountId != "" {
changelog.OriginalToValue = accountIdGen.Generate(connectionId, toAccountId)
if v, ok := issueFieldMap[row.FieldId]; ok && v.SchemaType == "user" {
if row.FromValue != "" {
changelog.OriginalFromValue = accountIdGen.Generate(connectionId, row.FromValue)
}
if row.ToValue != "" {
changelog.OriginalToValue = accountIdGen.Generate(connectionId, row.ToValue)
}
}
}

return []interface{}{changelog}, nil
},
})
Expand All @@ -181,25 +173,6 @@ func ConvertIssueChangelogs(taskCtx plugin.SubTaskContext) errors.Error {
return converter.Execute()
}

func tryToResolveAccountIdFromAccountLikeField(fieldName string, tmpAccountId string, fromOrToValue string, issueFieldMap map[string]models.JiraIssueField) string {
if tmpAccountId != "" {
// process other account-like fields, it works on jira9 and jira cloud.
if fromOrToValue != "" {
return fromOrToValue
} else {
return tmpAccountId
}
} else {
// it works on jira8
// notice: field name is not unique, but we cannot fetch field id here.
if v, ok := issueFieldMap[fieldName]; ok && v.SchemaType == "user" {
// field type is account
return fromOrToValue
}
}
return ""
}

func convertIds(ids string, connectionId uint64, sprintIdGenerator *didgen.DomainIdGenerator) (string, errors.Error) {
ss := strings.Split(ids, ",")
var resultSlice []string
Expand Down
6 changes: 5 additions & 1 deletion backend/plugins/jira/tasks/issue_changelog_extractor.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,10 @@ func ExtractIssueChangelogs(taskCtx plugin.SubTaskContext) errors.Error {
return nil
}
connectionId := data.Options.ConnectionId
userFieldMap, err := getUserFieldMap(taskCtx.GetDal(), connectionId, taskCtx.GetLogger())
if err != nil {
return err
}
extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
RawDataSubTaskArgs: api.RawDataSubTaskArgs{
Ctx: taskCtx,
Expand Down Expand Up @@ -76,7 +80,7 @@ func ExtractIssueChangelogs(taskCtx plugin.SubTaskContext) errors.Error {
// collect changelog_items
for _, item := range changelog.Items {
result = append(result, item.ToToolLayer(connectionId, changelog.ID))
extractedUsersFromChangelogItem := item.ExtractUser(connectionId)
extractedUsersFromChangelogItem := item.ExtractUser(connectionId, userFieldMap)
for _, u := range extractedUsersFromChangelogItem {
if u != nil && u.AccountId != "" {
result = append(result, u)
Expand Down
10 changes: 7 additions & 3 deletions backend/plugins/jira/tasks/issue_extractor.go
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,10 @@ func ExtractIssues(taskCtx plugin.SubTaskContext) errors.Error {
if err != nil {
return err
}
userFieldMap, err := getUserFieldMap(db, connectionId, logger)
if err != nil {
return err
}
extractor, err := api.NewApiExtractor(api.ApiExtractorArgs{
RawDataSubTaskArgs: api.RawDataSubTaskArgs{
Ctx: taskCtx,
Expand All @@ -75,7 +79,7 @@ func ExtractIssues(taskCtx plugin.SubTaskContext) errors.Error {
Table: RAW_ISSUE_TABLE,
},
Extract: func(row *api.RawData) ([]interface{}, errors.Error) {
return extractIssues(data, mappings, row)
return extractIssues(data, mappings, row, userFieldMap)
},
})
if err != nil {
Expand All @@ -84,7 +88,7 @@ func ExtractIssues(taskCtx plugin.SubTaskContext) errors.Error {
return extractor.Execute()
}

func extractIssues(data *JiraTaskData, mappings *typeMappings, row *api.RawData) ([]interface{}, errors.Error) {
func extractIssues(data *JiraTaskData, mappings *typeMappings, row *api.RawData, userFieldMaps map[string]struct{}) ([]interface{}, errors.Error) {
var apiIssue apiv2models.Issue
err := errors.Convert(json.Unmarshal(row.Data, &apiIssue))
if err != nil {
Expand All @@ -99,7 +103,7 @@ func extractIssues(data *JiraTaskData, mappings *typeMappings, row *api.RawData)
if apiIssue.Fields.Created == nil {
return results, nil
}
sprints, issue, comments, worklogs, changelogs, changelogItems, users := apiIssue.ExtractEntities(data.Options.ConnectionId)
sprints, issue, comments, worklogs, changelogs, changelogItems, users := apiIssue.ExtractEntities(data.Options.ConnectionId, userFieldMaps)
for _, sprintId := range sprints {
sprintIssue := &models.JiraSprintIssue{
ConnectionId: data.Options.ConnectionId,
Expand Down

0 comments on commit 76cfd3d

Please sign in to comment.