feat(backend): fetch the first page of outbox when resolving Person #11130

saschanaz · 2023-07-06T01:16:33Z

What

Fixes #9652

Why

ユーザーを最初照会した場合にそのタイムラインが空っぽに見えるのを防止

Additional info (optional)

Checklist

Read the contribution guide
Test working in a local environment
(If needed) Add story of storybook
(If needed) Update CHANGELOG.md
(If possible) Add tests

codecov · 2023-07-06T01:17:56Z

Codecov Report

Patch coverage has no change and project coverage change: -0.15% ⚠️

Comparison is base (814e284) 78.88% compared to head (a926e84) 78.74%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #11130      +/-   ##
===========================================
- Coverage    78.88%   78.74%   -0.15%     
===========================================
  Files          925      174     -751     
  Lines        97589    22783   -74806     
  Branches      7746      511    -7235     
===========================================
- Hits         76986    17940   -59046     
+ Misses       20603     4843   -15760

see 751 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

syuilo · 2023-07-06T04:28:03Z

packages/backend/src/core/activitypub/models/ApPersonService.ts

+			collection;
+
+		// Perform activity but only the first 100 ones
+		await this.apInboxService.performActivity(user, firstPage, 100);


100はちょっと多い気がしたけどどうだろう

50? 20? 適当に100にしましたが多すぎるかも

100 でいいんじゃね

~~30ぐらいでいいんじゃない？~~

15でもいいかも

15にしようぜ

Misskeyのoutbox limitが20ですのでせめて20はしたいです

misskey/packages/backend/src/server/ActivityPubServerService.ts

Line 360 in eacc90d

const limit = 20;

tamaina · 2023-07-07T06:24:06Z

なんかsignedGetでCPU使用率がめちゃくちゃやばい

tamaina · 2023-07-07T06:28:34Z

misskey/packages/backend/src/core/activitypub/ApInboxService.ts

Line 93 in 71d7467

const act = await resolver.resolve(item);

resolver.resolveをアイテム分だけ呼び出してるけどどうにか1回のリクエストで取得できないの

tamaina · 2023-07-07T06:38:17Z

1回ならまあなんとかなる

tamaina · 2023-07-07T07:42:03Z

これ、#11161 は変わらないけど

{ message: "Duplicated note", code: "duplicated", id: "d301f236-d4c1-4f1c-8007-b8985e9a6d93" }

にならない？

tamaina · 2023-07-07T12:48:54Z

outbox読むのにsignatureって必要なのかしら？→フォロワー向け投稿周りで必要なのか

saschanaz · 2023-07-07T12:50:14Z

どういう条件でそうなるのでしょうか。同じユーザーは再び照会しても二つ目はローカルからロードするはずですので、同じactivityは再びperformされないものなのではないかと

saschanaz · 2023-07-07T12:51:08Z

outbox読むのにsignatureって必要なのかしら？→フォロワー向け投稿周りで必要なのか

あとsignature必修なサーバーとか（フォロワーは最初照会の時は関係ないはず）

packages/backend/test/unit/activitypub.ts

tamaina · 2023-07-07T13:00:58Z

エラーの話は手違いの可能性が高い、すみませんでした

tamaina · 2023-07-07T13:01:38Z

とにかくsignedGetでCPU使用率がどうしようもなくなるのでsignedGetを1回きりにする必要がある

tamaina · 2023-07-07T13:44:13Z

エラーはこのPRのせいだった

どういう条件でそうなるのでしょうか。

ap/showで、未知のユーザーの未知のノートを取得するとこうなる。

ap/showで色々呼んでいる途中の

misskey/packages/backend/src/core/activitypub/models/ApNoteService.ts

Line 147 in 71d7467

    
           const actor = await this.apPersonService.resolvePerson(getOneApId(note.attributedTo!), resolver) as RemoteUser;

が問題で、これの後に作業中のノートをthis.noteCreateService.createで改めて挿入してしまうためコンフリクトが発生する

saschanaz · 2023-07-08T13:54:45Z

ap/showで色々呼んでいる途中の

misskey/packages/backend/src/core/activitypub/models/ApNoteService.ts

Line 147 in 71d7467

const actor = await this.apPersonService.resolvePerson(getOneApId(note.attributedTo!), resolver) as RemoteUser;

が問題で、これの後に作業中のノートをthis.noteCreateService.createで改めて挿入してしまうためコンフリクトが発生する

この問題、featuredノートをfetchした場合はどうなるんでしょう

Note Aをfetch
そのactorをfetch
featuredノートをfetch (note Aが含まれている)
Note Aをinsert <- ?

saschanaz · 2023-07-08T14:03:46Z

とにかくsignedGetでCPU使用率がどうしようもなくなるのでsignedGetを1回きりにする必要がある

updateOutboxFirstPage自体は最大2回signedGetしますので、その場合はそれに含まれるノートに新しいものが沢山入ってるのかも

tamaina · 2023-07-14T03:38:42Z

conflicted

saschanaz · 2023-07-15T02:14:39Z

(#11130 (comment) の対応をしてなかったので）

tamaina · 2023-07-15T13:52:37Z

うーーん、リプライとかを永久に引っ張ってきておりデータベースが肥大化してきている

performActivityでフェッチする数をdefault.ymlで調整できるようにする必要がある（0で無効化）

saschanaz · 2023-07-15T14:11:15Z

resolvePersonじゃなくtimeline APIとかでtimelineが空いてたらfetchするようにしたらいいかも

元々リモートでも空いてるアカウントとなんとか区別しないと相手サーバーにfetchし続けるようになるのでそれは困難

tamaina · 2023-07-15T14:32:50Z

とりあえず設定できるようにした

saschanaz · 2023-07-15T14:33:46Z

placeholderノート作りたい（実際必要な場合にだけfetchするように）

tamaina · 2023-07-15T14:37:03Z

placeholderにしておいてpack(?)が呼ばれたらresolveするようにということかしら

saschanaz · 2023-07-15T14:41:53Z

そうやってfetch回数をへらしたり、dropboxのplaceholderファイルのように古くて使わないリモートのノートをplaceholderに変えて容量を確報するとか、使い道がいろいろありそうです

tamaina · 2023-08-24T06:03:50Z

conflicted

(p1.a9z.devへマージしているため解消する必要がある)

tamaina · 2023-08-24T06:36:41Z

解消

feat(backend): fetch the first page of outbox when resolving Person

2e1de4f

github-actions bot added packages/backend Server side specific issue/PR packages/backend:test labels Jul 6, 2023

github-actions bot requested review from syuilo and tamaina July 6, 2023 01:16

saschanaz added 2 commits July 6, 2023 03:38

Merge remote-tracking branch 'origin/develop' into fetch-outbox

630e97b

Update CHANGELOG.md

a1388a8

syuilo reviewed Jul 6, 2023

View reviewed changes

tamaina added 2 commits July 7, 2023 05:02

100 -> 15

5077df2

Merge branch 'develop' into fetch-outbox

71d7467

tamaina marked this pull request as draft July 7, 2023 06:38

saschanaz commented Jul 7, 2023

View reviewed changes

packages/backend/test/unit/activitypub.ts Outdated Show resolved Hide resolved

saschanaz added 5 commits July 8, 2023 15:13

remove extra resolving in tests

7bf318a

adjust tests

6087d02

typo

45d0b46

Merge remote-tracking branch 'origin/develop' into fetch-outbox

aa78c29

reformat

a74af07

saschanaz added 6 commits July 9, 2023 22:36

Update activitypub.ts

26040c2

Merge remote-tracking branch 'origin/develop' into fetch-outbox

f34f0df

Update activitypub.ts

b93046c

Update activitypub.ts

70bb9a4

Update activitypub.ts

ca0c673

perform only create activities

08e2b6e

saschanaz marked this pull request as ready for review July 9, 2023 21:34

github-actions bot requested a review from syuilo July 9, 2023 21:34

Merge branch 'develop' into fetch-outbox

5a0d7d4

saschanaz marked this pull request as draft July 15, 2023 02:14

tamaina added 3 commits July 15, 2023 14:04

Merge branch 'develop' into fetch-outbox

6b26ce3

fix type error

da0804e

?

76def00

configable

60fd848

misskey-dev deleted a comment from wederuten6 Jul 17, 2023

Merge branch 'develop' into fetch-outbox

ff4b3d2

saschanaz mentioned this pull request Aug 5, 2023

ノートを照会してfeatured noteだったら照会失敗する #11471

Closed

Merge branch 'develop' into fetch-outbox

888cd2e

tamaina added 2 commits September 11, 2023 06:32

Merge branch 'develop' into fetch-outbox

df63665

Merge branch 'develop' into fetch-outbox

a926e84

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(backend): fetch the first page of outbox when resolving Person #11130

feat(backend): fetch the first page of outbox when resolving Person #11130

saschanaz commented Jul 6, 2023 •

edited

Loading

codecov bot commented Jul 6, 2023 •

edited

Loading

syuilo Jul 6, 2023

saschanaz Jul 6, 2023

acid-chicken Jul 6, 2023

tamaina Jul 7, 2023 •

edited

Loading

tamaina Jul 7, 2023

tamaina Jul 7, 2023

saschanaz Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023 •

edited

Loading

saschanaz commented Jul 7, 2023

saschanaz commented Jul 7, 2023 •

edited

Loading

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023 •

edited

Loading

saschanaz commented Jul 8, 2023

saschanaz commented Jul 8, 2023 •

edited

Loading

tamaina commented Jul 14, 2023

saschanaz commented Jul 15, 2023

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023 •

edited

Loading

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023

tamaina commented Aug 24, 2023

tamaina commented Aug 24, 2023

feat(backend): fetch the first page of outbox when resolving Person #11130

Are you sure you want to change the base?

feat(backend): fetch the first page of outbox when resolving Person #11130

Conversation

saschanaz commented Jul 6, 2023 • edited Loading

What

Why

Additional info (optional)

Checklist

codecov bot commented Jul 6, 2023 • edited Loading

Codecov Report

syuilo Jul 6, 2023

Choose a reason for hiding this comment

saschanaz Jul 6, 2023

Choose a reason for hiding this comment

acid-chicken Jul 6, 2023

Choose a reason for hiding this comment

tamaina Jul 7, 2023 • edited Loading

Choose a reason for hiding this comment

tamaina Jul 7, 2023

Choose a reason for hiding this comment

tamaina Jul 7, 2023

Choose a reason for hiding this comment

saschanaz Jul 7, 2023

Choose a reason for hiding this comment

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023 • edited Loading

saschanaz commented Jul 7, 2023

saschanaz commented Jul 7, 2023 • edited Loading

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023

tamaina commented Jul 7, 2023 • edited Loading

saschanaz commented Jul 8, 2023

saschanaz commented Jul 8, 2023 • edited Loading

tamaina commented Jul 14, 2023

saschanaz commented Jul 15, 2023

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023 • edited Loading

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023

tamaina commented Jul 15, 2023

saschanaz commented Jul 15, 2023

tamaina commented Aug 24, 2023

tamaina commented Aug 24, 2023

saschanaz commented Jul 6, 2023 •

edited

Loading

codecov bot commented Jul 6, 2023 •

edited

Loading

tamaina Jul 7, 2023 •

edited

Loading

tamaina commented Jul 7, 2023 •

edited

Loading

saschanaz commented Jul 7, 2023 •

edited

Loading

tamaina commented Jul 7, 2023 •

edited

Loading

saschanaz commented Jul 8, 2023 •

edited

Loading

saschanaz commented Jul 15, 2023 •

edited

Loading