Comments are not synced to my lemmy instance

[ravermeister]

Found a bug? Please fill out the sections below. 👍

For front end issues, use lemmy-ui

Issue Summary

I've created a Post on lemmy.ml from the Account of my own instance
Link from my instance: https://lemmy.rimkus.it/post/18380
Link from lemmy.ml: https://lemmy.ml/post/541198

All Comments from Users of lemmy.ml are not synced. Doing the Troubleshooting advice from here:
https://join-lemmy.org/docs/en/administration/troubleshooting.html#other-instances-cant-fetch-local-objects-community-post-etc
for e.g. the comment https://lemmy.ml/post/541198/comment/313692 for which the canonical url is: https://lemmy.ml/comment/313692

curl -H "Accept: application/activity+json" https://lemmy.ml/comment/313692
NotFound

'NotFound' is returned. I think this could be the cause that the comments are not synced to my instance.
But why are the comments not accessible by their direct url's?

I'm using the latest productive lemmy 0.16.7 via docker by the way

Thanks and kind regards
Jonny

[Nutomic]

Did you subscribe to the community before these comments were created? And did were previous posts/comments federated normally?

I also dont understand why the curl command fails, seems very strange. Here is how it looks in the database, so the comment is clearly not deleted and ap_id is correct.

# select * from comment where id=313692;
id   | creator_id | post_id | parent_id |                                                   content                                                  | removed | read |         published          | updated | deleted |              ap_id              | local
--------+------------+---------+-----------+-------------------------------------------------------------------------------------------------------------+---------+------+----------------------------+---------+---------+---------------------------------+-------
313692 |         34 |  541198 |           | Yep this is a feature I haven't implemented yet. Open up an issue on the github if there isn't one already.| f       | f    | 2022-10-16 18:57:09.549447 |         | f       | https://lemmy.ml/comment/313692 | t

The comment https://lemmy.ml/comment/311538 has the same problem, but everything else works fine.

[ravermeister]

Hi, both of the Questions yes..
I was subscribed to the community before these comments were created (these comments are answers of my own post in this community :) )
and comments of posts inside this community which are not created by me are synced correctly e.g.:
https://lemmy.rimkus.it/post/11341

[Nutomic]

It looks like there is a bug with those specific comments. We need a way to reproduce it in order to fix it.

[ravermeister]

Should I create a new post from within my instance in a specific community where we could investigate?

[Nutomic]

Yes, you can use https://lemmy.ml/c/test or https://voyager.lemmy.ml/ for testing.

[ravermeister]

Done: https://lemmy.rimkus.it/post/19785 only difference in this post is, that it has an URL (link to this issue).
If we can not reproduce with this post I'll try with anotherone without an URL, to have more exact matching of the original Issue.

[Nutomic]

I made a comment there, and lemmy-ui showed an error "Unknown" without further details. Same happens when i edit it, and the comment doesnt get federated.

Here is the error log from server:

lemmy_1     | 2022-10-21T15:04:50.732702Z ERROR Websocket Request{trace_id=00000000000000000000000000000000}: lemmy_websocket::handlers: Error during message handling NotFound
lemmy_1     |    0: lemmy_apub::mentions::get_comment_parent_creator
lemmy_1     |              at crates/apub/src/mentions.rs:101
lemmy_1     |    1: lemmy_apub::mentions::collect_non_local_mentions
lemmy_1     |            with request_counter=0
lemmy_1     |              at crates/apub/src/mentions.rs:45
lemmy_1     |    2: lemmy_apub::objects::comment::into_apub
lemmy_1     |              at crates/apub/src/objects/comment.rs:90
lemmy_1     |    3: lemmy_apub::activities::create_or_update::comment::send
lemmy_1     |            with request_counter=0
lemmy_1     |              at crates/apub/src/activities/create_or_update/comment.rs:36
lemmy_1     |    4: lemmy_api_crud::comment::update::perform
lemmy_1     |            with self=EditComment { content: "test 7", comment_id: CommentId(317540), form_id: Some("comment-form-3IO2NqM9zVvtFA3jUEV5"), auth: Sensitive }
lemmy_1     |              at crates/api_crud/src/comment/update.rs:35
lemmy_1     |    5: lemmy_websocket::handlers::Websocket Request
lemmy_1     |            with trace_id=00000000000000000000000000000000
lemmy_1     |              at crates/websocket/src/handlers.rs:68

[dginovker]

Also seeing that

Amazing feature idea though!

[ravermeister]

Please tell me if I can do anything on my side to help investigate. e.g is it worth, and if so, is there a way to check if my instance is reachable via websocket?

[ravermeister]

I checked if the websocket connection of lemmy.ml and lemmy.rimkus.it is generally working. Seems good, I can connect via wss://lemmy.rimkus.it/api/v3/ws and wss://lemmy.ml/api/v3/ws. So I'm out of Ideas, if I should check anything else, please let me know.

[Nutomic]

It has nothing to do with websocket, but seems to be some weird database problem on lemmy.ml. Could you also make posts in https://voyager.lemmy.ml/c/main and https://ds9.lemmy.ml/c/main to see what happens there? Im also curious if the problem happens with other user accounts from your instance.

[ravermeister]

I added a post on voyager.lemmy.ml and ds9.lemmy.ml
unfortunately I do no not have other Users on my instance to test. I could create a test user if we need one.
Please let me know, then I'll do the test with this user again.

[Nutomic]

Both of these work fine, so it really seems like a problem with lemmy.ml. Still no clue what could cause this. Could you create a test user on your instance, and post on lemmy.ml with that?

[ravermeister]

I've created a new user and made a post in lemmy.ml voyager.lemmy.ml and ds9.lemmy.ml

[Nutomic]

I just found the problem, your user was marked as deleted on lemmy.ml. Did you previously delete the account, or do anything unusual with it?

@dessalines It seems like lemmy-ui doesnt show any indication that a user was deleted, not even on the profile page.

[ravermeister]

The only thing that is special on that user is, that I completely removed my instance (due to messed up database after the postgreSQL version upgrade) and re-created from Scratch (under the same Domain). And added the exact Same Username+email combination.
Maybe that lead to weird behaviour. Could you please remove the the mark as deleted for my user on lemmy?

[dessalines]

@Nutomic wouldn't lemmy update that user after a day, in the same way that it refreshes its pubkey?

[Nutomic]

It doesnt update the deleted field. Should that be changed? Im not sure.

https://github.com/LemmyNet/lemmy/blob/main/crates/apub/src/objects/person.rs#L163

[dessalines]

Ya it should.