Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

List used as a Data Source cannot be deleted despite no longer a DS for any list #1660

Closed
bwiderski-HBS opened this issue Apr 21, 2023 · 34 comments · Fixed by #1662
Closed

List used as a Data Source cannot be deleted despite no longer a DS for any list #1660

bwiderski-HBS opened this issue Apr 21, 2023 · 34 comments · Fixed by #1662
Assignees
@ikedas
Labels
bug

Comments

@bwiderski-HBS
Copy link

List it_privileged_owners was used as an Owner Data Source for multiple other lists but has been superseded so needs to be closed.

After removing it as a DS from all lists, plus manually synchronizing those lists' owners, I still cannot close it because its admin page still says it is included in other lists.

Pressing the button "View lists including it_privileged_owners" takes me to https://lists.hbs.edu/lists/including_lists/it_privileged_owners which only displays a blue box saying "No Lists".

I restarted sympa and wwsympa - no change.
15 hours after first seeing this behavior it still exists.

Version 6.2.70
Screen shots attached.

Screenshots

sympa 01
sympa 02
@ikedas ikedas added the bug label Apr 21, 2023
@ikedas ikedas self-assigned this Apr 21, 2023
@bwiderski-HBS
Copy link
Author

FYI I just had a second occurrence - steps to have happen:

  1. Went to Admin of tss_privileged_owners - said it was included in one list: mba_auth
  2. Went to mba_auth, edited owner data source by
    a. changed data source name from "tss_privileged_owners" to "tier_2_privileged_owners"
    b. Went to Owners page and synchronized data sources: old tss_privileged_owners content was removed, new tier_2_privileged_owners content added.
  3. Returned to tss_privileged_owners Admin - still said it was included in another list.
  4. Clicked button to see where included - again the result was a blue box stating "no lists"

@ikedas
Copy link
Member

ikedas commented Apr 21, 2023

@bwiderski-HBS , what is the database you are using?

@ikedas
Copy link
Member

ikedas commented Apr 22, 2023

@bwiderski-HBS , additionally, could you please show the system log during the steps above?

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented Apr 22, 2023 via email

@bwiderski-HBS
Copy link
Author

042123 syslogs re sympa.txt
Attached is the system log segment from the period of my Sympa activity the afternoon of Friday April 21. During that period I was doing a number of things:

  • Removing list tss_privileged_owners as a data source from lists I wanted to close and syncing (e.g., list test-tss-closed-auth)
  • Renaming the include file tss_privileged_owners to tier_2_privileged_owners in lists I wanted to keep and syncing (e.g., laurencetestlist)

After removing or renaming tss_privileged_owners as a data source from all other lists I wanted to close it - but Sympa still thinks it is in use by other lists, but when asked to identify those it gives the blue box "no lists".

@ikedas
Copy link
Member

ikedas commented Apr 26, 2023 •
edited
Loading

Hi @bwiderski-HBS ,

Could you show us the inclusion settings of the list "mba2024@lists.hbs.edu"?

The log tells that this list is failing inclusion.

Apr 21 13:58:15 rhlists2 task_manager[680148]: err main::#148 > Sympa::Spindle::spin#83 > Sympa::Spindle::ProcessTask::_twist#74 > Sympa::Spindle::ProcessTask::_execute#140 > Sympa::Spindle::ProcessTask::_cmd_process#174 > Sympa::Spindle::ProcessTask::do_sync_include#1347 > Sympa::List::sync_include#4107 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::include::_twist#209 > Sympa::Request::Handler::include::_update_users#377 > Sympa::DataSource::open#113 > Sympa::DataSource::List::_open#60 > Sympa::List::new#137 Missing robot parameter, cannot create list object for mba2024b

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented Apr 26, 2023 via email

@ikedas
Copy link
Member

ikedas commented Apr 26, 2023

@bwiderski-HBS sorry, attachment in the e-mail reply is removed. Please post it on the web.

@bwiderski-HBS
Copy link
Author

mba2024 - inclusions.pdf

@ikedas
Copy link
Member

ikedas commented Apr 26, 2023

Thanks.

The log above tells that the list "mba2024b" does no longer exist.

  1. Open the data source config page of "mba2024" list.
  2. Check the "Delete" checkbox at the right side of the data source (source) "local_list (local_list)" for "mba2024b", then click "Apply modifications" at the bottom of the page.
  3. Wait for a whle (one hour or so).
  4. Check the log and confirm that the err log about "sync_include" is no longer shown.

Then check whether "it_privileged_owners" may be closed.

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented Apr 26, 2023 via email

@bwiderski-HBS
Copy link
Author

042623 syslog re sympa.txt

Today's logs. Still cannot delete it_privileged_owners or tss_privileged_owners.

Note - I'm OOO the rest of this week so will respond to anything 1 May.

@ikedas
Copy link
Member

ikedas commented Apr 27, 2023

You may have to log-in as a listmaster and check the "View lists including it_privileged_owners".

Perhaps the "Visibility of the list (visibility)" setting of these lists prevent them from being displayed. Listmasters can work around this limitation in most cases.

ikedas added a commit to ikedas/sympa that referenced this issue Apr 27, 2023
@ikedas
Listmaster should always view including lists (sympa-community#1660)
6b39480
@ikedas
Copy link
Member

ikedas commented Apr 27, 2023

Oh you are the listmaster @bwiderski-HBS .

Can you apply following modification on web template?

--- a/default/web_tt2/including_lists.tt2
+++ b/default/web_tt2/including_lists.tt2
@@ -8,7 +8,7 @@
         <span class="divider"></span>
         [% SET invisible_count = 0; visible_count = 0 ~%]
         [% FOREACH l = which ~%]
-            [% IF l.value.visible ~%]
+            [% IF l.value.visible || is_listmaster ~%]
                 <div class="item_content">
                     <a class="item_title" href="[% l.value.robot == robot ? l.value.url_rel : l.value.url_abs %]">
                         [%|obfuscate(conf.spam_protection) ~%]

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented Apr 27, 2023 via email

@ikedas
Copy link
Member

ikedas commented Apr 27, 2023 •
edited
Loading

Hi @bwiderski-HBS ,

Please check the manual to know how to customize template.

The colored texts mean that, you create the template file web_tt2/including_list.tt2 replacing the line

            [% IF l.value.visible ~%]

with

            [% IF l.value.visible || is_listmaster ~%]

This will not break Sympa.

@ikedas
Copy link
Member

ikedas commented Apr 28, 2023 •
edited
Loading

Hi @bwiderski-HBS again,

Apr 21 14:25:02 rhlists2 wwsympa[680421]: err main::#1557 > main::do_create_list#9515 > Sympa::Spindle::spin#95 > Sympa::Request::Handler::create_list::_twist#215 > Sympa::List::new#181 > Sympa::List::load#699 > Sympa::List::_load_list_config_file#5044 Unknown key "visibility" in paragraph "available_user_options" in /var/lib/sympa/list_data/lists.hbs.edu/test01-post-edit/config

The config of the list "test01-post-edit" seems broken. Maybe any create list template is broken and the lists created using it is/are also broken.

Please check the log and find out the other broken lists, and if these lists are unnecessary, please close them.

ikedas added a commit to ikedas/sympa that referenced this issue Apr 28, 2023
@ikedas
Listmaster should always view including lists (sympa-community#1660)
3f44dca
@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented Apr 28, 2023 via email

@ikedas
Copy link
Member

ikedas commented Apr 28, 2023

I’ll be on vacation in the next week, so I’ll be less responsive.

@ikedas
Copy link
Member

ikedas commented Apr 29, 2023

@bwiderski-HBS , additionally, please also show us the log for a few minutes including the time when you clicked on the "Admin" link.

ikedas added a commit to ikedas/sympa that referenced this issue Apr 29, 2023
@ikedas
Manually deleted list blocks closure of the list which has been inclu…
b916398 
…ded by the former (sympa-community#1660)
@ikedas ikedas mentioned this issue Apr 29, 2023
Merged
@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented May 1, 2023 •
edited by ikedas
Loading

Replies to your various comments while I was OOO:

  • Sorry, I was only looking at the email fo your comment, not the github entry, so I wasn't seeing the formatting, including color.

  • I made the change in default/web-tt2/including_lists.tt2 and restarted sympa and wwsympa. I am attaching syslogs from 1) around using admin before doing the including_lists edit, and 2) after doing it and restarting. I'm seeing no difference in behavior.

  • Looks like ALL my lists had that 'Unknown key "visibility" in paragraph "available_user_options' on creation. I've been using templates of a format created by the consultant we hired to set up Sympa. All of the templates put "visibility conceal" in a location that makes Sympa read it as a variable pertaining to available_user_options:

my templates:
send no_post_bounce

available_user_options
reception mail
visibility conceal

Default Sympa template html-news-letter:

visibility noconceal

send editorkeyonly

available_user_options 
reception mail,nomail,txt,html

Thus all my lists have "visibility noconceal" since the visibility parameter wasn't seen properly. This error only appeared in logs during list creation, so it seems there is nothing "broken" about the created lists, just that I need to manually set their visibility to "conceal" and fix my templates for new creations.

This doesn't seem to be related to my data sources problem, as the templates are the same on my dev server which does not have the DS problem.
pre-including_list-tt2-change.txt
post-including_list-tt2-edit-and restart.txt

@ikedas
Copy link
Member

ikedas commented May 3, 2023

Hi @bwiderski-HBS ,

Replies to your various comments while I was OOO:

  • Sorry, I was only looking at the email fo your comment, not the github entry, so I wasn't seeing the formatting, including color.

I understand. From now on I'll write without assuming the web.

  • I made the change in default/web-tt2/including_lists.tt2 and restarted sympa and wwsympa. I am attaching syslogs from 1) around using admin before doing the including_lists edit, and 2) after doing it and restarting. I'm seeing no difference in behavior.

These are the logs that I asked for showing us 'the log for a few minutes including the time when you clicked on the "Admin" link'.

Next, please show us the log from when you clicked on the "Admin" link' to when you clicked on the "View lists including ..." and see the text "No lists".

  • Looks like ALL my lists had that 'Unknown key "visibility" in paragraph "available_user_options' on creation. I've been using templates of a format created by the consultant we hired to set up Sympa. All of the templates put "visibility conceal" in a location that makes Sympa read it as a variable pertaining to available_user_options:

my templates: send no_post_bounce

available_user_options
reception mail
visibility conceal

This shiould be as below:

available_user_options
reception mail

visibility conceal

Thus all my lists have "visibility noconceal" since the visibility parameter wasn't seen properly. This error only appeared in logs during list creation, so it seems there is nothing "broken" about the created lists, just that I need to manually set their visibility to "conceal" and fix my templates for new creations.

This doesn't seem to be related to my data sources problem, as the templates are the same on my dev server which does not have the DS problem.

This may not be directly related to the "No lists" issue, but it is not working correctly and should be fixed.

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented May 3, 2023 •
edited
Loading

Yes, I've fixed the incorrect templates.

Log:

  • showed date
  • tail -f /var/log/messages
  • in Sympa UI chose list it_privileged_owners
  • clicked Admin
  • clicked "view lists including"

[sympa@rhlists2 sympa]$ date
Wed May 3 08:56:17 EDT 2023
[sympa@rhlists2 sympa]$ tail -f /var/log/messages
May 3 08:55:06 rhlists2 systemd[1]: pmlogger_check.service: Succeeded.
May 3 08:55:10 rhlists2 systemd[1]: Starting Check and migrate non-primary pmlogger farm instances...
May 3 08:55:10 rhlists2 systemd[1]: Started Check and migrate non-primary pmlogger farm instances.
May 3 08:55:10 rhlists2 systemd[1]: pmlogger_farm_check.service: Succeeded.
May 3 08:55:43 rhlists2 wwsympa[1849747]: info main::do_lists(, ) [robot lists.hbs.edu] [session 60721619194124] [client 192.168.3.1] [user bwiderski@hbs.edu]
May 3 08:55:43 rhlists2 wwsympa[1849747]: info Sympa::WWW::Session::renew() [robot lists.hbs.edu] [session 60721619194124] [client 192.168.3.1] [user bwiderski@hbs.edu] new session 68812186092189
May 3 08:56:00 rhlists2 systemd[1]: Starting system activity accounting tool...
May 3 08:56:00 rhlists2 systemd[1]: sysstat-collect.service: Succeeded.
May 3 08:56:00 rhlists2 systemd[1]: Started system activity accounting tool.
May 3 08:56:19 rhlists2 rsyslogd[1561]: -- MARK --
May 3 08:56:37 rhlists2 wwsympa[1849745]: info main::do_info() [robot lists.hbs.edu] [session 68812186092189] [client 192.168.3.1] [user bwiderski@hbs.edu] [list it_privileged_owners]
May 3 08:56:37 rhlists2 wwsympa[1849745]: info Sympa::WWW::Session::renew() [robot lists.hbs.edu] [session 68812186092189] [client 192.168.3.1] [user bwiderski@hbs.edu] new session 25733895167847
May 3 08:56:40 rhlists2 wwsympa[1849746]: info main::do_admin() [robot lists.hbs.edu] [session 25733895167847] [client 192.168.3.1] [user bwiderski@hbs.edu] [list it_privileged_owners]

@ikedas
Copy link
Member

ikedas commented May 3, 2023

Could you please show the log after you clicked the "Admin" link?

@bwiderski-HBS
Copy link
Author

I did - after I did "Date" I went to the UI, went to the list, clicked Admin, then the view button - this is the entire log from doing those steps immediately after 08:56:17. This is /var/log/messages - is there a different log you want to see?

@ikedas
Copy link
Member

ikedas commented May 4, 2023

Did you see "No lists"?

@bwiderski-HBS
Copy link
Author

bwiderski-HBS commented May 4, 2023 via email

@ikedas
Copy link
Member

ikedas commented May 4, 2023 •
edited
Loading

Can you use psql command? If you can, could you connect to database, run following query and show the result?

SELECT * FROM inclusion_table WHERE source_inclusion = 'it_privileged_owners@lists.hbs.edu';

And, if the result is not empty, do the list names included in it really exist?

@bwiderski-HBS
Copy link
Author

I did this for both it_privileged_owners and tss_privileged_owners. The results for both were not empty, and none of the lists are either active or in "closed".

owner it_privileged_owners@lists.hbs.edu tss-closed-open-dharvill@lists.hbs.edu 1680100563
owner it_privileged_owners@lists.hbs.edu testopenpost@lists.hbs.edu 1680104315
owner it_privileged_owners@lists.hbs.edu testopenpost2@lists.hbs.edu 1680105162
owner it_privileged_owners@lists.hbs.edu test-tss-closed-closed-bw@lists.hbs.edu 1680099594
owner it_privileged_owners@lists.hbs.edu testbrian@lists.hbs.edu 1678467125

owner tss_privileged_owners@lists.hbs.edu tss-closed-open-dharvill@lists.hbs.edu 1680100563
owner tss_privileged_owners@lists.hbs.edu bwmigrationtest_auth@lists.hbs.edu 1648671518
owner tss_privileged_owners@lists.hbs.edu testbrian@lists.hbs.edu 1678467125
owner tss_privileged_owners@lists.hbs.edu test-tss-closed-closed-bw@lists.hbs.edu 1680099594
owner tss_privileged_owners@lists.hbs.edu bwmigrationtest@lists.hbs.edu 1648659218

I have full Postgres database nightly dumps, if seeing one of those would be helpful.

@ikedas
Copy link
Member

ikedas commented May 6, 2023

So those lists do not really exist. Then I don't understand why nothing is showing up in the logs....
Anyway, could you please apply this patch and restart the services to see if it solves the problem?

@bwiderski-HBS
Copy link
Author

Please advise how to "apply this patch" - I see it is changes to /usr/share/sympa/lib/List.pm on my server, but I do not know how to get a copy of the modified List.pm to replace existing. We do not have our Sympa components in Git, and I'm hoping there's an easier way than my manually viming and editing by hand?

@ikedas
Copy link
Member

ikedas commented May 8, 2023

You may use patch(1) command to apply the patch I presented. For more details to use the command please ask your colleagues who have knowleges about software development. Or, please wait for a new release of Sympa in the future which includes this patch.

@bwiderski-HBS
Copy link
Author

I have figured out how to use patch(1) to update List.pm. I am waiting for a scheduled patching window to apply it long enough to remove the problem lists, then will probably revert to the current file version until you issue a new release, since this is our prod instance.

@ikedas
Copy link
Member

ikedas commented Oct 8, 2023

The patch will be included in the next release. At that time, this issue will be closed.

racke added a commit that referenced this issue Dec 7, 2023
@racke
Merge pull request #1662 from ikedas/issue-1660_try-3
8668cb4 
Manually deleted list blocks closure of the list which has been included by the former (#1660)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ikedas ikedas

Labels
bug
Projects
None yet
Milestone
No milestone
2 participants
@ikedas @bwiderski-HBS