[V3] The trivia list repository and namespace packages

[Tobotimus]

Alright, I figure I should write up my thoughts on this topic here to get a proper discussion going.

Currently, for those who are not aware, the trivia lists (YAML files containing question and answers for trivia) are located in the Red-Trivia repo. This is its own (PyPI-hosted) python package, and it is listed as a dependency of Red-DiscordBot. However, in order to make the trivia repo install in the same site-packages folder of a user's python environment (and to keep import names uniform), we install the package inside a namespace package, redbot.ext.

Namespace packages have a few rules and there are multiple ways to set them up, the recommended way being to make them just a plain folder (without an __init__.py). This is how redbot.ext is currently set up, however it is still wrong. Namespace packages cannot be beneath normal python packages, in our case the redbot package. This is because setuptools will not only declare redbot.ext as a namespace package, but also any the parent packages of that name. So, redbot.ext and redbot get declared as namespace packages.

redbot should not be declared as a namespace package, because this means redbot/__init__.py will be executed every time setuptools or pkg_resources is imported from any application inside the python envrionment. Inside redbot/__init__.py exists the line import discord. This can throw an error in many ways, particularly if there is something incomplete with the python environment's dependencies (perhaps discord.py isn't installed, or one of its sub-dependencies). This is why often whenever we are missing a dependency of some sort, our pip constantly spews meaningless tracebacks and the only solution is to nuke the python environment and start again.

Alright, so they are mostly the issues with our current use of namespace packages (sorry for the wall of text). Now, I wanna talk about solving this.

Do we need namespace packages?

After all of the annoyance they've been, their unpythonic nature and even a comment from Donald Stufft describing them as "broken" (although I've admittedly taken this out of context), I'm leaning towards no. The trivia repo is not worth this hassle. My proposed alternative would simply be to have Red-Trivia just install to a red_trivia package totally separate from redbot.

Do we need Trivia in a separate repo?

Yes, I'm opening this question back up again. I wasn't a core developer when the decision was made to separate the lists, and whilst I'm not just trying to uproot far-gone decisions, I do sometimes question if our current setup is worth the hassle. Here are some reasons why:

Separate release cycles and versions

I don't see a real advantage in separating the release cycle of the trivia lists. If anything, I see it as a disadvantage because:

• When users update the bot, Red-Trivia won't be updated (even if there is a new version) unless Red-DiscordBot pins the latest version. That means every time Red-Trivia does a release, we have to manually bump the version on Red-DiscordBot, or else the vast majority of users will never see updates to Red-Trivia (unless they manually update it). We shouldn't need to manually bump the version of a tiny and internally-developed dependency, especially if it is 99% data and 1% code.
• More releases is simply more to manage for developers. It's much easier for us to just stick some notes in Red's changelog release to say the trivia lists have been updated.

Having its own PyPI package

Red-Trivia is 756KB of YAML and 5 lines of python code, the code which trivially returns the paths to the lists. I don't think this is very deserving of a PyPI package in retrospect. I don't see a problem with simply adding one extra line of code in the trivia cog to get the list of paths to the YAML files, and locate the YAML files somewhere appropriate in the redbot repo.

Neglect

I often miss things going on at the trivia repo, or forget about them, because I'm constantly pouring over the issues here on this repo. The separate repo doesn't get much attention and can be neglected at times. This might be slowing down progress on trivia - I know it takes a while for PR's to get merged over there sometimes, let alone wait for a new release and for the version pin to be updated here.

Locating the lists in this repository

I think redbot/cogs/trivia/data/ would be an appropriate place to locate them, since we already recommend a data/ subdirectory for managing bundled data with 3rd party cogs - this is essentially following that convention. The only difference is there is no need for bundled data to be copied here, since it's a core cog which stays where it is and the core lists should be shared across instances. This approach is really simple, keeps everything structured categorically (all core trivia things together) and keeps us sane. Simple is better than complex, and the above essay could be swept aside if we simply put up with some YAML files being kept in this repo.

[Tobotimus]

Closing this now #2028 is merged. I understand this conversation might not be over, however.