Standardize documentation for things other than functions

[NeunEinser]

The current IMP doc only covers documentation for functions.
There are already some extensions to this, for example by Datapack Helper Plus.

I think it would be good to come up with a standard for this before we get more conflicting standards.

Things I think are worth to be documented:

• Scoreboard objectives
• Scoreboard players
• Storages (as in the namespaced ID of a storage)
• Bossbars
• Tags (as in tags inside an Entity's Tags:[] nbt-tag)
• Teams
• All sorts of JSON files

Currently, IMP doc has to be in the first line of a function file. For non-function file declarations that doesn't make sense, of course. There are several ways such definitions could be applied.

Right where you execute a defining command

This would still be a function comment and is basically what dph+ does.

Since for functions the first line has to include the function's name, I think it would make sense for other things to do the same.
To be able to know what is defined, I propose using a prefix to the resource location (which is optional for function definitions for backwards compatibility)

Example

#> scoreboard 91.ns.main
# The main storage of this data pack
# @internal
scoreboard objectives add 91.ns.main

Considerations

While this works well for things that are actually defined with a single command, such as objectives, bossbars and teams, other things datapacks use are not defined in such a way, including tags, scoreboard players, storages or any JSON definition.

In order to mitigate this, the restriction that the IMP doc has to appear right above the definition could be removed in this case

In a separate file

With this idea, definitions could be put into a separate file (with a custom extension such as .imp or .impdoc.
A file could hold multiple such definitions. This might be nice for giving a good overview of a specific directory, for example, such a file could be present in a namespace/advancements/... directory.

Example

> advancement namespace:technical/detect_kill
Detects a player killing another entity and runs the function namespace:detection/player_kill
@within function namespace:detection/player_kill

> advancement namespace:technical/in_water
Detects a player entering a water source block and runs the function namespace:detection/enter_water
@within function namespace:detection/enter_water

Considerations

While this might be nice for some things, like giving an overview of declarations in a datapack, It would also allow to separate different definitions, for example by declaring a tags.impdoc including all tags. I don't think it should be the only way. Thinking of e.g. java this feels more like an exported form of Javadoc (or impdoc in this case) instead of an in-code documentation

In a non-vanilla JSON tag (for JSON definitions only)

Pretty self-explanatory, just add a custom JSON tag at the beginning of a JSON file

Example

{
	"_impDoc": {
		"description": "Detects a player killing another entity and runs the function namespace:detection/player_kill"
		"within": {
			"type": "function",
			"value": "namespace:detection/player_kill"
		}
	},
	// ... (advancement definition here)
}

Considerations

I think this works pretty well and should be possible for anything that's defined as JSON, but of course does not work for anything else, so another solution is needed for those cases.

Anything else?

Feel free to add/ discuss other ideas here

[Arcensoth]

This is interesting. I like the idea of documenting more than just functions, but it's not something I've done consistently myself so it's difficult for me to gauge the pros and cons of each approach.

Right where you execute a defining command

The first approach as you suggest is inherently lacking in that not everything can be targeted this way. However it is has the benefit of being right where a human would probably expect it to be in the cases where it does apply, and could be used as an optional replacement for things like declaration macros.

Something else I want to comment on here is that I feel it's somewhat redundant to include the prefix (e.g. "scoreboard"), because the type of declaration can be inferred from the defining command that is required to immediately follow. With this generalization in mind, the special case for functions could become "the first block comment in the function, if it is not already a declaration for something else."

In a separate file

The second approach sounds like it would work for all current and future types of resources, as well as anything else we want to target such as objectives, entity tags, and teams. It feels the most "out of the way" in that it doesn't require integration with existing vanilla resources and can be done fully externally and optionally. That's not necessarily all good, though, as it does require more time and effort to maintain separate files with a different syntax/schema. This approach also raises questions regarding the file/directory structure and how the custom files are expected to be organized.

In a non-vanilla JSON tag (for JSON definitions only)

The third approach covers some of the remaining cases that the first approach naturally cannot, but it requires inventing an "intrusive" JSON sub-schema specifically for the documentation format. I'm mostly undecided whether this approach is worthwhile in light of the second approach... but, in the interest of human convenience (i.e. being able to embed the documentation directly within the resource itself), I think there's a fair argument for having this approach as an option as well.

[NeunEinser]

So, I thought a little more about this, and there seem to be 2 types of things to document.
One which has a declaration somewhere, like json resources, or scoreboard objectives, which need to be explicitly registered to Minecraft.

But then there are things which don't share this property, and the 1st and 3rd method cannot be directly applied. For example, for tags or storages, you can just add arbitary tag names and write to arbitary storages.
However we implement an IMP doc for such things, the IMP doc itself sort of implies that the thing we are documenting actually exists.

So, this raises the question, if it might make sense to separate declaring IMP doc from simple documenting IMP doc, or if IMP doc should even cover these cases.
Furthermore, there may be cases where one only wants to declare the existance of such a thing, but not actually document anything about it.

Dph+ solves this with the #declare directive, which then can also have an IMP doc associated with it.
I am unsure whether this is the way to go. Also it should be noted, that dph+ also allows to #declare anything else, like scoreboard objectives or advancements.

I am unsure what the best solution is for this, maybe just a beginning that's different from #> could be used here.

---

Something else I want to comment on here is that I feel it's somewhat redundant to include the prefix (e.g. "scoreboard"), because the type of declaration can be inferred from the defining command that is required to immediately follow. With this generalization in mind, the special case for functions could become "the first block comment in the function, if it is not already a declaration for something else."

Well, the act of saying the name of the thing you are documenting is also kind of redundant. Btw, why is that needed in the definition?
I basically just thought the idea behind this might be to just see what's being defined when looking in the file but not having the file name in view.
If that's the case, then even saying the name for things declared in a function file is not really a necessity either.

This approach also raises questions regarding the file/directory structure and how the custom files are expected to be organized.

Yeah, that's also something I thought about and I'm not really sure about. You could just put one file in each directory listing all the definitions, but that might not be very handy, and maybe you want to separate different kinds of definitions instead.

but it requires inventing an "intrusive" JSON sub-schema specifically for the documentation format.

I don't really know what would be intrusive about it, could you perhaps elaborate on that a bit more?

---

It also might make sense to ask a bit around on MCC to see what others think of this.

[Arcensoth]

Well, the act of saying the name of the thing you are documenting is also kind of redundant.

Fair point. The original motivating reason for function headers at the top of the file was to be able to easily copy-paste the function name into a function command. This was also an inspiring reason for the way the @input and @output annotations are formatted. I suppose adding additional contextual information to the beginning wouldn't take away from the ability to copy-paste, and it would probably help the computer decide what's what.

I don't really know what would be intrusive about it, could you perhaps elaborate on that a bit more?

By "intrusive" I basically just meant "not part of the actual vanilla resource" since we'd need to add our own extra, custom data to the file and we can't ever be absolutely 100% certain this won't cause conflicts in the future. IIRC some third-party tools like Blockbench have done this for comments and credit tags without issue for years, so by no means do I think it's a deal-breaker.

---

But then there are things which don't share this property, and the 1st and 3rd method cannot be directly applied. For example, for tags or storages, you can just add arbitary tag names and write to arbitary storages. [...]

Yeah, that's more-or-less what I was thinking while writing my thoughts about the second approach being the most generic. Were I to define these with a macro somewhere I'd probably do it in the setup/init/install function, but that's still an arbitrary choice and difficult to concretely define exactly what is a "setup" function.

[...] However we implement an IMP doc for such things, the IMP doc itself sort of implies that the thing we are documenting actually exists.

So, this raises the question, if it might make sense to separate declaring IMP doc from simple documenting IMP doc, or if IMP doc should even cover these cases.
Furthermore, there may be cases where one only wants to declare the existance of such a thing, but not actually document anything about it.

Right. It might make sense to declare the existence of something without documenting it, but I can't think of any case where you can document something that hasn't been declared. So I do think it would be useful to be able to both declare the existence of something at the same as documenting it, instead of having to take those two actions separately.

I am unsure what the best solution is for this, maybe just a beginning that's different from #> could be used here.

Possibly. I remember suggesting a similar idea about using a different block comment prefix to denote documentation vs just a header comment, here: SpyglassMC/Spyglass#802 (comment)

People like to use #> to highlight things as a visual indicator, without realizing it's also prompting the documentation syntax/analyzer. I'm reminded that it might be worth worth separating "comment headers" from "documentation blocks" to reduce the amount of confusion and, potentially, ambiguity.

An alternative to what's mentioned in that link: keep #> to denote declaration/documentation blocks, and introduce a new prefix such as #! or ## that simply highlights the line as a header comment. We could have a third prefix specifically to replace the special case for function headers, or it might just be better to keep them as a special case - there are fair arguments for either case.

[NeunEinser]

By "intrusive" I basically just meant "not part of the actual vanilla resource" since we'd need to add our own extra, custom data to the file and we can't ever be absolutely 100% certain this won't cause conflicts in the future. IIRC some third-party tools like Blockbench have done this for comments and credit tags without issue for years, so by no means do I think it's a deal-breaker.

Well, it's basically the same idea as IMP's custom item tags. You just add a underscore as first char, or even multiple, to make it pretty certain even with Mojang's non-existing conventions, you don't get a conflict.

Yeah, that's more-or-less what I was thinking while writing my thoughts about the second approach being the most generic. Were I to define these with a macro somewhere I'd probably do it in the setup/init/install function, but that's still an arbitrary choice and difficult to concretely define exactly what is a "setup" function.

Yeah, that's my thinking as well. Of course when in doubt you could create a dummy mcfunction file with only such declarations, but that's approach #2 again through the backdoor.

## would clash with dph+'s folding ranges.

Also, one more thought. This is example kind of unfortunate, if at the beginning of the function file

#>
# Main scoreboard
# @internal
scoreboard objectives add 91.ns.main

There could be a situation where you don't want to document the function, but you do want to document something you are declaring in the first line.
It might just make sense to introduce different indicators for non-function documentations alltogether. Maybe something like #: for non-function documentations and #! for declarations (which then could optionally have a documentation below it? Or do we just use #: above a #! declaration?)

[Arcensoth]

I think the solution here is to define separate a prefix for each required behaviour. So probably keep #> for the function header and introduce a second prefix for other types of declarations and a third prefix for simple highlighted headers.

[Arcensoth]

(cc @NeunEinser @SPGoding)

Coming back to this after a while, I see three distinct cases here:

1. Comment headings

For highlighting arbitrary text and creating a visual indicator:

#> Count the number of players and store the result.
execute store result score $players my.score if entity @a

Based on what I see in the wild, this is how most people use #> with language-mcfunction and DHP and it is often used throughout the whole function. Because of this (and because language-mcfunction introduced "block comments" on its own and operates entirely independent of IMP-Doc) I think #> should remain supported as-is and any additional functionality should be made available through new, separate comment types/prefixes.

2. Function headers

As IMP-Doc describes, "a series of comments that describe [a function's] purpose, context, and interface:"

#> imp:core/load
#
# Main loading procedure with the core entity as context.
#
# Anything branching from here can use `@s` instead of the core UUID.
#
# @context core

These would still be special-cased to the first line of a function to describe the function itself, but with a dedicated comment type there'd be zero ambiguity. For example, this would eliminate the edge-case where "you don't want to document the function, but you do want to document something you are declaring in the first line."

We could use #!, but I have this mild perhaps irrational fear that it might somehow be confused with the unix shebang in one way or another. I don't know if this is necessarily an argument against it, but it is at least something worth making note of.

I hadn't thought of it before, but #@ feels like it might work for function headers because (1) the two symbols are easy to type side-by-side (at least, on NA keyboards) and (2) the @ is already associated with the annotations used within the doc comments. However, this doesn't couple as well if other comment types also use annotations.

3. Declarations

Part of the inspiration behind this entire thread, being able to declare and document more than just functions:

#> scoreboard 91.ns.main
# The main storage of this data pack
# @internal
scoreboard objectives add 91.ns.main

This would provide a solid foundation for future expansion, and, in conjunction with a dedicated function header prefix, would help eliminate ambiguity.

We could probably use #: for this as you suggest, although it is quite visually similar to #!. At the end of the day it's mostly an arbitrary decision, apart from a some some small nitpicks here and there.

(Note: this does not account for the various types of JSON resources, only mcfunction comments. Standardizing some form of JSON documentation can be addressed as a potential extension to IMP-Doc following this one.)

[SPGoding]

I don't really have much input for this as it just seems like some arbitrary decisions. I will just comply with whatever is specified 👀

[NeunEinser]

Sounds good to me.
I was not really aware of the usage as block comments, but why not?

Also, I don't think anyone will/should run an mcfunction file as a shell script ever. This would also require the file's mode to be set executable. I also don't really care about the character, though 😉

[Arcensoth]

I recently pushed an update to language-mcfunction (version 0.18.0) that expanded the pool of possible block comment prefix characters.

Excerpt from Arcensoth/language-mcfunction#45 (comment):

I'll probably consider removing the block comment highlighting (including the @annotation highlights) entirely from the default grammar again in the future, but not until extension grammars that provide detailed block comment highlighting are available. (For example, a mcfunction-imp grammar that injects itself into the mcfunction grammar's block comment scope.)

If/when the mcfunction-imp grammar becomes a thing, it will probably be maintained as its own project separate from language-mcfunction available under a separate extension. That way it can evolve in its own without having to worry about side-effects on the default grammar and the vast majority of users who aren't concern with IMP-Doc.

[ChenCMD]

Have you decided anything about this issue?

[Arcensoth]

Have you decided anything about this issue?

No, and, if anything, I'm further from a decision than I was before.

I've done very little related to commands and data packs lately. I would like to find the time to commit to this, but until I can this project is in a state of limbo. IMP was founded on my own personal practices, and I don't want to move forward without being able to back up the decisions with my own experience.