Add several more Roff extensions #4309

jordemort · 2018-11-01T18:01:08Z

To go along with github/markup#1196 this PR adds several more extensions to the Roff language.

Description

I surveyed the manpages in the Ubuntu Cosmic, Debian Stretch, and CentOS 7, and in the OpenIndiana Hipster Live DVD. Here are the extensions used, by count, aggregated by unique filename: https://gist.github.com/c69fa7739d6e4731e90d5749c887f29b

I then went and searched for any of the extensions that had more than 100 uses on GitHub - this PR adds the extensions that, as a result of that search:

Appear to be in wide use on GitHub
Appear to primarily be used for manpages, as opposed to some other format
Appear in more than one or two repositories

They went through a further round of winnowing as I was writing up this PR and deciding if I was prepared to defend the addition of each extension.

This ended up excluding a lot of the nonstandard ones from OpenIndiana, which only appear in one or two mirrors on GitHub, along with a lot of assorted exotica. Those will have to wait until @Alhadis adds a shiny new smart Roff strategy.

Checklist:

I am associating a language with a new file extension.
- The new extension is used in hundreds of repositories on GitHub.com
  - 0p: https://github.com/search?p=1&q=extension%3A0p+NOT+nothack&type=Code
  - 1p: https://github.com/search?p=1&q=extension%3A1p+NOT+nothack&type=Code
  - 1pm: https://github.com/search?q=extension%3A1pm+NOT+nothack&type=Code
  - 3G/3g: https://github.com/search?p=1&q=extension%3A3G+NOT+nothack&type=Code (used by Apple SDK docs, apparently)
  - 3o: https://github.com/search?p=1&q=extension%3A3o+NOT+nothack&type=Code&utf8=%E2%9C%93 (Common for Ocaml, it seems)
  - 3p: https://github.com/search?q=extension%3A3p+NOT+nothack&type=Code
  - 3perl: https://github.com/search?q=extension%3A3perl+NOT+nothack&type=Code
  - 3pm: https://github.com/search?q=extension%3A3pm+NOT+nothack&type=Code
- I have included a real-world usage sample for all extensions added in this PR:
- I have included a change to the heuristics to distinguish my language from others using the same extension.
  - There don't appear to be any other uses of this exension

Alhadis

LGTM. Covering every in-the-wild extension used by Roff is as daunting as covering every XML-based file format, which is why we seriously need a similarly dedicated strategy.

pchaigno

Thanks @jordemort for the pull request!

I've checked the usage of the new file extensions, and several are well below our current threshold for in-the-wild usage of hundreds of repositories. I'm not too much worried about conflicts with other languages; even if such conflicts arise, Roff looks fairly easy to distinguish from other languages. Nevertheless, I'm not very comfortable accepting some of these file extensions (those below 100 repositories) as they will make it much harder for us to reject other candidates for support in the future: it's difficult to stay credible when rejecting a file extension with 80 repositories if you accepted one with 20 repositories a month ago...

For those extensions, maybe working on a dedicated strategy as @Alhadis suggested would be a better approach. (With a dedicated strategy, we don't need to add new file extensions.)

Alternatively, if you believe their usage will grow over time, I can add them to #4219, and I'll add support for them myself when they reach the threshold.

pchaigno · 2018-11-02T13:44:37Z

lib/linguist/languages.yml

@@ -4091,14 +4091,22 @@ Roff:
  color: "#ecdebe"
  extensions:
  - ".roff"
+  - ".0p"


I counted only 19 users with 19 repositories using this file extension. That's very low considering our usual threshold of hundreds of repositories. That might be less of an issue if we consider .h.0p instead of .0p (because it's very unlikely we ever get a conflicting language for that longer extension), but it will only match 13 repositories out of the 19. We will have a hard time rejecting other file extensions if we accept that one.

pchaigno · 2018-11-02T14:01:10Z

lib/linguist/languages.yml

  - ".1"
  - ".1in"
  - ".1m"
+  - ".1p"


This file extension is used in 170 repositories by 153 users. It's a bit low but I'd say it's okay, because Roff looks fairly easy to distinguish from other languages with a heuristic rule, if needed.

pchaigno · 2018-11-02T14:18:23Z

lib/linguist/languages.yml

  - ".1"
  - ".1in"
  - ".1m"
+  - ".1p"
+  - ".1pm"


This file extension is used by only 11 users with 11 repositories. Again, this is very low.

pchaigno · 2018-11-02T15:01:02Z

lib/linguist/languages.yml

  - ".3in"
  - ".3m"
+  - ".3o"


This file extension is used by 13 users with 15 repositories.

pchaigno · 2018-11-02T15:26:12Z

lib/linguist/languages.yml

  - ".3in"
  - ".3m"
+  - ".3o"
+  - ".3p"
+  - ".3perl"


This file extension is used by only 19 users (20 repositories), but perhaps it's less likely to conflict with other languages in the future due to the perl?

Alhadis · 2018-11-04T11:38:54Z

[…] maybe working on a dedicated strategy as @Alhadis suggested would be a better approach.

I got this.

jordemort · 2018-11-05T17:11:47Z

@pchaigno

.Np extensions - I added these because they're used in https://mirrors.edge.kernel.org/pub/linux/docs/man-pages/man-pages-posix/ and there seemed to be a bit of concern about supporting them over in the discussion on Add support for manpages via mandoc github/markup#1196 but I'm fine with leaving them out in favor of waiting for a better Roff strategy
.1pm / .3pm - The .3pm extension is really the one I want added most out of all of these, since it seems to be in wide use by various Perl projects. I waffled on .1pm but it seems weird to add .3pm but not .1pm
.3perl - added because it seemed weird to do .3pm but not .3perl - I think .3perl might mostly be a Solarisism, though; it's probably fine to wait
.3o - not sure why I added this, it can also probably wait

Alhadis · 2018-11-06T09:25:00Z

Okay, I need help. Which do you folks prefer for a manpage-strategy?

Highly accurate but complicated and ugly code?
Lower accuracy but much clearer and idiomatic Ruby?

I'm trying to juggle performance, accuracy, and something that makes sense to Ruby programmers, but I realise trying to uphold all three is going to be impossible.

For the record, I'm only concentrating on matching manpages (Roff documents which use man(7) and mdoc(7) macros specifically)...

pchaigno · 2018-11-06T10:43:38Z

I'll always be leaning for the less-accurate but clearer solution for heuristics. Ugly code is harder to maintain, and we have to consider the performance impact.

Alhadis · 2018-11-06T10:58:54Z

and we have to consider the performance impact.

Actually, that's a large part of why this looks like a mess. 😀 Everything is hard-wired to short-circuit as soon as possible, leaving to a very inside-out looking flow of logic.

I think I'll open a W.I.P pull-request so we can discuss this a bit clearer. Just wait until I've finished rewriting the first mess. :D

stale · 2018-12-06T11:01:54Z

This pull request has been automatically marked as stale because it has not had recent activity, and will be closed if no further activity occurs. If this pull request was overlooked, forgotten, or should remain open for any other reason, please reply here to call attention to it and remove the stale status. Thank you for your contributions.

stale · 2018-12-22T10:24:10Z

This pull request has been automatically closed because it has not had activity in a long time. Please feel free to reopen it or create a new issue.

References: #4258, #4309, #4317

* Add strategy to identify Roff man pages References: #4258, #4309, #4317 * Remove `# coding: utf-8` junk injected by accident

Nweman

Nwes Thailand

Add several more Roff extensions

c3d5aa1

jordemort requested review from lildude and Alhadis November 1, 2018 18:01

Alhadis approved these changes Nov 1, 2018

View reviewed changes

pchaigno suggested changes Nov 2, 2018

View reviewed changes

stale bot added the Stale label Dec 6, 2018

stale bot closed this Dec 22, 2018

Alhadis added a commit that referenced this pull request Feb 24, 2019

Add strategy to identify Roff man pages

e79e407

References: #4258, #4309, #4317

Alhadis mentioned this pull request Feb 24, 2019

Add strategy to identify Roff man pages: Take 2 #4433

Merged

2 tasks

Alhadis added a commit that referenced this pull request Aug 12, 2019

Add strategy to identify Roff man pages: Take 2 (#4433)

2991273

* Add strategy to identify Roff man pages References: #4258, #4309, #4317 * Remove `# coding: utf-8` junk injected by accident

Alhadis deleted the more-roff branch August 12, 2019 12:11

Nweman reviewed Oct 31, 2021

View reviewed changes

github-linguist locked as resolved and limited conversation to collaborators Jun 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add several more Roff extensions #4309

Add several more Roff extensions #4309

jordemort commented Nov 1, 2018

Alhadis left a comment

pchaigno left a comment •

edited

Loading

pchaigno Nov 2, 2018

rixardo84 Nov 3, 2018

pchaigno Nov 2, 2018

pchaigno Nov 2, 2018

pchaigno Nov 2, 2018

pchaigno Nov 2, 2018

Alhadis commented Nov 4, 2018

jordemort commented Nov 5, 2018

Alhadis commented Nov 6, 2018 •

edited

Loading

pchaigno commented Nov 6, 2018

Alhadis commented Nov 6, 2018

stale bot commented Dec 6, 2018

stale bot commented Dec 22, 2018

Nweman left a comment

Add several more Roff extensions #4309

Add several more Roff extensions #4309

Conversation

jordemort commented Nov 1, 2018

Description

Checklist:

Alhadis left a comment

Choose a reason for hiding this comment

pchaigno left a comment • edited Loading

Choose a reason for hiding this comment

pchaigno Nov 2, 2018

Choose a reason for hiding this comment

rixardo84 Nov 3, 2018

Choose a reason for hiding this comment

pchaigno Nov 2, 2018

Choose a reason for hiding this comment

pchaigno Nov 2, 2018

Choose a reason for hiding this comment

pchaigno Nov 2, 2018

Choose a reason for hiding this comment

pchaigno Nov 2, 2018

Choose a reason for hiding this comment

Alhadis commented Nov 4, 2018

jordemort commented Nov 5, 2018

Alhadis commented Nov 6, 2018 • edited Loading

pchaigno commented Nov 6, 2018

Alhadis commented Nov 6, 2018

stale bot commented Dec 6, 2018

stale bot commented Dec 22, 2018

Nweman left a comment

Choose a reason for hiding this comment

pchaigno left a comment •

edited

Loading

Alhadis commented Nov 6, 2018 •

edited

Loading