[INFRA] Use linkchecker to verify URLs #79

yarikoptic · 2018-10-31T03:04:23Z

More TODOs outside the scope of this PR:

Unfortunately verification of HTML URL anchors seems to be plain broken. See e.g. ~~Anchor Link Checking Only Checks One Link Per Thread wummel/linkchecker#513~~ link anchors checks broken linkchecker/linkchecker#179 so would be very nice if someone looked into it. Should be something really fixable
I have not enabled --check-extern because apparently there is quite a number of URLs which seems to be broken, including the ones to cognitive atlas (e.g. http://www.cognitiveatlas.org/term/id/trm_54e69c642d89b)

yarikoptic · 2018-10-31T03:19:25Z

hm, did I screw up circle-ci or it is not enabled at all? I see only Travis build, which doesn't run mkdocs, so I was enhancing circle-ci configuration

chrisgorgo · 2018-10-31T04:13:53Z

I eneabled it on forks, push something new, maybe it will trigger.

yarikoptic · 2018-10-31T06:01:34Z

ok, linkchecker quick and dirty fixes proposed now and fresh finding now is (edits from maintainers are allowed, so welcome to push the proper fix)

URL        `#heading=h.5u721tt1h9pe'
Name       `8.3.1. Common MR metadata fields'
Parent URL file:///home/yoh/proj/bids/bids-specification/site/04-modality-specific-files/01-magnetic-resonance-imaging-data.html, line 1300, col 8
Real URL   file:///home/yoh/proj/bids/bids-specification/site/04-modality-specific-files/01-magnetic-resonance-imaging-data.html
Check time 0.280 seconds
D/L time   0.001 seconds
Size       65.02KB
Modified   2018-10-31 05:58:49.047498Z
Warning    [None] Anchor `heading=h.5u721tt1h9pe' not found.
           Available anchors: `__drawer', `__search', `__toc',
           `a-real-fieldmap-image', `anatomical-landmarks',
           `anatomy-imaging-data',
           `case-4-multiple-phase-encoded-directions-pepolar',
           `common-metadata-fields', `diffusion-imaging-data',
           `fieldmap-data', `fmri-task-information',
           `in-plane-spatial-encoding',
           `institution-information',
           `magnetic-resonance-imaging-data', `nav-1', `nav-1-4',
           `other-recommended-metadata',
           `phase-difference-image-and-at-least-one-magnitude-image',
           `required-fields', `rf-contrast', `scanner-hardware',
           `sequence-specifics', `slice-acceleration',
           `task-including-resting-state-imaging-data',
           `timing-parameters', `timing-parameters_1',
           `two-phase-images-and-two-magnitude-images'.
Result     Valid

Statistics:
Downloaded: 3.40MB.
Content types: 5 image, 96 text, 0 video, 0 audio, 55 application, 0 mail and 118 other.
URL lengths: min=8, max=130, avg=72.

That's it. 274 links in 274 URLs checked. 1 warning found. 0 errors found.
Stopped checking at 2018-10-31 01:59:01-004 (2 seconds)

ATM it reveals way too many problems to deal at once, e.g.: URL 'http://www.cognitiveatlas.org/term/id/trm_54e69c642d89b' Name 'http://www.cognitiveatlas.org/term/id/trm_54e69c642d89b' Parent URL file:///home/yoh/proj/bids/bids-specification/site/04-modality-specific-files/02-magnetoencephalography.html, line 639, col 184 Real URL http://www.cognitiveatlas.org/term/id/trm_54e69c642d89b Check time 1.214 seconds Result Error: 404 Not Found

I am not sure if there is any other than this config file way - could not find

yarikoptic · 2018-11-21T22:53:57Z

anyone has a clue what is up with circle CI? I hoped for current run to depict the correctly detected anchor but got IMHO unrelated to my changes

Virtualenv location: /home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q
Pipfile.lock (b67896) out of date, updating to (ab1abf)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
env/utils.py", line 402, in resolve_deps
    req_dir=req_dir
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 250, in actually_resolve_deps
    req = Requirement.from_line(dep)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 704, in from_line
    line, extras = _strip_extras(line)
TypeError: 'module' object is not callable

Exited with code 1

now I will push the fix for the detected wrong anchor

…other points

chrisgorgo · 2018-11-21T23:05:11Z

Not sure what is going on. This error should not have affected pip 18.0 and master is not failing. Try committing updated Pipenv.lock

yarikoptic · 2018-11-22T03:31:21Z

time will come when I loose my superpowers of breaking things:

$> pipenv lock 
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv==11.9.0', 'console_scripts', 'pipenv')()
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pipenv/cli.py", line 512, in lock
    verbose=verbose, clear=clear, pre=pre, keep_outdated=keep_outdated
  File "/usr/lib/python3/dist-packages/pipenv/core.py", line 1140, in do_lock
    vcs_deps = convert_deps_to_pip(project.vcs_packages, project, r=False)
  File "/usr/lib/python3/dist-packages/pipenv/utils.py", line 672, in convert_deps_to_pip
    extra = '{0}+{1}'.format(vcs, deps[dep][vcs])
TypeError: string indices must be integers

mild wild guess -- probably due to my introduced git+https url

…he "file" Got it after manually doing pipenv install http:/... ; pipenv lock

For some reason they aren't installed automagically

…r is

yarikoptic · 2018-11-22T04:28:28Z

I almost won! but damn things doesn't give up ;)

ImportError: /home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python3.6/site-packages/linkcheck/HtmlParser/htmlsax.cpython-36m-x86_64-linux-gnu.so: undefined symbol: PyString_FromStringAndSize

;-)

yarikoptic · 2018-11-22T06:43:40Z

oh, python3 is not supported by linkchecker yet... overall it is starting to look more ugly than more beautiful

sappelhoff · 2019-03-22T12:24:05Z

I think that this PR would be a valuable addition. What is holding you off from trying to make it work @yarikoptic ?

yarikoptic · 2019-03-22T12:54:47Z

hiccup:

needs my patched version to work correctly for anchors checks. Do include anchors within cache_url, mutex around the parsing linkchecker/linkchecker#194 was not finished/merged and got stale
linkchecker is a python project but still py2 only
when I switched circle ci to use py2, installation failed since

singledispatch was not installed

#!/bin/bash -eo pipefail
pipenv run mkdocs build --clean --strict --verbose
DEBUG   -  Loading configuration file: /home/circleci/project/mkdocs.yml 
Traceback (most recent call last):
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/bin/mkdocs", line 11, in <module>
    sys.exit(cli())
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/__main__.py", line 162, in build_command
    site_dir=site_dir
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/base.py", line 197, in load_config
    errors, warnings = cfg.validate()
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/base.py", line 107, in validate
    run_failed, run_warnings = self._validate()
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/base.py", line 62, in _validate
    self[key] = config_option.validate(value)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/config_options.py", line 132, in validate
    return self.run_validation(value)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/config_options.py", line 572, in run_validation
    plgins[item] = self.load_plugin(item, cfg)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/mkdocs/config/config_options.py", line 580, in load_plugin
    Plugin = self.installed_plugins[name].load()
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2345, in load
    self.require(*args, **kwargs)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2368, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/circleci/.local/share/virtualenvs/project-zxI9dQ-Q/lib/python2.7/site-packages/pkg_resources/__init__.py", line 784, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'singledispatch' distribution was not found and is required by tornado
Exited with code 1

probably could be easily mitigated by adjusting pipenv setup to include it explicitly (for py2 only). But I already forgot how to use that pipenv beast... help would be welcome! ;)

sappelhoff · 2019-03-22T13:01:58Z

Okay thanks for the summary. Not sure whether linkchecker will support python 3 anytime soon though --> linkchecker/linkchecker#40

yarikoptic · 2019-03-22T15:39:17Z

yeap, but bids-standard afaik can be built under python2 right?
moreover linkchecker is just a tool, so in principle could be installed in an independent environment. I was just trying to piggyback on the existing one, but it is not strictly required

sappelhoff · 2019-03-22T15:47:28Z

yeap, but bids-standard afaik can be built under python2 right?

I'd rather keep it under Python3, but that might be my irrational dislike for python2.

in principle could be installed in an independent environment

right, ... we could perhaps have a second "environment" or "workflow", however one would call it.
I think that using travis for this may be a bit more straight forward, because there is not so much going in our travis flow (compared to the busy circleci config.yml)

yarikoptic · 2019-03-22T15:57:12Z

although true - it would entail setting up the full website building env on travis too then.
Anyways - it would "work for me", just that I would not be able to take a crack at it soon

sappelhoff · 2019-03-22T16:00:28Z

although true - it would entail setting up the full website building env on travis too then.

ah, I had hoped that the linkchecker could work on the source data (markdown). Okay, then circle ci might work just as well in a separate py2 workflow.

it would "work for me", just that I would not be able to take a crack at it soon

okay. It'll have to wait for some more then.

yarikoptic · 2019-04-15T15:43:40Z

Small update: There is an ongoing active effort to make linkchecker py3 compatible (linkchecker/linkchecker#210) so I expect to come back to this one as soon as there is a version to try.

yarikoptic · 2019-05-10T02:14:47Z

FWIW note -- not yet, e.g. here is the most recent encounter linkchecker/linkchecker#230 (comment)

sappelhoff · 2019-05-16T09:53:52Z

is this MkDocs plugin perhaps already sufficient? --> https://github.com/manuzhang/mkdocs-htmlproofer-plugin

yarikoptic · 2019-05-16T13:32:36Z

who knows? might as well be and shouldn't hurt to be enabled regardless if it makes build less buggy!

sappelhoff · 2019-05-16T16:15:32Z

who knows? might as well be and shouldn't hurt to be enabled regardless if it makes build less buggy!

I am just surprised that the plugin has a class with three methods (less than 100lines of code) ... whereas the linkchecker is a huge software project ...

but both should serve the same purpose (identify urls / links that lead to bad pages). Makes me suspicious 🤔

sappelhoff · 2019-08-01T20:45:39Z

I just stumbled over this: https://github.com/davidtheclark/remark-lint-no-dead-urls

it would probably be easy to implement, because we are using remark already:

bids-specification/.travis.yml

Lines 1 to 10 in e151e41

    
           language: node_js 
        
           node_js: 
        
             - "10" 
        
           cache: 
        
             directories: 
        
             - node_modules # NPM packages 
        
           before_script: 
        
             - npm install remark-cli@5.0.0 remark-lint@6.0.2 remark-preset-lint-recommended@3.0.2 remark-preset-lint-markdown-style-guide@2.1.2 
        
           script: 
        
             - remark src/*.md src/*/*.md --frail

bids-specification/.remarkrc

Lines 1 to 9 in e151e41

    
           { 
        
             "plugins": [ 
        
               "preset-lint-markdown-style-guide", 
        
               ["lint-no-duplicate-headings", false], 
        
               ["lint-list-item-indent", "tab-size"], 
        
               ["lint-emphasis-marker", "consistent"], 
        
               ["lint-maximum-line-length", false] 
        
             ] 
        
           }

yarikoptic · 2019-08-01T21:09:35Z

Shouldn't hurt but seems to only care about external URLs. I wanted to check all the #id ones since they are prone to break. Since python 3 ification is going too slow I will try to find time to provide singularity based run

yarikoptic · 2019-08-02T17:52:00Z

I have worked out locally a docker (since should already be present on travis/circle) based recipe only to realize that circle-ci environments are already docker environments! d'oh! Learning how to chain the jobs now.... Replacement PR might come shortly or I will comment on here that I give up again ;)

yarikoptic force-pushed the bf-links branch from 05b1735 to ba58ad6 Compare October 31, 2018 04:35

yarikoptic added 6 commits November 21, 2018 17:03

ENH: add linkchecker to Pipfile

fb13124

ENH: run linkchecker afer mkdocs build

efc2150

BF: fixed a url (https -> http)

cb2e9f9

Install patched (to work anchors correctly) linkchecker

49c5bdb

Configure linkchecker to do the anchorcheck

be68cb0

I am not sure if there is any other than this config file way - could not find

yarikoptic force-pushed the bf-links branch from ba58ad6 to be68cb0 Compare November 21, 2018 22:30

BF: fix the stale anchor left from google doc, also add reference at …

f4c222e

…other points

yarikoptic force-pushed the bf-links branch from d029ec2 to 9e36c11 Compare November 22, 2018 03:11

wrapping long lines

fa039ca

yarikoptic force-pushed the bf-links branch from 9e36c11 to fa039ca Compare November 22, 2018 03:31

yarikoptic added 5 commits November 21, 2018 22:34

Let's try regular https:// url to .zip instead of git+https

0960092

Fresh Pipfile.lock

4851bf3

BF: tune up Pipfile entry for linkchecker - should be a record with t…

820e53e

…he "file" Got it after manually doing pipenv install http:/... ; pipenv lock

BF: move linkchecker into develop and install its depends

eb0c8f1

For some reason they aren't installed automagically

Pipenv - Install development packages, since that is where linkchecke…

b204f62

…r is

yarikoptic force-pushed the bf-links branch from baacb25 to b204f62 Compare November 22, 2018 04:20

yarikoptic added 2 commits November 22, 2018 00:47

Disable forcing python 3.6 in Pipfile

1826446

Try python 2.7 since linkchecker is no good with python3 yet

180421c

yarikoptic mentioned this pull request Nov 25, 2018

[FIX] fix some urls (as detected to be broken/inconsistent) #95

Merged

yarikoptic mentioned this pull request Dec 13, 2018

[FIX] repair links #106

Merged

sappelhoff changed the title ~~Use linkchecker to verify that URLs are Ok~~ [INFRA] Use linkchecker to verify URLs Apr 12, 2019

yarikoptic mentioned this pull request May 10, 2019

[ENH] BIDS software development guidelines appendix #227

Closed

1 task

sappelhoff mentioned this pull request Aug 2, 2019

[FIX] 404 error link for fsl dtifit #292

Closed

yarikoptic mentioned this pull request Aug 2, 2019

[INFRA] Use linkchecker (from a dedicated docker image) to check all URLs #293

Merged

6 tasks

yarikoptic closed this Aug 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[INFRA] Use linkchecker to verify URLs #79

[INFRA] Use linkchecker to verify URLs #79

yarikoptic commented Oct 31, 2018 •

edited

Loading

yarikoptic commented Oct 31, 2018

chrisgorgo commented Oct 31, 2018

yarikoptic commented Oct 31, 2018

yarikoptic commented Nov 21, 2018

chrisgorgo commented Nov 21, 2018

yarikoptic commented Nov 22, 2018

yarikoptic commented Nov 22, 2018

yarikoptic commented Nov 22, 2018

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Apr 15, 2019

yarikoptic commented May 10, 2019

sappelhoff commented May 16, 2019

yarikoptic commented May 16, 2019

sappelhoff commented May 16, 2019

sappelhoff commented Aug 1, 2019 •

edited

Loading

yarikoptic commented Aug 1, 2019

yarikoptic commented Aug 2, 2019

[INFRA] Use linkchecker to verify URLs #79

[INFRA] Use linkchecker to verify URLs #79

Conversation

yarikoptic commented Oct 31, 2018 • edited Loading

yarikoptic commented Oct 31, 2018

chrisgorgo commented Oct 31, 2018

yarikoptic commented Oct 31, 2018

yarikoptic commented Nov 21, 2018

chrisgorgo commented Nov 21, 2018

yarikoptic commented Nov 22, 2018

yarikoptic commented Nov 22, 2018

yarikoptic commented Nov 22, 2018

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Mar 22, 2019

sappelhoff commented Mar 22, 2019

yarikoptic commented Apr 15, 2019

yarikoptic commented May 10, 2019

sappelhoff commented May 16, 2019

yarikoptic commented May 16, 2019

sappelhoff commented May 16, 2019

sappelhoff commented Aug 1, 2019 • edited Loading

yarikoptic commented Aug 1, 2019

yarikoptic commented Aug 2, 2019

yarikoptic commented Oct 31, 2018 •

edited

Loading

sappelhoff commented Aug 1, 2019 •

edited

Loading