Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Steamline hosting documentation #144

Closed
bramtayl opened this issue Jul 20, 2016 · 27 comments
Closed

Steamline hosting documentation #144

bramtayl opened this issue Jul 20, 2016 · 27 comments

Comments

@bramtayl
Copy link

bramtayl commented Jul 20, 2016

I've tried to follow through the steps in the hosting documentation section twice now and I've gotten stuck both times. I wonder if there would be a way to streamline some (or all) of the steps? Something like hostdocs()? Either that, or update Pkg.generate() to automatically include some (or all) of the necessary infrastructure?

@MichaelHatherly
Copy link
Member

Yes, we can certainly improve parts of that page if they're not clear enough. Could you describe which parts in particular weren't clear, since that'll help with figuring out what wording to change. Which steps did you get stuck on?

@bramtayl
Copy link
Author

bramtayl commented Jul 20, 2016

Hmm, I just tried a few hours ago and it took me a while to figure out how to install command line travis on Windows. First I had to install Ruby, and then I had to use Ruby's package manager to install travis. Then when I tried to rerun make.jl, I got an error like "ERROR: LoadError: stat: operation not permitted (EPERM)". I could have spent some more trying to figure out what was going wrong, but at that point I realized I hadn't even made it through the first of eight sections so I decided to give up. I'm not sure better documentation can help when the process itself seems overly involved.

@MichaelHatherly
Copy link
Member

install command line travis on Windows.

Yeah, I've not tried any of this on Windows, so it's not too surprising that it's a bit more difficult there. I think there may be a way to do the travis upload step manually, so I can try look into that — it would be great to not have to install all that extra stuff.

I got an error like "ERROR: LoadError: stat: operation not permitted (EPERM)"

If you could recreate that error and post the complete backtrace that would be great. Definitely shouldn't be throwing ugly errors like that.

@bramtayl
Copy link
Author

bramtayl commented Jul 20, 2016

Here is the code

pkgname = "ChainMap"

indexstring = """
using Documenter, $pkgname

makedocs()
"""

makestring = """
# $pkgname.jl Documentation

```@autodocs
Modules = [$pkgname]
\```
"""

mkpath("docs/src")
write("docs/make.jl", indexstring)
write("docs/src/index.md", makestring)
run(`julia docs/make.jl`)

This seems to work once or maybe twice and then stop working. The backslash isn't in the original (I bet there's a real escape sequence). My traceback is currently

ERROR: LoadError: stat: operation not permitted (EPERM)
 [inlined code] from stat.jl:44
 in stat at stat.jl:55
 in isdir at stat.jl:107
 in runner at C:\Users\jsnot\.julia\v0.4\Documenter\src\Builder.jl:97
 in dispatch at C:\Users\jsnot\.julia\v0.4\Documenter\src\Selectors.jl:164
 in anonymous at C:\Users\jsnot\.julia\v0.4\Documenter\src\Documenter.jl:153
 in cd at file.jl:32
 in makedocs at C:\Users\jsnot\.julia\v0.4\Documenter\src\Documenter.jl:152
 in include at boot.jl:261
 in include_from_node1 at loading.jl:320
 in process_options at client.jl:280
 in _start at client.jl:378
while loading C:\Users\jsnot\.julia\v0.4\ChainMap\docs\make.jl, in expression starting on line 3

@bramtayl
Copy link
Author

bramtayl commented Jul 20, 2016

Continuing on:

I'm trying to run Travis.genkeys("ChainMap") right now.

In Atom I get this error:

could not spawn `which which`: no such file or directory (ENOENT)
 in _jl_spawn at process.jl:262

In the REPL I get a key output, but then

ERROR: could not spawn `travis login --auto`: no such file or directory (ENOENT)
 in _jl_spawn at process.jl:262

I would continue trying to add my key to the repository, but I don't see any instructions displayed?

@bramtayl
Copy link
Author

P.S. I can't figure out how to install command line which for Windows. ssh-keygen seems to have come with git but I have to add it to my path

@MichaelHatherly
Copy link
Member

run(`julia docs/make.jl`)

Does just running julia make.jl from the docs/ folder directly work, rather than via run?

operation not permitted (EPERM)

I've come across those before, unrelated to Documenter, when packages are installed on a network drive. Is your package local or on a network folder?

In Atom I get this error:

Perhaps run all the commands in a normal Julia terminal rather than through Atom's terminal just so we can avoid possible interference from there.

could not spawn which which``

The which checks are probably not all that useful really, for now maybe just comment then out with the following diff:

diff --git a/src/Documenter.jl b/src/Documenter.jl                                                                             
index 91b5111..27fb6bd 100644                                                                                                  
--- a/src/Documenter.jl                                                                                                        
+++ b/src/Documenter.jl                                                                                                        
@@ -431,10 +431,10 @@ following command lines programs to be installed:                                                        
 function genkeys(package)                                                                                                     
     # Error checking. Do the required programs exist?                                                                         
     isdir(Pkg.dir(package))     || error("'$package' could not be found in '$(Pkg.dir())'.")                                  
-    success(`which which`)      || error("'which' not found.")                                                                
-    success(`which git`)        || error("'git' not found.")                                                                  
-    success(`which travis`)     || error("'travis' not found.")                                                               
-    success(`which ssh-keygen`) || error("'ssh-keygen' not found.")                                                           
+    # success(`which which`)      || error("'which' not found.")                                                              
+    # success(`which git`)        || error("'git' not found.")                                                                
+    # success(`which travis`)     || error("'travis' not found.")                                                             
+    # success(`which ssh-keygen`) || error("'ssh-keygen' not found.")                                                         

     directory = "docs"                                                                                                        
     filename  = ".documenter"

@bramtayl
Copy link
Author

bramtayl commented Jul 20, 2016

No, everything is local. Running make.jl from within doc does work. Commenting out the which lines seems to have helped atom out, but it freezes as a y/n question (.documenter already exists, do you want to replace it). I tried adding ssh-keygen to my path (which is hiding in C:\Program Files\Git\usr\bin) but I'm still getting the same error in the repl.

@MichaelHatherly
Copy link
Member

Running make.jl from within doc does work.

Presumably cd that makedocs calls internally isn't actually changing directory to docs if you're outside of the docs directory. Perhaps an upstream issue with cd and Windows, might be worth trying to reduce that to a non-Documenter-dependant example if possible if you can?

Commenting out the which lines seems to have helped atom out, but it freezes as a y/n question

Not sure if Atom has implemented user input in it's terminal. That's probably why it froze. Just use the normal Julia terminal instead.

I tried adding ssh-keygen to my path (which is hiding in C:\Program Files\Git\usr\bin) but I'm still getting the same error in the repl.

What steps did you take to add ssh-keygen to your path? I'm not a Windows user though, so not sure how helpful I can be with debugging that one.

@bramtayl
Copy link
Author

bramtayl commented Jul 21, 2016

ssh-keygen is correctly in my path. It's in system properties, the advanced tab, then click the environmental variables... button. I finally got Travis.genkeys to work by following through the command line code manually. The auto login does not seemed to have worked: I still had to enter my github credentials.

@bramtayl
Copy link
Author

PS the which command-line tool is also packaged with git in the same folder that ssh-keygen is.

@MichaelHatherly
Copy link
Member

The auto login does not seemed to have worked: I still had to enter my github credentials.

For the travis login? That should happen the first time you use it, but after that it should cache credentials for you and not require a login afterwards.

ssh-keygen is correctly in my path

which command-line tool is also packaged with git in the same folder that ssh-keygen is.

So in Julia's shell mode neither of those are available, but are on your path? Probably an upstream issue and not related to Documenter then I'd assume.

I'll see if I can set up a Windows VM sometime next week perhaps and see if I can work out what's happening myself.

@bramtayl
Copy link
Author

bramtayl commented Jul 22, 2016

I think I'm getting closer. I've put together all the changes I've attempted to make into a single function. This kind of model (one function to do everything) is what I was hoping for.

function defaultdocumentation(PACKAGE_NAME, USER_NAME)

  cd(Pkg.dir(PACKAGE_NAME))

  indexstring = """
  using Documenter, $PACKAGE_NAME

  makedocs(
    modules = $PACKAGE_NAME
  )

  deploydocs(
    repo = "github.com/$USER_NAME/$PACKAGE_NAME.jl.git"
    deps = Deps.pip("pygments", "mkdocs", "mkdocs-material", "python-markdown-math")
  )
  """

  makestring = """
  # $PACKAGE_NAME.jl Documentation

  ```@index
  \```

  ```@autodocs
  Modules = [$PACKAGE_NAME]
  \```
  """

  traviscode = """
    - julia -e 'Pkg.add("Documenter")'
    - julia -e 'cd(Pkg.dir("$PACKAGE_NAME")); include(joinpath("docs", "make.jl"))'
  """

  mkdocscode = """
  site_name:        $PACKAGE_NAME.jl
  repo_url:         https://github.com/$USER_NAME/$PACKAGE_NAME.jl
  site_author:      $USER_NAME

  theme: readthedocs

  extra:
    palette:
      primary: 'indigo'
      accent:  'blue'

  extra_css:
    - assets/Documenter.css

  extra_javascript:
    - https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML
    - assets/mathjaxhelper.js

  markdown_extensions:
    - codehilite
    - extra
    - tables
    - fenced_code
    - mdx_math

  docs_dir: 'build'

  pages:
    - Home: index.md
  """

  gitignorecode =
  """
  docs/build/
  docs/site/
  """

  readmecode = [
  "[![](https://img.shields.io/badge/docs-stable-blue.svg)](https://$USER_NAME.github.io/$PACKAGE_NAME.jl/stable)\n"
  "[![](https://img.shields.io/badge/docs-latest-blue.svg)](https://$USER_NAME.github.io/$PACKAGE_NAME.jl/latest)\n"
  ]

  mkpath("docs/src")
  write("docs/make.jl", indexstring)
  write("docs/src/index.md", makestring)

  open(".travis.yml","a") do x
    write(x, traviscode)
  end

  open(".gitignore","a") do x
    write(x, gitignorecode)
  end

  # I bet there's a better way to insert into the third line of a file

  readmeexisting = readlines("README.md")
  output = [readmeexisting[1:2]; readmecode; readmeexisting[3:end]]
  write("README.md", output)

end

@MichaelHatherly
Copy link
Member

This kind of model (one function to do everything) is what I was hoping for.

This is kind of what Documenter.generate does, though without modifying files that already exist (which I wouldn't really what it to ever do). I've noticed though that it's not mentioned in the either the "guide" or "hosting" pages... which it clearly should be.

@bramtayl
Copy link
Author

In travis, under all four builds, I'm getting "skipping docs deployment". It looks like that could happen for any number of reasons. Would it be possible for documenter to also list a reason why (or a list of reasons why) it's skipping deployment?

@MichaelHatherly
Copy link
Member

You can set the DOCUMENTER_DEBUG environment variable in the travis settings page for the repo. Set it to true and it should then print out some useful debugging info if you rerun the last build (no need to push new commits for this, just use the rebuild button in the top right.)

From looking at the ChainMap.jl repo here's a few things that might be help:

  • .documenter.enc should be in the docs/ directory. Have you added your private key to the GtiHub settings page for the repo? (Since you committed the private key to the repo itself by accident I assume the current .documenter.enc file is a new one and not associated with that private key. If not you'll need to redo that step completely since that repo is currently unsecured.)
  • deploydocs will only run on a single build, not all four, and by default that's the Linux nightly one. So that's the one to pay closest attention to when debugging this.

@bramtayl
Copy link
Author

Ok, I must of messed up everything when I tried to do it manually. So I'm trying to get travis login --auto to work.

I ran

travis login --auto --debug and I'm getting
** Faraday::SSLError: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

I've poked around on the internet for a fix. People are saying to making sure AddTrustExternalCARoot-2048.pem is in my C:\Ruby22-x64\lib\ruby\2.2.0\rubygems\ssl_certs folder, and I've checked and it is.

@MichaelHatherly
Copy link
Member

How did you install travis (and Ruby)? Googling that error gave me a couple of pages suggesting it was a problem with how they had been installed, but nothing terribly specific about how to fix it.

(Note that there's also the alternative route for giving Travis push access to the needed repo, using the GITHUB_API_KEY mentioned in the https://juliadocs.github.io/Documenter.jl/latest/man/hosting/#travis-environment-settings section. It's not the best approach since Travis gets access to all your repos on your account, but it does tend to less difficult to get working.)

@bramtayl
Copy link
Author

bramtayl commented Jul 26, 2016

Ok, well, that second method seems to have worked. But now I'm getting another error:

Documenter: building documentation.
INFO    -  Building documentation to directory: /home/travis/.julia/v0.5/ChainMap/docs/site 
ERROR   -  Error building page index.md 
Traceback (most recent call last):
  File "/home/travis/.local/bin/mkdocs", line 11, in <module>
    sys.exit(cli())
  File "/home/travis/.local/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/travis/.local/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/travis/.local/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/travis/.local/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/travis/.local/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/__main__.py", line 137, in build_command
    ), clean_site_dir=clean)
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/commands/build.py", line 289, in build
    build_pages(config)
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/commands/build.py", line 249, in build_pages
    dump_json)
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/commands/build.py", line 169, in _build_page
    site_navigation=site_navigation
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/commands/build.py", line 35, in convert_markdown
    extension_configs=config['mdx_configs']
  File "/home/travis/.local/lib/python2.7/site-packages/mkdocs/utils/__init__.py", line 337, in convert_markdown
    extension_configs=extension_configs or {}
  File "/home/travis/.local/lib/python2.7/site-packages/markdown/__init__.py", line 159, in __init__
    configs=kwargs.get('extension_configs', {}))
  File "/home/travis/.local/lib/python2.7/site-packages/markdown/__init__.py", line 185, in registerExtensions
    ext = self.build_extension(ext, configs.get(ext, {}))
  File "/home/travis/.local/lib/python2.7/site-packages/markdown/__init__.py", line 264, in build_extension
    module = importlib.import_module(module_name_old_style)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: Failed loading extension 'mdx_math' from 'mdx_math', 'markdown.extensions.mdx_math' or 'mdx_mdx_math'
ERROR: LoadError: failed process: Process(`mkdocs build`, ProcessExited(1)) [1]
 in pipeline_error(::Base.Process) at ./process.jl:612
 in run at ./process.jl:588 [inlined]
 in (::Documenter.##10#16)() at /home/travis/.julia/v0.5/Documenter/src/Documenter.jl:240
 in (::Documenter.##6#12{String,String,String,Documenter.##10#16,String,String,String,String,Bool})() at /home/travis/.julia/v0.5/Documenter/src/Documenter.jl:297
 in cd(::Documenter.##6#12{String,String,String,Documenter.##10#16,String,String,String,String,Bool}, ::String) at ./file.jl:59
 in #deploydocs#4(::String, ::String, ::String, ::String, ::String, ::String, ::String, ::Documenter.Deps.##1#2{Tuple{String,String}}, ::Documenter.##10#16, ::Documenter.#deploydocs) at /home/travis/.julia/v0.5/Documenter/src/Documenter.jl:293
 in (::Documenter.#kw##deploydocs)(::Array{Any,1}, ::Documenter.#deploydocs) at ./<missing>:0
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in process_options(::Base.JLOptions) at ./client.jl:239
 in _start() at ./client.jl:318
while loading /home/travis/.julia/v0.5/ChainMap/docs/make.jl, in expression starting on line 5

(MH: edited formatting)

@MichaelHatherly
Copy link
Member

Failed loading extension 'mdx_math' from 'mdx_math', 'markdown.extensions.mdx_math' or 'mdx_mdx_math'

Those are additional plugins for mkdocs that make LaTeX syntax parse correctly. If you don't have any LaTeX in your docs, which it seems you don't, then you should be able to remove the line https://github.com/bramtayl/ChainMap.jl/blob/c1b392211ce9507a3740fd11563e29f3e774720d/docs/mkdocs.yml#L19 from your mkdocs.yml file and it should build fine I think.

(Does mkdocs build your docs fine locally?)

@bramtayl
Copy link
Author

bramtayl commented Jul 29, 2016

It took me a while to get mkdocs up and running as well; downloading python, adding pip to my path (which came with the version of python I downloaded), downloading mkdocs with pip, figuring out the syntax to mkdocs...

Well I got at least a latest version of the docs up and running. It's only a list of docstrings at the moment. I decided that I wanted to move my tests to the examples in the docstrings. That meant that the coverage statistics weren't working for me. Anyway, I wrote a short script to put all the code in the examples in my docstrings into my runtests.jl file. I had to rerun this every time I updated my docstrings. I'm sure there is a better way?

function doclines(filein)
  text = readlines(filein)

  starts = map(_ -> ismatch(r"```.+", chomp(_) ), text)
  ends = map(_ -> chomp(_) == "```", text)
  Test.@test sum(starts) == sum(ends)

  withbounds = text[cumsum(starts) - cumsum(ends) .== 1]
  withoutbounds = filter(x -> !startswith(x, "```"), withbounds)
  backslash_removed = map(x -> replace(x, r"\\", ""), withoutbounds)
end

lines = ["using ChainMap\n\n"
         doclines(joinpath(Pkg.dir("ChainMap", "src", "doc.jl")));
         doclines(joinpath(Pkg.dir("ChainMap", "README.md")))]

write(joinpath(Pkg.dir("ChainMap"), "test", "runtests.jl"),
      lines)

@MichaelHatherly
Copy link
Member

It took me a while to get mkdocs

Yeah, it can be a bit of a pain. We'll be adding some HTML output soonish so that you don't need mkdocs at all.

That meant that the coverage statistics weren't working for me.

What could work is running docs/make.jl as part of your tests. You'd of course need to add Documenter to test/REQUIRE, but I think that should give you coverage results for any code that Documenter runs in your doctests by adding something like

include(joinpath("..", "docs", "make.jl"))

to your runtests.jl file.

At the moment even if there's doctest errors makedocs won't throw an error that you could actually test for, but I think it might be worth adding a keyword like strict = true/false to makedocs that throws an error if anything fails when set to true.

@bramtayl
Copy link
Author

Hey, so I finally went back and fixed my tests so all that's there is makedocs(strict = true). I'm running into an issue now where on 0.5 a bunch of doctests are failing because of extraneous warnings: WARNING: could not attach metadata for @simd loop. See https://travis-ci.org/bramtayl/ChainMap.jl/jobs/178656636. This is only happening in 0.5, it seems because of https://groups.google.com/forum/#!topic/julia-dev/u6G7f9mxaHI. I was wondering if you had any advice?

@MichaelHatherly
Copy link
Member

Since those kind of warnings seem to usually only show up the first time some code is run you could probably run the code that causes them some time prior to the doctests, perhaps in docs/make.jl before calling makedocs. Then the calls in the actual doctest shouldn't print any warnings I'd think — haven't actually tested that out though.

@bramtayl
Copy link
Author

I'm not sure I understand. I tried duplicating the doctest call; first with strict = false then with strict = true, and it didn't seem to have helped.

@bramtayl
Copy link
Author

bramtayl commented Nov 26, 2016

Solved with putting if VERSION > v"0.6.0-dev"; Pkg.test("ChainMap"; coverage = true); else; Pkg.test("ChainMap"); end into my travis .yml file

@MichaelHatherly
Copy link
Member

To avoid the warnings what I mean is:

$ julia --inline=no

julia> using ChainMap

julia> _ = [1, -1];

julia> @over abs
WARNING: could not attach metadata for @simd loop.
2-element Array{Int64,1}:
 1
 1

julia> @over abs
2-element Array{Int64,1}:
 1
 1

So the first call to @over produces a warning, but the second one doesn't, so you can find the expressions in your jldoctest blocks that are causing warnings and duplicate them in your make.jl above makedocs, something like

using Documenter, ChainMap

_ = [1, -1]
@over abs

makedocs(strict = true)

which should silence the warnings within the actual docstrings hopefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants