Skip to content
This repository has been archived by the owner on Nov 9, 2022. It is now read-only.

Commit

Permalink
Update template (#19)
Browse files Browse the repository at this point in the history
* add image-with-shadow class

* Ignore .jekyll-metatada

* Speed up builds of R-based lessons

R-based lessons might take a while to build because packages need to be compiled from source. RStudio Package Manager has compiled versions of packages for ubuntu distros starting with 16.04: https://packagemanager.rstudio.com/client/#/repos/1/overview

I've added the necessary magic in the actions yaml to make it work.

* No need for User Agent string

* Update repo names

* permissive checks for pre-alpha lessons

This will fix #533

* bin/lesson_check.py: allow 'language-*' code blocks (#532)

* bin/lesson_check.py: allow 'caution' blockquote

* avoid ansi color characters from being printed

* deploy from "website" action

* deploy R-based lessons without using another action

* also delete _site

* include @zkamvar suggestions

Co-authored-by: Zhian N. Kamvar <zkamvar@gmail.com>

* pin ubuntu version to 20.04 (#540)

Co-authored-by: Zhian N. Kamvar <zkamvar@gmail.com>

* GitHub Actions: cache required R packages (#534)

* add missing parenthesis

* Add incubator option for carpentry field. (#542)

Closes #541

* .editorconfig: don't trim trailing spaces in markdown

* lesson.scss: HTML block

carpentries/styles#519

* add patch to clean gh-pages before committing (#545)

This will address #544

* Fix Kramdown parser crash

... by using GFM (GitHub-flavored Markdown) parser (`kramdown-parser-gfm`)
instead of the default one (`kramdown`).

The default one fails to produce an AST (Abstract Syntax Tree) when
there is no blank line before the line with the opening code fence.

Related:
 - gettalong/kramdown#530
 - Python-Markdown/markdown#807

Fixes: carpentries/styles#543

* bin/util.py: Change ruby executable to "bundle exec ruby"

Closes: carpentries/styles#547

* Change link colours (#549)

...to make them distinguishable from regular text. And for accessibility!

* bin/workshop_check.py: update default contact email address

* Gemfile: add 'webrick' dependency for Ruby 3.0.0 and above

Fixes carpentries/styles#552

* lesson_check.py allow for missing life_cycle

This will fix carpentries/styles#556

* update with Maxim's suggestion

* Add catch for None type code block in lesson_check

There are times when the AST is malformed and does not emit a class for
the code element. We do not want the parser to crash when this happens,
but we also want to notify ourselves that the AST is malformed.

This should not result in an error because as we saw in
carpentries/styles#543, the parser itself can
cause these malformations when the lesson itself renders well. Even
though we fixed the previous issue with an updated parser, problems
still persist:
swcarpentry/r-novice-gapminder#696 (comment)

I fully admit that this is a kludge.

* fix syntax

I've removed the print condition, because it will just result in an error no matter what (sigh)

* Makefile: fix 'bundle config' command flags

* Makefile: clean target: remove .vendor, .bundle, Gemfile.lock

Clean up:
1. `.vendor` directory where Bundler installs all the gems.
2. `.bundle` directory where Bundler stores its settings.
3. `Gemfile.lock` file generated by the Bundler.

* Makefile: silence Docker commands

* use Ruby's official GH Actions

* Makefile: use SHELL to call bin/knit_lesson.sh

* Makefile: fix up PHONY targets

* Fix GitHub actions for lessons in Rmarkdown

Specifically, set CRAN repository to https://cran.rstudio.com

* apply single shadow to image class

* use grey shadow instead of transparent black

* expand image-with-shadow selection

* Makefile: don't fail when Python isn't found

* bin/dependencies.R: handle 'no packages were specified' error

Fixes the following issue:

```
$ make site
lib paths: /Library/Frameworks/R.framework/Versions/3.5/Resources/library
Error in install.packages(missing_pkgs, lib = lib, repos = repos) :
  no packages were specified
Calls: install_required_packages -> install.packages
Execution halted
make: *** [install-rmd-deps] Error 1
```

* Don't check links.md in lessons that use remote theme

Fixes carpentries/styles#570

* add link references to code_of_conduct.md (#572)

* Update links.md

* add source_dir argument

This will fix carpentries/styles#576

* Improved relative_root_path

* update contributing guide

* add further languages for box titles (#580)

Will be useful for HPC-Carpentry lessons, GPU programming lesson as well as Julia lessons which are currently in the incubator.

* bin/lesson_check.py: allow comments and empty lines in links.md

* bin/lesson_check.py: one more fix for using_remote_theme()

* Template workflow: add two more lessons

* add make lesson-check-all step

* Set working directory for the 'make site' step

* Template workflow: smarter syncing with the styles repo

Current syncing procedure that used in the Template workflow fails for:

1. Lessons that are, in fact, nsync with the styles repo.
2. For lessons that use The Carpentries' remote theme and have deleted
   some of the files.

This PR makes this step a little bit more intelligent and takes into
account the above two scenarios.

* Apply Zhian's suggestions

Co-authored-by: Zhian N. Kamvar <zkamvar@gmail.com>

* add math support with katex (#573)

* bin/util.py: remove unused 'IMAGE_FILE_SUFFIX' var (#590)

It should've been removed in 7e835fd.

* bin/lesson_check.py: use proper function

* bin/lesson_check.py: allow exceptions to line length limit

Allow lines that contain a single image or a single link
to go over the suggested line length limit.

* lesson_check.py: harden single-line image/link pattern

This change hardens the pattern that matches single-line
image or link:

1. It extends the pattern to be matched in a heading
2. It allows the line to contain {: ...} customizations
3. It allows the line to end with \

* lesson_check.py: relax P_LINK_IMAGE_LINE pattern

This PR allows up to 3 non-word (`\W` in Python's `re`-speak) characters
in the beginning and end of the pattern that matches links and images.
This is to allow lesson developers place punctuation marks, parentheses,
or other symbols before or after the link or image on the same line in
Markdown.

* bin/util.py: Factor out reporter class. Define __all__

* Don't force hostname into relative_root_path

* lesson_check.py: add a comment about importing * from a package

Co-authored-by: Zhian N. Kamvar <zkamvar@gmail.com>

* lesson.scss: define 'inline' class for images

Define `inline` class for images that should not be displayed as block
elements.

By appending `{:class="inline"}` or `{: .inline}` to the image definition
in Markdown, one can create an inline image that doesn't break the
current line and is embedded in the paragraph. Useful for showing
special symbols and hieroglyphs that we can't display by other means.

* Fix Reporter class imports

* update R dependency search; Allow Bioconductor packages (#600)

* Automatically add deep anchor links using AnchorJS

* Makefile: require index.md (#607)

* Makefile: docker-serve target: ensure Docker is installed (#608)

* Fix broken "How to contribute" link

* lesson_check.py: report check status at the end

* util.py: load_yaml: Don't fail when it's not necessary

Also, make 'require()' function not fail by default.
The only case where we really need to fail is when 'kramdown' parser is
not specified. This is a highly unlikely scenario, tbh (because
arguments to `lesson_check.py` are set in the Makefile), but we can
think about reworking/optimizing this part later.

* lesson_check.py: fix error message for the 'defaults' check

* lesson_check.py: use proper regex for matching episode files

Co-authored-by: Toby Hodges <tbyhdgs@gmail.com>
Co-authored-by: Zhian N. Kamvar <zkamvar@gmail.com>
Co-authored-by: Maxim Belkin <maxim.belkin@gmail.com>
Co-authored-by: François Michonneau <francois.michonneau@gmail.com>
Co-authored-by: Trevor Keller <trevor.keller@gmail.com>
Co-authored-by: Renato Alves <alves.rjc@gmail.com>
Co-authored-by: Andrew Reid <andrew.reid@nist.gov>
Co-authored-by: Bailey Harrington <baileythegreen@gmail.com>
Co-authored-by: Michael Joseph <josephmje.22@gmail.com>
Co-authored-by: Alan O'Callaghan <alan.ocallaghan@outlook.com>
Co-authored-by: Benson Muite <bkmgit@users.noreply.github.com>
Co-authored-by: K.E. Koziar <katherine.koziar@ucr.edu>
  • Loading branch information
13 people authored Jul 21, 2021
1 parent f45bb84 commit 493844a
Show file tree
Hide file tree
Showing 25 changed files with 569 additions and 230 deletions.
1 change: 1 addition & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ trim_trailing_whitespace = true
indent_size = 2
indent_style = space
max_line_length = 100 # Please keep this in sync with bin/lesson_check.py!
trim_trailing_whitespace = false # keep trailing spaces in markdown - 2+ spaces are translated to a hard break (<br/>)

[*.r]
max_line_length = 80
Expand Down
101 changes: 77 additions & 24 deletions .github/workflows/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,16 +6,18 @@ on:
jobs:
check-template:
name: ${{ matrix.lesson-name }} (${{ matrix.os-name }})
if: github.repository == 'carpentries/styles'
if: github.repository == 'hsf-training/hsf-styles'
runs-on: ${{ matrix.os }}
continue-on-error: ${{ matrix.experimental }}
strategy:
fail-fast: false
matrix:
lesson: [swcarpentry/shell-novice, datacarpentry/r-intro-geospatial, librarycarpentry/lc-git]
os: [ubuntu-latest, macos-latest, windows-latest]
os: [ubuntu-20.04, macos-latest, windows-latest]
experimental: [false]
include:
- os: ubuntu-latest
os-name: Ubuntu
- os: ubuntu-20.04
os-name: Linux
- os: macos-latest
os-name: macOS
- os: windows-latest
Expand All @@ -26,16 +28,27 @@ jobs:
lesson-name: (DC) R Intro Geospatial
- lesson: librarycarpentry/lc-git
lesson-name: (LC) Intro to Git
- lesson: datacarpentry/astronomy-python
lesson-name: (DC) Foundations of Astronomical Data Science
experimental: true
os: ubuntu-20.04
os-name: Linux
- lesson: carpentries/lesson-example
lesson-name: (CP) Lesson Example
experimental: false
os: ubuntu-20.04
os-name: Linux
defaults:
run:
shell: bash # forces 'Git for Windows' on Windows
env:
RSPM: 'https://packagemanager.rstudio.com/cran/__linux__/bionic/latest'
RSPM: 'https://packagemanager.rstudio.com/cran/__linux__/focal/latest'
steps:
- name: Set up Ruby
uses: actions/setup-ruby@v1
uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
bundler-cache: true

- name: Set up Python
uses: actions/setup-python@v2
Expand All @@ -44,7 +57,7 @@ jobs:

- name: Install GitHub Pages, Bundler, and kramdown gems
run: |
gem install github-pages bundler kramdown
gem install github-pages bundler kramdown kramdown-parser-gfm
- name: Install Python modules
run: |
Expand All @@ -61,25 +74,54 @@ jobs:
path: lesson
fetch-depth: 0

- name: Determine the proper reference to use
id: styles-ref
- name: Sync lesson with carpentries/styles
working-directory: lesson
run: |
if [[ -n "${{ github.event.pull_request.number }}" ]]; then
echo "::set-output name=ref::refs/pull/${{ github.event.pull_request.number }}/head"
echo "::group::Fetch Styles"
if [[ -n "${{ github.event.pull_request.number }}" ]]
then
ref="refs/pull/${{ github.event.pull_request.number }}/head"
else
echo "::set-output name=ref::gh-pages"
ref="gh-pages"
fi
- name: Sync lesson with carpentries/styles
working-directory: lesson
run: |
git config --global user.email "team@carpentries.org"
git config --global user.name "The Carpentries Bot"
git remote add styles https://github.com/carpentries/styles.git
git config --local remote.styles.tagOpt --no-tags
git fetch styles ${{ steps.styles-ref.outputs.ref }}:styles-ref
git merge -s recursive -Xtheirs --no-commit styles-ref
git commit -m "Sync lesson with carpentries/styles"
git fetch styles $ref:styles-ref
echo "::endgroup::"
echo "::group::Synchronize Styles"
# Sync up only if necessary
if [[ $(git rev-list --count HEAD..styles-ref) != 0 ]]
then
# The merge command below might fail for lessons that use remote theme
# https://github.com/carpentries/carpentries-theme
echo "Testing merge using recursive strategy, accepting upstream changes without committing"
if ! git merge -s recursive -Xtheirs --no-commit styles-ref
then
# Remove "deleted by us, unmerged" files from the staging area.
# these are the files that were removed from the lesson
# but are still present in the carpentries/styles repo
echo "Removing previously deleted files"
git rm $(git diff --name-only --diff-filter=DU)
# If there are still "unmerged" files,
# let's raise an error and look into this more closely
if [[ -n $(git diff --name-only --diff-filter=U) ]]
then
echo "There were unmerged files in ${{ matrix.lesson-name }}:"
echo "$(git diff --compact-summary --diff-filter=U)"
exit 1
fi
fi
echo "Committing changes"
git commit -m "Sync lesson with carpentries/styles"
fi
echo "::endgroup::"
- name: Look for R-markdown files
id: check-rmd
Expand All @@ -95,8 +137,10 @@ jobs:

- name: Install needed packages
if: steps.check-rmd.outputs.count != 0
working-directory: lesson
run: |
install.packages(c('remotes', 'rprojroot', 'renv', 'desc', 'rmarkdown', 'knitr'))
source('bin/dependencies.R')
install_required_packages(.libPaths()[1])
shell: Rscript {0}

- name: Query dependencies
Expand All @@ -106,26 +150,35 @@ jobs:
source('bin/dependencies.R')
deps <- identify_dependencies()
create_description(deps)
use_bioc_repos()
saveRDS(remotes::dev_package_deps(dependencies = TRUE), ".github/depends.Rds", version = 2)
writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), ".github/R-version")
shell: Rscript {0}

- name: Cache R packages
- name: Restore Package Cache
if: runner.os != 'Windows' && steps.check-rmd.outputs.count != 0
uses: actions/cache@v1
uses: actions/cache@v2
with:
path: ${{ env.R_LIBS_USER }}
key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }}
restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-

- name: Install stringi from source
if: runner.os == 'Linux' && steps.check-rmd.outputs.count != 0
run: install.packages('stringi', repos='https://cloud.r-project.org')
shell: Rscript {0}

- name: Install system dependencies for R packages
if: runner.os == 'Linux' && steps.check-rmd.outputs.count != 0
working-directory: lesson
run: |
while read -r cmd
do
eval sudo $cmd
done < <(Rscript -e 'cat(remotes::system_requirements("ubuntu", "18.04"), sep = "\n")')
eval sudo $cmd || echo "Nothing to update"
done < <(Rscript -e 'cat(remotes::system_requirements("ubuntu", "20.04"), sep = "\n")')
- run: make site
working-directory: lesson

- run: make lesson-check-all
working-directory: lesson
73 changes: 55 additions & 18 deletions .github/workflows/website.yml
Original file line number Diff line number Diff line change
@@ -1,20 +1,26 @@
name: Website
on:
push:
branches: gh-pages
branches:
- gh-pages
- main
pull_request: []
jobs:
build-website:
if: ${{ !endsWith(github.repository, '/styles') }}
runs-on: ubuntu-latest
if: ${{ !endsWith(github.repository, '/hsf-styles') }}
runs-on: ubuntu-20.04
env:
RSPM: "https://packagemanager.rstudio.com/cran/__linux__/focal/latest"
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
defaults:
run:
shell: bash
steps:
- name: Set up Ruby
uses: actions/setup-ruby@v1
uses: ruby/setup-ruby@v1
with:
ruby-version: '2.7'
bundler-cache: true

- name: Set up Python
uses: actions/setup-python@v2
Expand All @@ -23,7 +29,7 @@ jobs:

- name: Install GitHub Pages, Bundler, and kramdown gems
run: |
gem install github-pages bundler kramdown
gem install github-pages bundler kramdown kramdown-parser-gfm
- name: Install Python modules
run: |
Expand All @@ -46,10 +52,19 @@ jobs:
with:
r-version: 'release'

- name: Restore R Cache
if: steps.check-rmd.outputs.count != 0
uses: actions/cache@v2
with:
path: ${{ env.R_LIBS_USER }}
key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }}
restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-

- name: Install needed packages
if: steps.check-rmd.outputs.count != 0
run: |
install.packages(c('remotes', 'rprojroot', 'renv', 'desc', 'rmarkdown', 'knitr'))
source('bin/dependencies.R')
install_required_packages()
shell: Rscript {0}

- name: Query dependencies
Expand All @@ -58,28 +73,50 @@ jobs:
source('bin/dependencies.R')
deps <- identify_dependencies()
create_description(deps)
use_bioc_repos()
saveRDS(remotes::dev_package_deps(dependencies = TRUE), ".github/depends.Rds", version = 2)
writeLines(sprintf("R-%i.%i", getRversion()$major, getRversion()$minor), ".github/R-version")
shell: Rscript {0}

- name: Cache R packages
if: steps.check-rmd.outputs.count != 0
uses: actions/cache@v1
with:
path: ${{ env.R_LIBS_USER }}
key: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-${{ hashFiles('.github/depends.Rds') }}
restore-keys: ${{ runner.os }}-${{ hashFiles('.github/R-version') }}-1-

- name: Install system dependencies for R packages
if: steps.check-rmd.outputs.count != 0
run: |
while read -r cmd
do
eval sudo $cmd
done < <(Rscript -e 'cat(remotes::system_requirements("ubuntu", "18.04"), sep = "\n")')
eval sudo $cmd || echo "Nothing to update"
done < <(Rscript -e 'cat(remotes::system_requirements("ubuntu", "20.04"), sep = "\n")')
- name: Render the markdown and confirm that the site can be built
run: make site

- name: Checkout github pages
if: ${{ github.event_name == 'push' && steps.check-rmd.outputs.count != 0 && github.ref != 'refs/heads/gh-pages'}}
uses: actions/checkout@master
with:
ref: gh-pages
path: gh-pages

- name: Commit and Push
if: ${{ github.event_name == 'push' && steps.check-rmd.outputs.count != 0 && github.ref != 'refs/heads/gh-pages'}}
run: |
# clean up gh-pages
cd gh-pages
git rm -rf . # remove all previous files
git restore --staged . # remove things from the stage
cd ..
# copy everything into gh-pages site
cp -r `ls -A | grep -v 'gh-pages' | grep -v '.git' | grep -v '.bundle/' | grep -v '_site'` gh-pages
# move into gh-pages, add, commit, and push
cd gh-pages
# setup git
git config --local user.email "actions@github.com"
git config --local user.name "GitHub Actions"
git add -A .
git commit --allow-empty -m "[Github Actions] render website (via ${{ github.sha }})"
git push origin gh-pages
# return
cd ..
- run: make site
- run: make lesson-check
if: always()
- run: make lesson-check-all
if: always()
3 changes: 2 additions & 1 deletion CODE_OF_CONDUCT.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ we pledge to follow the [Carpentry Code of Conduct][coc].
Instances of abusive, harassing, or otherwise unacceptable behavior
may be reported by following our [reporting guidelines][coc-reporting].

{% include links.md %}
[coc]: https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html
[coc-reporting]: https://docs.carpentries.org/topic_folders/policies/incident-reporting.html
4 changes: 4 additions & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,7 @@ git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
ruby '>=2.5.5'

gem 'github-pages', group: :jekyll_plugins

if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new('3.0.0')
gem 'webrick', '>= 1.6.1'
end
Loading

0 comments on commit 493844a

Please sign in to comment.