Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

old files keep lying around in staging-dir and output-dir #72

Open
sternenseemann opened this issue Dec 6, 2014 · 22 comments
Open

old files keep lying around in staging-dir and output-dir #72

sternenseemann opened this issue Dec 6, 2014 · 22 comments

Comments

@sternenseemann
Copy link
Contributor

when not using the git backend.

@kingcons
Copy link
Collaborator

kingcons commented Dec 6, 2014

What do you mean by the "git backend"? More details on how to reproduce would be appreciated.

@sternenseemann
Copy link
Contributor Author

.coleslawrc:

(:author "Lukas Epple"
 :deploy-dir "/home/lukas/Documents/blog/output/"
 :domain "http://lukasepple.de"
 :feeds ("lisp")
 :plugins ((static-pages))
 :routing ((:post           "posts/~a")
           (:tag-index      "tag/~a")
           (:month-index    "date/~a")
           (:numeric-index  "~d")
           (:feed           "~a.xml")
           (:tag-feed       "tag/~a.xml"))
 :sitenav ((:url "/" :name "Blog")
           (:url "/about.html" :name "About me")
           (:url "https://twitter.com/lukasepple" :name "Twitter")
           (:url "https://github.com/lukasepple" :name "Code")
           (:url "https://flickr.com/photos/sternenseemann" :name "Fotos")
           (:url "/imprint.html" :name "Imprint"))
 :staging-dir "/tmp/coleslaw/"
 :title "Lukas' blog"
 :theme "stars"
 :lang "de"
 :license "<a rel=\"license\" href=\"//creativecommons.org/licenses/by-sa/4.0/\"><img alt=\"Creative Commons License\" src=\"//i.creativecommons.org/l/by-sa/4.0/80x15.png\" /></a>")

After generating old generated files keep showing up in deploy-dir. That can be stopped by deleting staging-dir.

I myself fixed this like this: http://lukasepple.de/posts/coleslaw-helpers.html

@sternenseemann
Copy link
Contributor Author

Just now an idea came to my mind. We could write a cli interface (using cl-launch) to solve that elegantely:

  • coleslaw build builds the site
  • coleslaw clean removes the files from output-dir and staging-dir
  • coleslaw rebuild does clean and then build

(Just like Hakyll)

@sternenseemann
Copy link
Contributor Author

I think it would be nice if we agree on one set of cli command that gets implemented.

@kingcons
Copy link
Collaborator

kingcons commented Dec 7, 2014

I would love to see such a pull request, or at least more details on what a solution like that looks like.
In fact, there's a note about looking into this in the HACKING docs here: https://github.com/redline6561/coleslaw/blob/master/docs/hacking.md#scripting-conveniences

I'm happy with your proposed set of commands. It might be nice for people who aren't doing "git-style" deploys to have a coleslaw post or coleslaw new command that takes a filename and creates it in the repo with the header filled out appropriately (i.e. date, format: md, title, empty tags field).

@ghost
Copy link

ghost commented Dec 8, 2014

Good idea! Maybe we can also have a coleslaw serve command just like jekyll sreve to test site in local. Or coleslaw new a repo dir with shlyfile.lisp(just like caveman).I use shelly build and test my site now. Just put shlyfile.lisp in the repo dir with code

(ql:quickload :hunchentoot)
(ql:quickload :coleslaw)

(defun build-blog ()
  (coleslaw:main #p"/home/qwfwq/blog/post/"))
(defun serve ()
  (hunchentoot:start (make-instance 'hunchentoot:easy-acceptor
                        :port 4242 :document-root #p"/home/qwfwq/blog/www/")))

Then I can use shly build-blog to build blog and use shly serve to test site in local.

(I am a common lisp newbie now,maybe I can do something some day.)

@kingcons
Copy link
Collaborator

kingcons commented Dec 8, 2014

@erlFelaP Good idea, thanks! I'd forgotten about shelly. That might be perfect for this.

@PuercoPop
Copy link
Collaborator

I currently use cl-launch to build, as the script is pretty much the same as the githook. The pros of cl-launch is that is looks like any other CLI tool from the perspective of the user, the cons are that they require an additional installation from the perspective of the user and that error messages are make clojure's stack traces look concise. a problem when not when debugging (as one can just use the REPL) but certainly for when an user wants to submit an error report. Maybe shelly fares better in this regard?

#!/usr/bin/cl -Q -sp coleslaw --entry entry
;;;;
;; This script assumes it is executed from the repository's top level directory
;; to determine correctly the blog-dir variable.
;;;;

(defun entry (argv)
  (declare (ignorable argv))

  (let 
      ((old-rev (inferior-shell:run/s
                 "git log --oneline -1 | awk -e '{print $1}'"))
       (blog-dir (uiop/os:getcwd)))
    (main blog-dir old-rev)))

If recently found about eazy-project which supports skeletons for scaffolding (akin to leiningen). Although it doesn't appear to be easily extendable. Should we open a branch to start working the command set proposed by Lucas + the serve commands?

@kingcons
Copy link
Collaborator

kingcons commented Dec 9, 2014

I would love that. I think shelly and cl-launch are both good options. I'd be happy to bundle code for both. Let's cut a branch called cli-commands.

@sternenseemann
Copy link
Contributor Author

I would like to help doing that. As a downside of shelly you would not have a coleslaw binary, you'd have to invoke shelly rebuild which does not seem optimal to me.

@kingcons
Copy link
Collaborator

kingcons commented Dec 9, 2014

That's true. We could always have a shell alias alias coleslaw=shelly coleslaw or function to drop in your .profile. I agree it's not ideal and there may be better workarounds but I'll look into that later. If y'all just want to look into cl-launch for now I can have a look at shelly later on.

@sternenseemann
Copy link
Contributor Author

Sounds good!

@sternenseemann
Copy link
Contributor Author

I was able to create a simple coleslaw binary like this:

buildapp \
    --output coleslaw \
    --asdf-path "." \
    --load-system coleslaw \
    --asdf-tree "~/quicklisp/dists/quicklisp/software/" \
    --eval '(defun main (argv)
                (declare (ignore argv))
                (coleslaw:main "."))'

Biggest Problem: Size is 83MB.

Using buildapp would cause having a shell script which:

  1. Checks if quicklisp is available
  2. If true, calls sbcl to quickload everything needed (buildapp needs the asdf foo to be there before compiling)
  3. Call buildapp like above
  4. Else fails

@sternenseemann
Copy link
Contributor Author

The buildapp binary is currently weirdly broken. A more sophisticated build script woud look like this:

#!/bin/sh
fatal() {
    echo "$1"
    exit 1
}

test -f ~/quicklisp/setup.lisp || fatal "You need quicklisp to use coleslaw"

sbcl --no-userinit --no-sysinit --non-interactive \
    --load ~/quicklisp/setup.lisp \
    --eval '(dolist (systems (quote ("closure-template" "3bmd" "3bmd-ext-code-blocks" "alexandria" "local-time"
                                     "inferior-shell" "cl-fad" "cl-ppcre" "closer-mop" "cl-unicode")))
                    (ql:quickload systems))' \
    --eval '(ql:write-asdf-manifest-file "quicklisp-manifest.txt")'

buildapp \
    --output coleslaw \
    --asdf-path "." \
    --load-system "coleslaw" \
    --manifest-file "quicklisp-manifest.txt" \
    --eval '(defun main (argv) 
                (coleslaw:main "."))' || fatal "buildapp failed!"

Running resulting binary fails for me with strange errors maybe someone can find out what they mean:

ASDF could not load sb-bsd-sockets because
Don't know how to REQUIRE sb-bsd-sockets.
See also:
  The SBCL Manual, Variable SB-EXT:*MODULE-PROVIDER-FUNCTIONS*
  The SBCL Manual, Function REQUIRE.

debugger invoked on a SB-INT:EXTENSION-FAILURE in thread
#<THREAD "main thread" RUNNING {10051D67A3}>:
  Don't know how to REQUIRE sb-bsd-sockets.
See also:
  The SBCL Manual, Variable SB-EXT:*MODULE-PROVIDER-FUNCTIONS*
  The SBCL Manual, Function REQUIRE

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry
                                     completing load for #<REQUIRE-SYSTEM "sb-bsd-sockets">.
  1: [ACCEPT                       ] Continue, treating
                                     completing load for #<REQUIRE-SYSTEM "sb-bsd-sockets">
                                     as having been successful.
  2: [TRY-RECOMPILING              ] Recompile impl and try loading it again
  3:                                 Retry
                                     loading FASL for #<CL-SOURCE-FILE "quicklisp" "impl">.
  4:                                 Continue, treating
                                     loading FASL for #<CL-SOURCE-FILE "quicklisp" "impl">
                                     as having been successful.
  5:                                 Retry ASDF operation.
  6: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  7: [RETRY                        ] Retry EVAL of current toplevel form.
  8: [CONTINUE                     ] Ignore error and continue loading file "/home/lukas/quicklisp/setup.lisp".
  9: [ABORT                        ] Abort loading file "/home/lukas/quicklisp/setup.lisp".
 10:                                 Retry EVAL of current toplevel form.
 11:                                 Ignore error and continue userinit file "/home/lukas/.sbclrc".
 12:                                 Abort userinit file "/home/lukas/.sbclrc".
 13:                                 Skip to toplevel READ/EVAL/PRINT loop.
 14: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(SB-IMPL::REQUIRE-ERROR "Don't know how to ~S ~A." REQUIRE "sb-bsd-sockets")
0] ; 
; compilation unit aborted
;   caught 4 fatal ERROR conditions
*

@sternenseemann
Copy link
Contributor Author

I think cl-launch could be a more reasonable alternative to buildapp

@kingcons
Copy link
Collaborator

Yeah, I wouldn't worry with standalone binaries/buildapp just yet. Some users would probably be fine with installing an 80mb executable to run coleslaw. We can look into it later. For now, let's just get a nice CLI interface up with cl-launch or shelly.

Trying to get a binary for folks that don't want to install SBCL, Quicklisp, etc would be very cool. I'll open an issue for that.

@PuercoPop
Copy link
Collaborator

I've pushed some preliminary changes to the cli-commands branch.

A couple of things:

  • I changed load-config to not call discover in order to be able to override the config file. Now load-config does one thing less. Another option would be to add a key/optional parameter parameter to load-config with the path of the config file.
  • I added a cli.lisp for misc argv processing functions. Currently it just turns argv to to an alist. We could look at pgcharts for ideas on how to handle argv in a 'lispy' way.
  • the serve command doesn't work, it appears to be related to how cl-launch works (compiles, builds an images and then restortes with uiop:restart-image) I have to understand better how cl-launch works to fix it I think.
  • Another point work considering is if a one executable multiple commands (a la git) would enable for commands to be composed for other ones (ej. rebuild)

Anyhow, thoughts/feedback?

Happy Holidays

@kingcons
Copy link
Collaborator

Awesome work, @PuercoPop! 👍 I skimmed it and it looks good on the whole. I feel like process-parameters could be cleaner for some reason but haven't thought about it too carefully. Will try to test it out and report back in the next few days. :)

@PuercoPop
Copy link
Collaborator

I've pushed some more work to the cli-commands branch, rebasing to current master. I've switched from cl-launch to buildapp. Also using command-line-arguments to parse argv. however because I'm using a 'command' cli interface (a la git) we can't use the auto-generated usage. I've yet to evaluate if Clon handles the usage case better. Following pjb's writting style of using the documentation option of defpackage I've written the help text of the command in the defpackage form, I'm not convinced it is the best idea, maybe just using the docsting would be better? Also I've followed a one-package per-file (which I don't really like tbh because it makes working with the REPL a little bit harder) but could use the traditional one packages. file used in coleslaw. Anyhow below I list the items that need work before a merge, feel free to add what you think is missing and any feedback.

Things I want to do before merging

  • Remove cl-launch scripts
  • Evaluate if clon is more suitable that command-line-arguments
  • The Makefile to work on CCL
  • Serve to work on CCL (find the equivalent of (sb-impl::toplevel-repl nil)
  • Polish on the usage/help text.

It also addresses #77.

@kingcons
Copy link
Collaborator

kingcons commented Jul 2, 2015

@PuercoPop: Quite excited to have a look this weekend. Thanks! :)

@kingcons
Copy link
Collaborator

@PuercoPop: Great job getting buildapp set up and making solid progress on the cli-commands branch. I definitely prefer the buildapp version to cl-launch. As noted, the config logic is a little messy. That probably just means it could use a rewrite anyway though. :)

The one other thing that bugs me is that the current plugin system interacts badly with binary deployment. In an ideal world, I'd love to be able to have a build farm kick out binary releases that users could just download, add configuration, and start blogging. Plugins must be compiled in at build time though. I'll think more on this but I we definitely should not block the initial release because of that.

@PuercoPop
Copy link
Collaborator

One way to handle plugins would be to make a proper asdf definition for each one. And an :after load-op method could then read a variable to figure which modules to load[1].

[1]: The variable would be cl-user for, as eclipse the wm, or a coleslaw.config package.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants