deprecate pcreate and update scaffolds to cookiecutter

[mmerickel]

[edited by @stevepiercy to insert checklist]

• add deprecation notice in pcreate
• update docs/designdefense.rst This is OK, as we still ship scaffolds with Pyramid, but should be updated when we ultimately remove them.
• update docs/glossary.rst
• update docs/index.rst
• update docs/latexindex.rst
• update docs/typographical-conventions.rst
• add docs/narr/cookiecutters.rst to immediately precede scaffolds.rst
• update docs/narr/assets.rst
• update docs/narr/commandline.rst
• update docs/narr/extending.rst
• update docs/narr/i18n.rst - deferred. See convert scaffold to cookiecutter pyramid_jinja2#139
• update docs/narr/install.rst
• update docs/narr/introduction.rst
• update docs/narr/logging.rst
• update docs/narr/myproject/* source files and wherever it is literincluded
• update docs/narr/paste.rst
• update docs/narr/project.rst
• update docs/narr/scaffolding.rst
• update docs/narr/startup.rst
• update docs/narr/testing.rst
• update docs/narr/webob.rst
• update docs/tutorial/modwsgi/index.rst - deferred. See Update docs/tutorials/modwsgi.rst for cookiecutters #2890
• update tutorial steps, replacing invocation of scaffold with cookiecutter, in the following:
  • docs/tutorials/wiki/*
  • docs/tutorials/wiki2/*
  • docs/quick_tutorial/*
  • docs/quick_tour*
• Change scaffolds to use Jinja2 instead of Chameleon
  • starter
  • alchemy
  • zodb Need to update PR Move to jinja2 #2798, removing this one part of @palindromed's effort
• Duplicate zodb cookiecutter, rename copy to pyramid-cookiecutter-zodb-chameleon, retain it in perpetuity
• Change cookiecutters to use Jinja2 instead of Chameleon
  • https://github.com/Pylons/pyramid-cookiecutter-starter
  • https://github.com/Pylons/pyramid-cookiecutter-alchemy
  • https://github.com/Pylons/pyramid-cookiecutter-zodb
• Replicate Pyramid's scaffolds/tests.py to run on cookiecutters Moved to a new issue Replicate Pyramid's scaffolds/tests.py to run on cookiecutters #2868, so as not to hold up 1.8.
  • https://github.com/Pylons/pyramid-cookiecutter-starter
  • https://github.com/Pylons/pyramid-cookiecutter-alchemy
  • https://github.com/Pylons/pyramid-cookiecutter-zodb
• Configure Travis-CI to run scaffold tests on cookiecutters
  • https://github.com/Pylons/pyramid-cookiecutter-starter
  • https://github.com/Pylons/pyramid-cookiecutter-alchemy
  • https://github.com/Pylons/pyramid-cookiecutter-zodb

---

This is for a few reasons.

1. Cookiecutter supports full-blown jinja2 templating with conditional logic. This is much better than pcreate's stupid simple string replacement algorithm.
2. There is a bootstrap issue with the scaffolds right now which is a little weird.

1. create virtualenv
2. `pip install pyramid`
3. `pcreate -s starter myapp`
4. cd myapp
5. create virtualenv
6. `pip install -e .`

It's possible to re-use the original virtualenv but it can make the filesystem layout differently for someone cloning the repo versus someone creating a new project if you do. The goal should be:

1. create myapp starter
2. cd myapp
3. create virtualenv
4. `pip install -e .`

So now we always create a virtualenv from the same layout... a cloned or newly created project.

There is a big CON that maybe could be coordinated with the cookiecutter team. The hooks system is super simplistic in cookiecutter and does not seem to allow much runnable code. This means lots of logic we normally do in python would be put into the jinja2 templates. I feel to truly consider cookiecutter this would need to have a better story.

[pauleveritt]

@cguardia did some work on a cookiecutter template for Pyramid:

https://github.com/cguardia/cookiecutter-pyramid

He wound up filing a cookiecutter ticket about automating the creation of a virtualenv (behind an optional argument.) CookieCutter turned him down, but he did it his own way, although I disagreed with him (he required the virtualenv package to be installed to get Py 2/3, I suggested to ignore Py2 for the flag and just automate pyvenv.)

Anyway, details details. I very very much support looking at throwing our weight behind CookieCutter and the idea that you can install using recipes that aren’t pip-installed. In fact, I plan to lobby PyCharm at some point to make it a first-class experience.

—Paul

On Feb 28, 2016, at 6:35 PM, Michael Merickel notifications@github.com wrote:

This is for a few reasons.

Cookiecutter supports full-blown jinja2 templating with conditional logic. This is much better than pcreate's stupid simple string replacement algorithm.

There is a bootstrap issue with the scaffolds right now which is a little weird.

1. create virtualenv
2. pip install pyramid
3. pcreate -s starter myapp
4. cd myapp
5. create virtualenv
6. pip install -e .
   It's possible to re-use the original virtualenv but it can make the filesystem layout differently for someone cloning the repo versus someone creating a new project if you do. The goal should be:
7. create myapp starter
8. cd myapp
9. create virtualenv
10. pip install -e .
    So now we always create a virtualenv from the same layout... a cloned or newly created project.

There is a big CON that maybe could be coordinated with the cookiecutter team. The hooks system is super simplistic in cookiecutter and does not seem to allow much runnable code. This means lots of logic we normally do in python would be put into the jinja2 templates. I feel to truly consider cookiecutter this would need to have a better story.

—
Reply to this email directly or view it on GitHub #2384.

[mmerickel]

Another little nit is that it doesn't appear there's a way to host multiple cookiecutters from the same repo via subpaths so we would have to extract them from pyramid.git into their own repo and then try really hard to keep them in sync/working with new versions of pyramid (our tox.ini already can do this so it's not so bad). Unless we went to the effort that @cguardia did of making a single cookiecutter that hosts multiple project types.

[mmerickel]

Also, the work done by @cguardia in that repo is excellent and answers several of my other questions about using cookiecutter that I hadn't found in their docs.

[amol-]

@mmerickel I'm currently maintaining the turbogears2 project and as Pyramid and TG2 share some legacy over Pylons we faced the same concern in the past.

At the time we started porting TurboGears2 to Python3 I got in touch with Chris McDonough about the possibility of sharing a common project for what's pserve, pcreate and so on between Pyramid and TG.
In the end we took separate ways, but as you are considering a replacement for pcreate and so on I thought it might make sense to discuss the opportunity with you.

During the effort of moving the TG stack away from Pylons and porting it to Py3 we created the Gearbox project ( http://github.com/TurboGears/gearbox ) which in fact includes most of the features Pyramid p* suite of command provides. One interesting point is that the template based commands of Gearbox should actually be compatible with the currently existing Pyramid scaffolds both in terms of templates and directory structure (as both come from the Paster legacy) but they provide support for more advanced constructs like conditionals and loops.

That would allow to share not only the overhead of maintaining the scaffolding system but also pserve and pshell commands can be shared as both the frameworks use the PasteDeploy convention.

Off course I'm open to discuss any change gearbox might need to integrate/satisfy pyramid community needs but I think the idea is worth pursuing as it would only result in a better tool that can share the maintenance effort, bug reports and testing between the two communities and maybe lead also to a wider adoption outside of our communities in the future.

Any feedbacks is well appreciated!

[mmerickel]

Hey @amol- that's great! I hadn't heard of gearbox and will make a note to take a look. We're thinking about this stuff but it is mostly just thinking at this point. pcreate is one thing we would like to replace but the other p* commands fall into a separate discussion we're having on reducing our dependency on PasteDeploy ini files. You can see a little bit of that in #2419.

[amol-]

@mmerickel Well Gearbox relies on PasteDeploy mostly for the loadserver and loadapp functions to actually get the server and app that should be served, but has no specific constraint over PasteDeploy.

By the way I think that PasteDeploy (with all its limits) has a great advantage of being wildly supported by applications servers and supervisors like uwsgi or circus. And .ini files are usually appreciated by operations for the fact that it's simple to append options through scripts (just replace [section] with [section]\n option=value and you are done) while more structured formats like JSON require a parser.

Maybe it can be considered to improve it to resolve some of its limits? For example CherryPy took the solution to use .ini files with typed values or nest them with [parent/child] syntax.
Didn't investigate much in this direction, but I suppose it should be fairly easy to maintain PasteDeploy compatibility while supporting those features.

I think also it should be fairly viable to support other formats in gearbox, if we can came up with a common one it might be easier to spread it over other tools again and have application servers relying on it.

[cguardia]

We now have repositories for the three original scaffolds using Cookie cutter. The postgen hooks proved enough to create the venv without adding any code to the jinja2 templates, and even run initialize_db in the alchemy template. We also run the pip --upgrade and the pip install -e from the hook.

https://github.com/Pylons/pyramid-cookiecutter-starter
https://github.com/Pylons/pyramid-cookiecutter-alchemy
https://github.com/Pylons/pyramid-cookiecutter-zodb

[stevepiercy]

I started a checklist at the top of this issue. Please add any other items that need to be done.

[pauleveritt]

Good job @stevepiercy on the checklist. If we change the Quick Tour from scaffolds to cookiecutter templates, and point at the jinja2 template, then we need to change the quick tutorial to use jinja2. That might be a lot for closing this ticket. Should we split into several phases?

[stevepiercy]

I updated the checklist in preparation for the PloneConf 2016 Pyramid sprint.

@pauleveritt I think it is easier to track this whole thing in a single issue with a checklist, instead of several issues. Folks can take a bite size piece of the whole thing—say, Step 5 of Quick Tutorial—and submit a PR for that to master branch, while referring to this issue as the central hub for tracking.

We can also break out the checklists for quick_tutorial and quick_tour as we go through each step along the way for more detailed tracking.

[pauleveritt]

The PloneConf sprint is using material created by Cris Ewing, not the quick tutorial.

[stevepiercy]

@pauleveritt how does Cris Ewing's material affect what is done at the sprint for Pyramid at PloneConf? I am not familiar with Cris's material.

[pauleveritt]

My bad, you said sprint but I thought tutorial.

—Paul

On Sep 30, 2016, at 6:47 AM, Steve Piercy notifications@github.com wrote:

@pauleveritt https://github.com/pauleveritt how does Cris Ewing's material affect what is done at the sprint for Pyramid at PloneConf? I am not familiar with Cris's material.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2384 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAbx5gey6QxCT02u2ve1T-3eFZ-FyXayks5qvOjZgaJpZM4Hk_E2.

[mcdonc]

FTR, I mentioned this to @stevepiercy at a sprint, but as a reminder, while it should be no problem to remove the concrete scaffolds from pyramid itself at any time (alchemy, zodb, starter, etc), the pcreate command and the machinery to support it should hang around for a while and emit deprecation warnings because things like substanced actually rely on registering a scaffold (e.g. https://github.com/Pylons/substanced/blob/master/setup.py#L108) and instruct people in docs to use pcreate to bootstrap a project. Maybe instantiation of a pyramid.scaffolds.PyramidTemplate starts to emit a deprecation warning, just to signal to folks that the feature is going away.

[mmerickel]

Of course, there's no plan to remove pcreate for a while. Substanced should, however, offer a cookiecutter. The cornice folks added one for cornice [1][2] in just a couple days after I warned them that we were deprecating it, so I'd hope substanced could do the same.

[1] Cornices/cornice#388
[2] https://github.com/Cornices/cookiecutter-cornice

[stevepiercy]

Substanced should of course offer a cookiecutter.

@cguardia did this at the PyCon sprint.
https://github.com/Pylons/substanced-cookiecutter

[stevepiercy]

FTR, I put in a deprecation warning in
80b7e89

@mcdonc @mmerickel Did I miss any items for the checklist at the top of this issue?

[pauleveritt]

Agreed. I think it’s Pyramid’s style in general to leave stuff in, but deprecated, for a few releases.

—Paul

On Oct 23, 2016, at 11:53 AM, Chris McDonough notifications@github.com wrote:

FTR, I mentioned this to @stevepiercy https://github.com/stevepiercy at a sprint, but as a reminder, while it should be no problem to remove the concrete scaffolds from pyramid itself at any time (alchemy, zodb, starter, etc), the pcreate command and the machinery to support it should hang around for a while and emit deprecation warnings because things like substanced actually rely on registering a scaffold (e.g. https://github.com/Pylons/substanced/blob/master/setup.py#L108 https://github.com/Pylons/substanced/blob/master/setup.py#L108) and instruct people in docs to use pcreate to bootstrap a project. Maybe instantiation of a pyramid.scaffolds.PyramidTemplate starts to emit a deprecation warning, just to signal to folks that the feature is going away.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2384 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/AAbx5v1zEyCqZEKAB49RiEKk8a9nNgL-ks5q24MCgaJpZM4Hk_E2.

[stevepiercy]

Updated checklist

[mmerickel]

@stevepiercy I'm looking at what needs to be done on this and I see that you have a pcreate-to-cookiecutter branch. How can I help with this? Should we open a PR for it yet?

I started looking at the scaffolds themselves and have a couple issues so I'll just push commits to them directly. As far as gettings tests running on the scaffolds I'm starting to think about that and then I imagine there's a bit of work to be done on updating the tutorials.

[stevepiercy]

@mmerickel pcreate-to-cookiecutter is the correct branch for this issue.

There's PR #2798 which needs updating. I don't know the best way to undo the zodb scaffold work that @palindromed did at ploneconf.

I'll be returning to this issue after I finish #2633, and I can easily handle updating the docs.

I think "Replicate Pyramid's scaffolds/tests.py to run on cookiecutters" and "Configure Travis-CI to run scaffold tests on cookiecutters" would be good for you to work on. Tests and Travis are not my strength. The last piece, "Change cookiecutters to use Jinja2 instead of Chameleon", I can do, too, but you're welcome to have at it, if you're itching to go.

[mmerickel]

All of the cookiecutters have travis and appveyor setup to automatically run their test suites.

[zupo]

How can I help with this one? Where can I pick up the work already done?

[stevepiercy]

@zupo see the checklist above and my recent comment

[stevepiercy]

I'll start work on these items this week:

update tutorial steps, replacing invocation of scaffold with cookiecutter, in the following:

• docs/tutorials/wiki
• docs/tutorials/wiki2
• docs/quick_tutorial/*
• docs/quick_tour*

Is anyone working on the remaining items in the checklist at the top, or are they complete?

[stevepiercy]

I added a new task in the checklist at the top, which I would like to discuss and refine its scope:

• update docs/narr/scaffolding.rst to discuss new cookiecutters, but preserve its content (perhaps add a new page cookiecutters.rst, too?)

I am leaning toward not adding a new page, unless there is something specific to pyramid_cookiecutters versus general cookiecutters. Each of the cookiecutters should have its documentation (mostly just a readme) fairly well standardized while referring to other sources including the official cookiecutter docs.

[stevepiercy]

@mmerickel each of the pyramid-cookiecutters has tests that test the generation of a project as well as the generated project. So is this item in the checklist needed?

• Replicate Pyramid's scaffolds/tests.py to run on cookiecutters
  • https://github.com/Pylons/pyramid-cookiecutter-starter
  • https://github.com/Pylons/pyramid-cookiecutter-alchemy
  • https://github.com/Pylons/pyramid-cookiecutter-zodb

[stevepiercy]

I moved the scaffolds/tests.py item into a new issue #2868

[stevepiercy]

The docs for the ZODB wiki using cookiecutter are ready for review:
http://pyramid-stevepiercy.readthedocs.io/en/docs-scaffold-to-cookiecutter/tutorials/wiki/index.html

[stevepiercy]

Work for this has been shifted to a new branch and started over at stevepiercy/docs-cookiecutter-changes-only. Rendered docs may be viewed at http://pyramid-stevepiercy.readthedocs.io/en/docs-cookiecutter-changes-only/tutorials/wiki/index.html

[stevepiercy]

I found a bunch of other files that contain the word "scaffold", and updated the checklist at the top of this issue for further review. bah, humbug!