Define site structure in code (fields, sections, globals, etc.)

[trsh]

Clearly Craft CMS is not a thing for non-developers (like WordPress or Drupal is). Despite the option to create fields, sections, etc. in the administration panel, we still need to code templates (twig, html, CSS, etc.). So for me it's kind of a weird relationship where we define those things in administration with ease and luxury (no programming at all), and then still need to code standard way upon it in templates.

In my opinion, this relationship creates a bunch of issues:

1. Let's say we work in team and some other dev decides to remove some fields (or change handle names), because he/she does not know my part in templates. The page is screwed! And it's hard to rollback, because, well it's database.

2. Content replication problem! Thanks god there is something upcoming in this area https://craftcms.com/news/craft-3-content-migrations, but still this is fishy - read on.

3. Request to DB are made for content definitions, that mostly stay the same all the time. No need for that.

The solution I see, in my opinion, is very simple.

Let programmers to define fields, sections, globals, etc. (most of the settings contents) programmatically. If we know how to code with Twig, it shouldn't be so hard to edit some Yaml or PHP array. There is nothing to lose!

After that:

1. No request to DB for code defined stuff. Huge performance gain.

2. There is no content replication problem for programmatically defined settings. It is just delivered via version control!

3. We can create unit tests, do code reviews, etc. for those content parts. Even extend to more functionality ... and always refer to version control (revert/analyze bad changes etc.).

Am I talking bananas? BR, J

[narration-sd]

Welll, maybe not bananas -- possibly avocados?? As some people have taste for them more than others.

I suspect Craft is moving towards having the situations you're interested in covered, but could definitely use the input of persons such as yourself to tune this well.

Let me briefly explain, but first frame the discussion in this way: that there is always going to be a need for knowledge in those responsible for potentially breaking acts, and that this knowledge must be shared across boundaries. We can just help that along with abilities to make boundaries -- and nice abilities to communicate and cross them.

I.e., you just can't tell creative information designers that they can't arrange the project information, but must follow what a programmer says is the way, as you appear to propose.

Both of you need each other - -the key is that this kind of designer is in good knowledge and in good communication with you about potential consequences, that they have privileged ability to make changes that are non-breaking, and that you have an agreemennt together how to handle potentiallly breaking ones, when they must come up, as will definitely happen in a real-world dynamic circumstance of a useful (and well-used) application.

So, what's available to handle this, and what's coming or could be improved? Where could you helpfully propose additions or shaping for features?

• Craft already has a very nice and finely-grained permissioning system -- check out Settings>Users on a try-out Craft Pro instalation (free to immediately have with a button click, from the gear at the bottom of the CP screen on a .dev etc. environment). You can use such permissions to allow persons access according to those responsibilities/knowleddge/communication areas.

When you look this over, I think you'll find that t's not yet set up for what you need, so far concerning itself mainly with content-by-location and authorship editing privileges. It does have an interesting property of permissioning access to plugins, so that elements manipulable by plugins can be assigned.

It would likely be quite straightforward to extend the permissioning system to affect ability to alter Sections and Fields -- and as in other areas, break that down so that for example altering size of fields and their instructions might be permitted, but reserving ability to change their field type. Notice how your possibilities often expand when you click to allow abiliteis in a broad area: now you see the finer-grained details you can also control.

Given where you are speaking from, you might be a person to kick off or illuminate discussion with a feature request that gave specifics and use cases for the additions you think would build the ability to be safe yet flexible, giving those information architects only the responsible abilities.

• As you note, content migration is surely going to be key to allowing more information architecture flexibility without large consequences than is now the case. This is a difficult subject of course, and I would suggest with your real-world needs and experiences, you could be one of those contributing to the shape this long-held proposal will take in implementation.

Ok -- a little formal; what comes of a particular moment on a Saturday, but hope it's giving you some good ideas anyway, @trsh ;) I think we'd all look forward to your contribution.

Best,
Clive

[trsh]

Heh. For start than I'll try to come up with some clearer proposals & user cases. Will also try to explain why permissioning system isn't always a solution.

[daltonrooney]

I agree, this is one has hindered our developer workflow a couple of times. Especially would be nice to have for this reason:

There is no content replication problem for programmatically defined settings. It is just delivered via version control!

Also agree that the ability to easily rollback is really nice.

Surprisingly, this is one of the things Advanced Custom Fields for WordPress does well. In addition to configuring fields in the admin, a json configuration file is generated which can be included in version control. When you pull the latest changes, if the timestamp in the json is more recent than the timestamp in your database defined fields, the json takes precedence.

Sometimes I just edit the json file directly and update the timestamp, it can be quicker and more direct than using the admin, and the changes are synced between all of our developers and servers the next time they do a git pull.

[trsh]

@daltonrooney and then there can be conflicts. Work separation is not always an 100% option. Version control solves this, sometimes even automatically.

[RitterKnightCreative]

I know there's a few WP plugins that handle syncing fields / content , etc between environments. P&T have at least acknowledged this problem, would be interesting to see how they might solve it in a future update.

To narration's point, you're really asking for a technical solution to solve what's really a human leadership / process problem. Why are developers adding fields in the middle of development? Fields are really part of content strategy and should change infrequently. At least that's the goal.

We handle this very early on when we're working with the client on content. Content comes first, not development. It's like db schema, you can't create a schema until you know what you're going to be storing.

Clear communication amongst your team is the best answer. As far as VC is concerned, maybe you branch when your content strategy changes? "Client changed their mind about storing X / separate field required." so your templates don't break when asking for a field that doesn't exist.

But could Craft help us here? Sure! Would be interesting to see what you have in mind.

[trsh]

@RitterKnightCreative

1. Clear communication, rules, leadership in team is not always a 100% option. For example, in community projects, where a lot of people contribute simultaneously. Especially if it's non-profit/volunteer project, you can't just say -well we are Aliens and will do this like that. And, anyway, adding complex deployment rules, versus technical solution - you know who wins?

2. Still, versioning helps a lot. Code reviews, merges, rollbacks. You can can of course version/review also a migration script, but in practice it's very complex.

3. There are some plugins, yes. But non of them work 100% well. Some of them are un-maintained, etc. Should be in core package and always updated with new Craft version.

4. Database, in my opinion, should be used, only where it's needed. Something that changes frequently or/and by editors. In my current project, for example, fields ar 100% changed only by programmers, and there are permissions, that no other's can do that. So what's the Point in involving Database?

What I would like to see (very abstract) is, some Yaml/json/whatever config, where I can define Fields (etc.), and those are system defined ones, and can't be changed from admin panel.

[narration-sd]

trsh, this is still coming across my plate, so a few comments

• first, you may not be realizing how efficient a modern database is,, for the many small pieces of field config data. Both data and queries are the sort of thing that will remain in caches, and in fact be much more speedy to access than re-parsing yaml or other text files.

• I see you're trying to exercise control over a bunch of unruly gamers (looked at background). For that, you need permissioning. This is what Craft does in its CP access controls, and I think you already noticed it would handle permitting only certain persons to modify fields. User groups are a great way to assign (and retract) easily, and there are many safeties to keep you from making errors.

• json or yaml are hardly ideal formats. One doesn't allow comments, or safety commas, for example. The other is designed to be 'too simple', and so is not simple at all, especially when it comes to managing strucures of data -- you will find yourself visiting yaml lints often, and puzzling about what was happening just because you or your text editor altered a spacing or punctuariion -- or...quoting (bad news there)..

• you mention rollbacks. Well, databases are great on several levels for that. The usual way in development, though, is to keep as many backups of database structure with content as you do source changes. You do that in text files dumped by database utilities, and you use the same Git etc. repository to contains (and align...) both. Watch how long they take to dump or read in, to get an idea about performance issues with text files, as a side note.

• if you're uncomfortable with databases, the answer is probably to get more familiar - we all do such; it''s part of the process of constant learning which is central to working with software, and never stops. This will also help for understanding about why complex, inter-related data such as that which operates Craft is much better and more efficient also in a database, which keeps those relationships straight. That's what you effectively can't do with text files. Trying to use them would really limit the practical scope of what Craft can do, now and in the future.

• more familiarity this way will also teach why it's quite a challenging proposition to create an arrangement to manage content-with-structure transitions. I think this will come, though -- and that Craft is the place to expect it.

Best fortune on your project/s, trsh, and be sure that guiding any group of individuao tendencies is challenging, formal projects, or community efforts. Really the issures are the same, and most related to tendencies for independence which dependably with some discomfort also generate the new approaches. Giving people a playground, or asking them to make their own to demonstrate something, separately from the main project, is one pretty effective practice, and may work for you.

[jordanlev]

Fwiw, I think storing the field schema in a file would be immensely helpful in terms of having that configuration in version control and making it much easier to deploy local changes to a stage and production environment.

To me it's not really about preventing clients from messing with the configuration (because that can be managed via permission), but really about being able to track changes and share such changes across a development team.

@RitterKnightCreative says:

Why are developers adding fields in the middle of development? Fields are really part of content strategy and should change infrequently. At least that's the goal.
We handle this very early on when we're working with the client on content. Content comes first, not development. It's like db schema, you can't create a schema until you know what you're going to be storing.

This is very unrealistic. Development is a process, and while of course you want to try to hammer out all the content modeling as soon as possible, it's crazy to expect that it will never change once development has started. Even if you're a miracle worker about project management and can get this very old-school waterfall method to work, what happens after the site is launched and now it's time to add new features?

@narration-sd , it doesn't matter how familiar one is with how databases work... the point is that you cannot effectively put database content into version control. And when you've mingled field configuration with actual site content, it becomes incredibly difficult to extricate them from each other in a rollback (e.g. I want to roll back just a field change, but if I turn the whole database back to how it was a few days ago, I also then lost all other content that was added since then, even if unrelated to this specific field change).

I still think there's value in having a GUI in the Control Panel to manage the fields schema, but there's no reason that GUI can't be reading from and writing to config files instead of the database.

All that being said, looking at how Craft structures the database schema in the content and matrixBlock tables, it might be an impossibility given the current architecture. It looks like each field is a column in the content or matrixBlock tables, so right there is a very structural intertwining of the field schema with the database schema... moving this to a config file would only serve the purpose of migrations (not real-time "look to the config file to determine what the field schema is"), and since Craft 3 has the migrations feature anyway, this is all probably a moot point.

[RitterKnightCreative]

@jordanlev You're right, it's foolish to think a site is never going to change. And I agree that some sort of export and import via the file system would be handy. Craft 3's migrations should come in handy for this.

But reread my quote again; I mentioned fields should change infrequently not never. My point was that if you're changing your content strategy enough for it to become a big problem or bad communication is tripping others up, that's not really a Craft problem, that's a process and leadership problem. As someone who's sat on both sides of the hiring table, I admit this stuff is hard to get right so oftentimes we let the tools do it for us, which leads to problems like this. Someone screwed up? Blame the tool.

I'm sure your team has a protocol/process of when to branch, when and how frequently to check files in and out, deadlines and milestones, etc. Version Control exists to minimize some risk but you can certainly still break that system too.

[jordanlev]

@RitterKnightCreative Even making a small one-off change is annoying once a site is in production. Assuming a change requires both modifying the field schema and the code (e.g. adding a field to an entry type and then modifying the Twig template to output it), you only have 2 choices, neither of which is ideal for a modern dev workflow:

1. Make the changes locally as you develop, and write down on pen and paper the "recipe" to recreate the steps you made (e.g. "add this field with this type and this exact handle and these exact options, then assign that field to these entry types in these sections"). Recreate those steps on the test/stage and then production servers at the same time you bring in your code changes associated with them.

2. Make the field changes to the production DB, then copy that down to your local dev machine so you can then work on the code updates. When done, do your normal code deployment... the field changes already exist on production (but heaven forbid a client saw that and started using it before you got the update in there).

This is a sub-optimal workflow. Sure one should try to minimize it, but it happens. Why should we humans have to bend over backwards to satisfy the requirements of the CMS instead of making the computer do the work for us?

[narration-sd]

@jordanlev seems it would be relatively straightforward to have Craft able to emit a structure migration for such a Section and relations change, easy then to check that in along with the twig template changes.

Thus it's the database as ever properly managing the data and its structure -- and its effectiveness advantages continue.

This covers your point, but leaves open still the usefulness that given degrees of content migration could have, when they can be worked out.

It's still always going to take some care in communication and cooperation, as @RitterKnightCreative says.

[jordanlev]

@RitterKnightCreative and @narration-sd -- thanks both for the thoughtful replies. Totally agreed that communication and cooperation must always be part of the equation, but I just wanted to add my 2 cents to this conversation that there is a very real tactical/technical aspect to this.

Regardless, Content Migrations will likely solve the problem as best as it can be solved given Craft's database structure. It looks like right now in the Craft 3 beta it's just a framework to be able to write the migrations, which is a good start -- hopefully in the future there will also be a way to automatically emit the structure changes (as you suggest).

[brandonkelly]

We have taken a small step toward implementing this in Craft 3, which supports a config/volumes.php file that can override volume settings. You still need to create them in the CP first, but once they exist you can change all of their settings (even the volume type if you want) in code, on a per-environment basis.

It clearly hasn’t been enough, as many people have had a hard time understanding that the file is only there to override settings; not to define the volumes in the first place. So eventually we should go with the flow and find a way to fully implement this request.

One tricky thing though is, how do we handle renaming fields/sections/volumes/etc.? If you just rename a field handle in the config file, for example, how does Craft know that an existing field had been renamed, and not that a new field was created and an old one had been deleted? Maybe the keys for these components would need to be something like UIDs rather than handles, where you wouldn’t have any reason to rename it, but that would make for a pretty ugly config file…

[drifteaur]

This was one of the major initiatives for Drupal 8 - all of the configuration is importable and exportable. It was a huge effort and a big can of worms - deciding what's considered configuration vs. what's considered content, getting UUID support in (it was a prerequisite), deciding on JSON/YAML/XML format (YAML won out), translating configuration...

That said, it is a huge win, experimenting with structure on dev and then moving it into production, especially when several developers are working on the site.

Here is some of the archived discussion:

https://groups.drupal.org/build-systems-change-management/cmi

If you ever decide to go ahead with this, I'd be happy to summarize the process that Drupal went through.

[RitterKnightCreative]

how do we handle renaming fields/sections/volumes/etc.? If you just rename a field handle in the config file, for example, how does Craft know that an existing field had been renamed, and not that a new field was created and an old one had been deleted?

An interesting take on this is how, for example, WordPress handles slug/URIs. (On the very short list of good things I will say about WP...) If you rename a slug, WP actually keeps track of the old URL and puts it in its redirection database. Any requests for the old URL will redirect to the new location automatically so the old URI doesn't break. (It's clever but some sort of feedback/GUI would have been nice to see as well. FWIW, Craft should do this... but that's another request.)

I wonder if something like this could be done for fields? Craft could check "does this field exist? No? OK let's check the (renamed/aliased) fields database." (Maybe a FK database relationship.) At that point, if Craft does find a match, it could also log the problem and/or throw up a "deprecation" notice that "hey dev, you should really consider changing your code." I think that would effectively mitigate the need for at least a dev to handle some sort of UUID by themselves.

I would imagine this would make the code for creating a new field in the CP a bit harder since now Craft has to also check an aliases table. However, there's been some requests for the ability to simply rename a field's name (to the user) and use it elsewhere in the system.

Seems like both systems could potentially tie together, the only difference being one is used mostly for dev and the other is mostly used for the author / CP. On the field's layout, you might have a "Also known as..." alias that basically lets you rename both the id and the name shown in the CP. Like it does with the General "Site URL" and other settings, Craft could also show that "this field is also being overriden or defined by a config file..."

[jordanlev]

Re: UUID's making for an "ugly" config file... The primary purpose of the file shouldn't be niceness for human editing -- rather it's a way to encode configuration in versionable / diffable / deployable text files. If the files are being generated by the CP anyway, I don't think the individual contents of a setting matter much. (Not to mention that ugly is subjective -- I happen to think UUID's are beautiful in their own way :)

…

On Aug 9, 2017, at 9:34 AM, Brandon Kelly ***@***.***> wrote: We have taken a small step toward implementing this in Craft 3, which supports a config/volumes.php file that can override volume settings. You still need to create them in the CP first, but once they exist you can change all of their settings (even the volume type if you want) in code, on a per-environment basis. It clearly hasn’t been enough, as many people have had a hard time understanding that the file is only there to override settings; not to define the volumes in the first place. So eventually we should go with the flow and find a way to fully implement this request. One tricky thing though is, how do we handle renaming fields/sections/volumes/etc.? If you just rename a field handle in the config file, for example, how does Craft know that an existing field had been renamed, and not that a new field was created and an old one had been deleted? Maybe the keys for these components would need to be something like UIDs rather than handles, where you wouldn’t have any reason to rename it, but that would make for a pretty ugly config file… — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[brandonkelly]

This is coming in Craft 3.1. Announcement here: https://craftcms.com/news/craft-3-1-dev-preview

[timkelty]

project.yml The best news!