1.8.0 release notes [draft]

[aidantwoods]

The current release notes are included below. Any improvements on the text content for this release can be discussed here. This also serves as a good summary should you wish to critique specific parts of the upcoming release – please open separate issues for any of these critiques :)

---

This is a minor release, which includes new features and bugfixes.

To complement some necessary changes added in 1.7.0 to improve the safety of Parsedown, some new features have been added to help extensions take better advantage of the AST.

Summary of Changes

Added

The following have been added:

#576: Produce AST prior to render

#511: Allow element to have no name

#569: Add rawHtml option for extensions to use on trusted input

#589: Add strict mode to opt-in to strict CommonMark compliance

Fixed

The following have been resolved:

#566: Prevent Email autolink being started by HTML tags

Resolves: #565

#514: Fixes treatment of HTML blocks

By @Daniel-KM and @aidantwoods, resolves: #488, #567, #528, #259, #407, #405, #371, #370, #366, #356

#578: Fixes an issue where setext heading would not render if trailing spaces were present on the underline

Resolves: #571

#439: Improve CommonMark mixed-marker list compliance

By @PhrozenByte, resolves: #437, #581

#574: Remove legacy escaping

Resolves: #573, #287

#583: Fix trimming of internal #'s in ATX headers

Resolves: #337, #362

#584 (+ #612): Permit 1 column tables with less delimiters and assert header count matches delimiters in delimiter row

Resolves: #415

#586: Fix merging of adjacent blockquotes

Resolves: #344

#587: Fix ordered list interrupt

Resolves: #454

#592: Fix HTML comment endings

Resolves: #288

772c919: Recognise empty ATX headings

Resolves: #595

#600: Fix fenced code block closer length rules

Resolves: #599

#610: Fix lost line breaks

Resolves: #609

#621: Tilde characters may be escaped

By @paukenba, resolves a CommonMark issue pointed out in the PR

Public API Feature Notes

Strict Mode

#589 adds a new setter to opt-in to strict CommonMark compliance.

In #265 changes were made in favour of CommonMark compliance which introduced breaking changes to previously supported syntax. Strict mode aims to side step this dichotomy between supporting a commonly used deviation from the spec, and honouring the spec.

Currently requiring a space after a hash to form an ATX header is the only case that strict mode controls, but this opens up the possibility of releasing very breaking (but compliant) changes in the future into strict mode (similar to this case) since the intent of the setter is to be compliant.

We will aim to be as compliant as possible even when strict mode is off, but if there are changes where compliance generates major unforeseen breakage in existing texts (like #265) then it is nice to be able to limit that change to those that want strict compliance.

Extension Accessible Feature Notes

Text-element Adjacency

#511 adds the ability to omit (or set to null) the name of an element to have it render its contents only. This permits plaintext to sit adjacent to elements within the AST structure.

Conditionally Escaped Raw HTML

#569 adds a new element content key: rawHtml. This can be used in place of text, if both keys are present text will be preferred. rawHtml can be used to write HTML directly into the AST, however it is strongly recommended that this is only done when absolutely necessary.

rawHtml blinds Parsedown to (and thus breaks guarantees of) the output structure. You the author are therefore responsible for ensuring that HTML does not become malformed, and also for ensuring the safe encoding (and sanitisation if also necessary) of any user-input.

By default, rawHtml will be HTML encoded (in the same way as text) if safe-mode is enabled by the user. This facilitates conditional escaping of features which may be very powerful (like allowing arbitrary HTML), but also provides an automatic mechanism for disabling these features.
If you are confident that appropriate safety measures have been applied (i.e. contextual escaping, and sanitisation of user-input) then you can mark the rawHtml as trusted by setting the allowRawHtmlInSafeMode key to true.

AST Traversal

From #576, handlers have now been decoupled from playing a necessary role in the recursion process for rendering an element. In addition to the text in an element, there are two new keys: elements, and element. Only one of elements, element, text, and rawHtml should be used as the content of an element (i.e. only specify one of these keys). Their tentative order of precedence is as written – specifying multiple is an error, and exceptions may be thrown in future.

In text Parsedown will now only expect to find a string containing text to render inside the element. In elements, Parsedown will expect to find an array of sub-elements to be rendered inside the element. In element Parsedown will expect to find a single sub-element to be rendered inside the element.

More Expressive Handlers

Also from #576, previously a block using a handler in Parsedown might look something like:

$Block['li'] = array(
    'name' => 'li',
    'handler' => 'li',
    'text' => !empty($matches[3]) ? array($matches[3]) : array(),
);

Now, this should be implemented as follows:

$Block['li'] = array(
    'name' => 'li',
    'handler' => array(
        'function' => 'li',
        'argument' => !empty($matches[3]) ? array($matches[3]) : array(),
        'destination' => 'elements'
    )
);

Handler is now an array that contains information only relevant to handler delegation, in which the following keys should all be specified: function, argument, destination. function is the function which should be called during the handling process, argument is the argument that should be given to the function, destination is the key to which the result of this call should be written to within the element.

This serves to disambiguate the handling process, in particular the use of the text key to contain: text, elements, an element, and lines in an li isn't very intuitive and can be a source of hard to find bugs as a result.

For those wondering whether the handler key may be written to as a destination – yes, handlers may be defined recursively.

For any handlers still using the old string syntax – these will continue to to work via the compatibility layer summarised in #576 (comment). Note that if a user enables safe-mode then this will cause the output of the handler to be HTML encoded. To avoid this either update to the new method of writing handlers, or if the output from your handler is safe when user-input is not trusted then you may mark the output as safe by permitting raw HTML in safe mode for the elements which use this handler.

[aidantwoods]

PRs welcome for the above text :)

https://github.com/aidantwoods/parsedown/blob/1.8.0-release-draft/Release-Notes.md

[dregad]

So the beta's been out for a month now, and I was wondering if there was an ETA for the stable release, hopefully sometime soon, or if you are planning to cut another beta before that (considering that there's been a few commits on master since April 5th).

I have not encountered any issues with the beta code so far, and I have to say that I really like being able to manipulate the AST. This allows elegant solutions that were much more difficult to implement with earlier code. Nice work ! 👍

[aidantwoods]

I've updated the above draft release notes to reflect the latest changes, and have released another beta so that the latest changes are available for testing.

As mentioned in #580 (comment), I'm still not entirely sure about releasing this as 1.8.0 (despite there being no public API breakages, there is likely to be inevitable breakage to extensions that rely on observed internal behaviour).

If this goes ahead as 1.8.0, the latest beta release would essentially be a release candidate, with not much blocking the release other than a period for allowing people to test it. If this is to be 2.0.0 then it'll likely be a bit longer – unfortunately I won't really have the time this month to work on the other changes that would be worth putting into the next major release.

---

On making this a minor release, I think it is certainly "fair game" due to not breaking the public API. Extensions that break will have in theory always have been relying on behaviour not in the public interface, and IMO on internal behaviour that shouldn't really reasonably have the expectation of being static:

In order to assess anticipated breakage, I'll sum up what has changed:

1. Handlers are now arrays. This is backwards compatible: handler strings still work – string handlers are automatically converted to their equivalent representation as a handler array (i.e. string specifies function, they take their argument from the text key, as before).
2. The "correct" way of outputting markup is via the AST, carried inside elements. And the special cased markup key no longer bypasses the AST. This is backwards compatible: the special cased markup key for blocks and inlines also still works as it did previously (provided safe mode is not enabled) – it is now interpreted as a shorthand for an element with rawHtml.
3. The element structure returned by block* and inline* methods.
4. (added by edit :) ) The special block or inline key "hidden" which was used to avoid needing to provide a renderable result. This is backwards compatible: in the AST representation, we just translate this to an empty element.

As far as I know, there is nothing else that could cause breakage (please feel free correct me on this though).

1&2 are backwards compatible with safe mode disabled (1&2 may/will result in escaped HTML is safe mode is enabled though – this is resolvable if the extension marks the markup as trusted). 3 is not backwards compatible, however IMO this isn't a reasonable thing for extensions to expect to remain the same across versions (the structure could change for a lot of reasons – e.g. even a bugfix might have changed this).

I'd be interested to hear feedback on whether or not it seems reasonable to make this a minor release with the above considered.

[dregad]

Thanks for the update !

As far as I'm concerned a minor release should be acceptable, but I'm equally fine with a major version bump if it helps your conscience feel better 😉

[dregad]

Did you reach a decision on whether to make the next release major or minor ? Any ETA ?

Until then, are you going to cut 1.8.0-beta-5 to include #641 soon, or should I just test against master ?

[aidantwoods]

I've added a 1.8.0-beta-5 release including that fix. As far as actually releasing it goes, I think (as @PhrozenByte said in one of these threads): it's probably cleaner to just make the next stable release major and thus explicitly breaking. Since 2.0 is something that would be nice to have anyway with regard to being able to make some breaking decisions, I'd also rather people didn't have to update stuff twice (i.e. potentially once after 1.8 and then again quite soon after to go to 2.0).

Likely next week I'll be able to finally put some more time into stuff over here again, I'm not sure how long it would take to get to 2.0. I don't think it'll take a huge amount of time to make the code changes, but I'd expect to allow at least a few weeks with a 2.0 beta perhaps longer depending on how much feedback there is. For me the primary goal is to permit custom behaviours to be added in a way that is backed by an interface (so we can avoid building a dependence on observed behaviour like we have now).

---

I'd expect 2.0 to be able to avoid this problem: #534 :)

[dregad]

That's fine with me. Thanks for your support!

[markseuffert]

Thank you for the information. 👍

I am new with Parsedown and haven't found any problems while testing. The AST made it very easy to understand Parsedown's design and write extensions. It's great that you have many possibilities to manipulate the AST, you can add elements, add raw HTML or let the internal parsing continue. A far as I know, Parsedown Extra has some problems and bugfixes have been pushed back for a while.

Do you have any plans to make a release? Do you need help with something?

[dregad]

@aidantwoods next week has come and is long gone, and so I was wondering if you ever got around to deciding between 1.8.0 and 2.0, and if going for the latter, if there is any planned date for release.

[aidantwoods]

I apologise for leaving this hanging for quite a bit longer than anticipated. To answer your question, the next version will be 2.0, although I'm not entirely sure of a date for that yet. Something I want to be sure of before reaching 2.0 is that we get the future of "extensions" right in a way that they have sufficient freedom to add custom behaviour as needed, but also so that these custom behaviours can be used together and that there are clear (ideally somewhat user malleable) precedence rules about what happens if there are conflicts. I have some ideas towards this, but didn't really get round to experimenting with them unfortunately.

Going forward my spare time should be a little more predictable, so I hope to start getting some more commits up in the coming days :)

[xenocrat]

@aidantwoods Firstly I want to write: thank you for your contributions to this project. I'm very happy to see the progress you've helped make to Parsedown!

What I'm less happy about is the ongoing beta state of 1.8.0. I understand that life happens and plans get waylaid. All your reasons for moving directly to 2.0 are good and sensible, but the – unfair, I know – question is: how long? A stable release with a beta designation is a frustrating situation for a downstream integrator.

[stof]

@aidantwoods could you tag a 1.8.0 release to make all the fixes available to projects using stable releases ?

I would say that if you discover the need for more 1.x changes while working on 2.0 to prepare extensions, you could still decide to ship a 1.9.0 release with these changes rather than delaying 1.8.0 until 2.0 is ready. Even though your current plan is to have 2.0 being the next release, changing that plan to add an extra 1.9.0 release before it would not have much impact on the ecosystem.

[jamescridland]

One thing ('extension'?) that v1.8.0 has broken is Parsedownfilter, which now kicks up an error:

Undefined index:name in ParsedownFilter.php on line 29

I don't know how to get in touch with the original author of this.

[aidantwoods]

It looks like this is caused by the addition of text-element-adjacency, which makes the name key in an element optional or nullable: so when the extension above tries to read from that key, it finds it doesn't exist in some cases.

In future instead of using an array to try to encompass everything within the idea of an "element", I planned to add a Renderable interface, which seperate types Element, Text, and RawHtml can all implement—making them all interchangeable as Renderables within the AST structure. This also leaves room for each of these to individually support features that others do not: for example, it make sense for an Element to contain other Renderables, but the same cannot be said for Text. This interface replaces the need to overload the concept of a name in the element, while still acheiving the same result (text, elements, and rawHtml can sit interchangeably within the AST).

There's some work going on to add this (among other things) which you can take a look at here. I'm working on a fairly large refactor at the moment (moving block and inline rendering into distinct types), so haven't pushed commits for a little bit—there is still work going on though.

My hope is that in 2.0 we can sufficiently decouple the interface from the specific implementation throughout the code and declare it public, while still maintaining enough breathing room to have things like say, the way an element might be rendered by a block, to change without breaking extensions (as is the case at the moment).

[aidantwoods]

It's not quite finished yet, but I've put in a bit of work related to getting Parsedown ready for a new "extension" model towards 2.0 over in #685. The general idea will be that "extensions" can implement their own Blocks and Inlines, as well as their own Configurables to allow the setter-like behaviour we have currently.
The core Parsedown class now really just strings everything together, only really depending on the basic Paragraph and Plaintext components directly (we could remove the direct dependence if there is a good case for it though)—everything else can be swapped out by the user, so they could go so far as to pick and choose different Blocks and Inlines from various different sources if they so wish.

Perhaps another change worth mentioning is that parsing of particular components is now completely decoupled from the HTML generation. This means that there is no need (and you should not try!) to traverse the HTML structure produced by a particular component if you want to defer to it. Instead each component can present it's own API which makes sense uniquely to that component, and other code that wants to defer parsing can use that API—but not need to depend on the exact structure HTML that will be produced (which is really very subject to change!).
For an example of this, inline Images defer some of their parsing to the Link component
https://github.com/aidantwoods/parsedown/blob/8a48dcfe315bc7e37028626646291538dce24d10/src/Components/Inlines/Image.php#L45-L51
and then store that produced Link for later use, where the basic attributes are retrieved https://github.com/aidantwoods/parsedown/blob/8a48dcfe315bc7e37028626646291538dce24d10/src/Components/Inlines/Image.php#L62
and used to generate the image (without directly depending on the HTML that the Link ends up producing).
This should hopefully allow a lot of reusability without binding so close to the internal behaviour that things like bugfixes can break dependents (as is the case at the moment).

[PhrozenByte]

@aidantwoods: Amazing work, as always! ❤️ Please give a heads up as soon as you assume the new code base's design to be completed (i.e. you don't expect major design changes). Since it's a whole new code base, I'd love to do a QA if you wish. Time is very limited unfortunately, so I'd rather do this not until you think it's ready in a matter of the code's design.

[aidantwoods]

Since it's a whole new code base, I'd love to do a QA if you wish

@PhrozenByte I'll definitely take you up on that :)

As far as what there is left to do: it should be mainly around making sure I get the API right, and taking care of a few bits and pieces I still need to carry over from 1.0. Also documentation—lot's of that to write.
It's still a bit away from finished, but hopefully nothing too major from here on out (but let's wait till I actually finish before committing to that ;-) )

[aidantwoods]

Closing since this'll be super-seeded superseded by 2.0.0. If anyone would like to help, please send PRs to the 2.0.x branch or leave a review over at #708.

[dregad]

You mean superseded?😇

[aidantwoods]

You mean superseded?😇

Yes, and I have no idea why I decided to spell it like that aha