"integer" type removed?

[handrews]

As noted in PR #145 the "integer" type was removed in v5.

In issue #79 from last month, @awwright was still talking about "integer" as a type. Neither @epoberezkin nor I recall any discussion of removing it. @awwright , @Relequestual what was the rationale here?

Requiring the use of {"type": "number", "multipleOf": 1} for a very common use case seems burdensome.

[awwright]

This was an oversight, see #145 or #147.

The change was supposed to resolve that type:"integer" definitively includes both 1.0 and 1.1e1 as "integers" for validation purposes. But it seems the "validation" spec got left out of this clarification.

[Relequestual]

Yeah, removing it seems a bit much.

-sidebar
I think a lot of people are feeling that rushing draft 5 was more dentrimental than any benifit (which I'm not sure anyone was really clear on either)
/-

The issue at hand. I think we need to consider what has actually happened here.
What was said about integer in draft 4?
What was removed and why?
What were the implications?

As far as I can tell, at they very least, it made draft 4 documents not forward compatible with draft 5 documents (which I think was originally an aim of moving forward with drafts?)

[awwright]

At the end of the day, it's a draft. Its entire purpose is for collecting feedback just like this.

I don't see any reason this should break reverse compatibility. Originally, these were all defined in the "type" keyword anyways. Later, it became "core" and the validation keywords were moved out, but the seven type definitions were left in defined as "primitive" types. This caused some confusion as to if values like 11.0 or 1.1e1 should be "integer" or not, for the purpose of "type".

[handrews]

@awwright: @Relequestual and I have discussed your response and have quite a few concerns. Most of the words below are mine, but I think I am capturing the main points that concern both of us (and I’m sure Ben can comment if I misrepresented something).

At the end of the day, it's a draft. Its entire purpose is for collecting feedback just like this.

Just because it’s a draft as far as the IETF is concerned, doesn’t mean it can’t or won’t be used in production. As such, we need to recognized the impact of drafts and behave appropriately.

It makes a difference that the draft was rushed out with many bugs and surprises.  At least four of us have expressed concern about how draft 5 was rushed out ( @Relequestual and I here, and two others in #50 , for instance), and #96 shows yet another person having been caught off guard by the lack of time to review the hyper-schema changes. That’s nearly all of the most active participants in the JSON Schema project.

Your response is always "it doesn't bother me", but this is not your personal project and your actions have impact on others.  You cite your intention to make v5 just a cleanup of v4, but you've repeatedly brushed off the fact that you are the only person who thinks that's what v5 actually is.
 
It makes a difference whether problematic changes are pushed out in a draft (which is some level of official even with the I-D restrictions) vs just being discussed here.  You hold our proposals to high standards while dismissing our concerns about your own work.
 
I'm bothered by your unwillingness to acknowledge or even consider the possibility that v5 was a misstep.  It suggests that similar problems will occur with v6.  You refuse to discuss any sort of process, so none of us have any idea when or why you'll decide to publish v6.  Why should we feel comfortable with this?

It’s fine in principal to argue that IDs shouldn’t be used in production, but that doesn’t align with reality, and that needs to be recognised in the way everyone who is involved in this project conducts themselves.

You are fond of citing "rough consensus and running code". There was no opportunity given to any of us to arrive at any sort of consensus, rough or otherwise, on publishing draft 5, and that has continued to be problematic.

[awwright]

Thanks for the feedback. Perhaps the latest 2016-10-13 I-D should have been broken down into an even smaller series of updates -- at the time, we thought limiting the scope to bugfixes made the scope sufficiently small.

In any event, you know I've been very clear that the draft-05 I-D should not introduce anything that causes JSON Schemas or implementations to break. I've reviewed a lot of implementations to try and ensure this.

I also circulated several versions of rendered documents, prior to submitting the new I-D, and made several changes from the feedback.

But of course, some issues like this one slipped through. "integer" has not been removed, it has been a part of "type" since the very beginning. There are, indeed, seven valid values for "type". Not saying that "integer" is one of them was a simple oversight on my part. Do you think #147 fixes this?

[handrews]

In any event, you know I've been very clear that the draft-05 I-D should not introduce anything that causes JSON Schemas or implementations to break. I've reviewed a lot of implementations to try and ensure this.

You've stated that intent many times, but you have not convinced anyone (certainly not me, and as far as I can tell I am not alone) that you succeeded in that intent. You can make some tortured reasonings about how things are somehow "compatible" which, quite frankly, I've never found at all convincing. You've left a horribly confusing meta-schema situation, and you did that without any discussion or review that i noticed.

You did float somethings for review, but all of the sudden there was a giant surprise rework of hyper-schema, and then you submitted it within the next day or two without waiting for people to even finish reading the changes. I was shocked and it was and is clear that I was not the only one.

Never mind that when I submit a small fix using language identical to other parts of the spec implementing a keyword that has gotten broad support, it has to sit for two weeks even when everyone who has had anything to do with it has approved it. I think that's good, but I think the "hey I'm about to publish this" phase needs at least a two week freeze as well.

I'm going to be blunt. While I am appreciative of the work that you did to get a draft out there at all, and I continue to respect your deep technincal knowledge which we really need, I think that the draft that you published, and the confusion that continues to surround it, and the extremely unclear accounting of how to work with it on the web site or what happened with the meta-schemas, has done as much harm as good. And I do not have faith that the next draft will go any better, as you only respond to concerns in the most vague and general terms that boil down to "you'll do stuff and approve stuff and publish stuff when and why you think it needs to be done, without explaining yourself to us." Or you just repeat the same defenses of your actions that you state every time.

You are not effectively leading this project, and when Ben and I try to improve things such as with a contributors guide, you shut it down, once again saying that you've got your plan and it will be fine.

You need to drop your defenses of the past and address the concerns that @Relequestual , I, and others have in the present if you want to actually lead this project. Or if you are not trying to lead, you need to let us come up with a proposal for how to run this as a community project within the loose constraints established by the IETF process.

[awwright]

@handrews Yeah, there's a lot of wording changes that were really contrived. Unfortunately people were interpreting the previous draft in mutually incompatible ways, and eventually we have to settle on a side. We can continue to negotiate the details, but I think the I-D is a perfectly valid place to put forward proposals -- that's its entire purpose. To the extent people are treating it otherwise, well, hopefully the process will become more self-evident as we release more I-Ds more frequently in the future.

As far as making changes, I've tried my best to examine implementations, making sure that I was documenting existing behavior where possible.

But where breakage does exist, I rely on the feedback from people commenting on the I-D. It does not appear that #96, for example, would have ever been filed if we waited on publication. It's a good thing it was filed, that issue is still open, and we're still working on resolving it (pending the author getting back from vacation)! Worst case, we have to roll the definition back and introduce a new keyword.

Also remember, implementations continue to retain a wide discretion to determine how to approach interoperability and reverse compatibility. Core even has this to say on the matter:

JSON Schema implementations SHOULD implement support for
current and previous published drafts of JSON Schema vocabularies as
deemed reasonable.

First and foremost, I want to make sure this repository is a place for implementors and schema authors to decide on common functionality and behavior. That's not always an easy thing to do! These exact sorts of issues come up for other collaborative efforts, and it's something that gets worked through.

Now, I do want to make sure that the "integer" issue has been settled to your liking. Does #147 answer your outstanding conserns with this issue and #145?

If you have any other conserns, with process or other issues, I'm sure there's a different issue that would allow freer conversation; you also know you can mail or join IRC.

You'll also have to pardon me since I've been away from my desktop a lot for the past two weeks (I've been less than responsive to a lot of emails as a result), there's not a lot I can really do about that ;)

[handrews]

@awwright Yes #147 and #145 are fine to resolve this issue.

We can move the process questions over to #137 where you have also waved off all concerns.

I will tell you that if I had managed to get through the hyper-schema changes in the few days that you allotted, I would have filed #96. But it's clear your never going to admit that your approach was anything but ideal so I give up.

[awwright]

@handrews I hear what you're saying, but I'm trying to impress that there's no simple solution. Please don't take this as waving off your conserns; but I am saying we're in the ongoing process of working towards producing a document that everyone is happy with. If you're ever upset with anything, you've got a 50-50 shot I've got my IRC client open, and especially if you stick around a bit.

Now, does #147 address your issue's conserns, and resolve that "integer" was never supposed to be removed?

[handrews]

Now, does #147 address your issue's conserns, and resolve that "integer" was never supposed to be removed?

As I said in the previous comment, yes. Sorry I forgot to hit "close and comment" instead of "comment"

[epoberezkin]

@awwright:

@handrews I hear what you're saying, but I'm trying to impress that there's no simple solution.

There is simple solution - to agree on a process. I was equally shocked when the draft was published before I finished submitting my comments, two days after notification, as if to avoid addressing them. That is really disappointing.

[handrews]

@epoberezkin check out the commit log for a bunch more changes pushed directly to master this weekend without review :-(

[epoberezkin]

@handrews yeah... The guideline in our company btw is "no commits to master, ever" :)

@awwright For community driven project a "benevolent dictator" approach won't work. I think your resistance to agree on the process and put your changes through the same review process as everybody else's will lead us into some dead end.

[Relequestual]

I think after some chats today, @awwright agreed that, if the community wants no commits to master, and pull request everything, then he will comply... right @awwright ? =D

[epoberezkin]

If that's the case, the next step would be to stabilise the currently published draft with the original mission of "improving/clarifying the language without changing functionality". The current draft could have been a "minor version change" following semantic versioning approach, i.e. 4.1. Not sure if it is possible with IETF I-Ds. And then the changes we are discussing could still be v5 and the fixes to the current I-D would be 4.2. Is it possible?

[handrews]

The current draft could have been a "minor version change" following semantic versioning approach, i.e. 4.1. Not sure if it is possible with IETF I-Ds. And then the changes we are discussing could still be v5 and the fixes to the current I-D would be 4.2. Is it possible?

Nope. Whole numbers, and they could not be updated.

With the v6 road map thread I started on the mailing list, I was trying to converge onto a v6 that we could get out (with meta-schemas this time) soon by setting a clear scope. I still need to send out messages for validation and hyper-schema.

[epoberezkin]

If that's the case, I believe draft6 should deliver on the promise of "improving/clarifying the language without changing functionality" and that would provide a stable point

[handrews]

@epoberezkin we've already gotten a lot of work in to add things in v6. I don't want to back that out. We also have bug fixes in there. Let's just wrap up a nice chunk of work as v6 and release it. With updated meta-schemas.

Honestly, I don't see the point of having v4, v5, and v6 all theoretically having the same functionality. Bugs aside, v5 clarified plenty of things, and we should just fix the bugs and move on.

In any event, this should either go in the thread I started on the mailing list, in the contributor's guide issue, or get its own issue. Let's stop discussing road maps in a closed bug.