HTTP Resource Representation Variants #164

snuggs · 2018-01-25T04:27:10Z

This was already implemented but was sporadically problematic with indeterministic bugs. In other words...duct taped together with no tests. Tests now thanks to tap(e).

Installation

Ability to "install" snuggsi packaged library from https://snuggsi.es with the following:

<script src=//snuggsi.es></script>

Or from mounting server middleware

server.js

app // Express or Koa
  .use ( snuggsi )

index.html

<!-- served from `/snuggsi` endpoint (minified and compressed) -->
<!-- (Javascript / Ecmascript negotiated on browser capabilities) -->
<!-- (gzip / brotli negotiated on browser capabilities) -->
<script src=/snuggsi></script>

<!-- served from `/snuggsi` endpoint (minified and compressed) -->
<!-- (Javascript / Ecmascript negotiated on browser capabilities) -->
<script src=/snuggsi?debug></script>

<!-- Ecmascript version (minified and compressed) -->
<!-- (gzip / brotli negotiated on browser capabilities) -->
<script src=/snuggsi.es></script>

<!-- Ecmascript version (uncompressed) -->
<script src=/snuggsi.es?debug></script>

<!-- JavaScript version (minified and compressed) -->
<!-- (gzip / brotli negotiated on browser capabilities) -->
<script src=/snuggsi.js></script>

<!-- JavaScript version (uncompressed) -->
<script src=/snuggsi.js?debug></script>

References:

HTTP Representation Variants - https://mnot.github.io/I-D/variants
WHATWG URL Spec - https://url.spec.whatwg.org
Wikipedia - https://en.wikipedia.org/wiki/Content_negotiation
MDN Accept-Encoding - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
MDN - https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation
HTTP Accept Header - https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1

Resource Representations

Resource Default HTML `GET / Accept: text/html`

- GET /?debug Accept: text/html - (index.html) with no compression.
- GET / Accept: text/html Accept-Encoding: identity,(none) - (index.html) with no compression.

Resource Default CSS `GET / Accept: text/css`

- GET /?debug Accept: text/css - (`index.css') with no compression.
- GET / Accept: text/css Accept-Encoding: identity,(none) - (index.css) no compression.

Resource Default Javascript `GET / Accept: /`

Will determine whether or not to serve modern Ecmascript or legacy JavaScript

- GET /?debug Accept: */* - application/javascript (.js) no compression.
- GET /?debug Accept: */* Accept-Encoding: br - application/ecmascript (.es) no compression.
- GET / Accept: */* Accept-Encoding: identity,(none) - application/javascript (.js) no compression.
- GET / Accept: */* Accept-Encoding: br - application/ecmascript (.min.es.br) with brotli compression.
- GET / Accept: */* Accept-Encoding: gzip,deflate - application/javascript (.min.js.gz) gzip compression.

`GET /snuggsi`

Will determine whether or not to serve modern Ecmascript or legacy JavaScript

- GET /snuggsi?debug - application/ecmascript (.js) with no compression.
- GET /snuggsi?debug Accept-Encoding: br,identity - application/ecmascript (.es) with no compression.
- GET /snuggsi?debug Accept-Encoding: gzip,compress,deflate,identity,*,(none) - application/javascript (.js) with no compression.
- GET /snuggsi - application/ecmascript _(.js`)_ with no compression.
- GET /snuggsi Accept-Encoding: identity - application/javascript (.js) with no compression.
- GET /snuggsi Accept-Encoding: * - application/javascript (.min.js.gz) with gzip compression.
- GET /snuggsi Accept-Encoding: gzip - application/javascript (.min.js.gz) with gzip compression.
- GET /snuggsi Accept-Encoding: br - application/ecmascript (.min.es.br) with brotli compression.

`GET /snuggsi.es`

- GET /snuggsi.es - application/ecmascript with no compression.
- GET /snuggsi.es?debug - application/ecmascript with no compression.
- GET /snuggsi.es Accept-Encoding: identity - application/javascript with no compression.
- GET /snuggsi.es Accept-Encoding: * - application/ecmascript with default compression.
- GET /snuggsi.es Accept-Encoding: gzip - application/ecmascript with gzip compression.
- GET /snuggsi.es Accept-Encoding: br - application/ecmascript with brotli compression.

`GET /snuggsi.js`

- GET /snuggsi.js - application/javascript with no compression.
- GET /snuggsi.js?debug - application/javascript with no compression.
- GET /snuggsi.js Accept-Encoding: identity - application/javascript with no compression.
- GET /snuggsi.js Accept-Encoding: * - application/javascript with default compression.
- GET /snuggsi.js Accept-Encoding: gzip application/javascript with gzip compression.
- GET /snuggsi.js Accept-Encoding: br - application/javascript with brotli compression.

References

ES6 Modules & Client/Server support

Anyone touching an app in the browser should at least get familiar.
/cc @mrbernnz @btakita @halorium @janz93 @pachonk @robcole @scottmacdowell @kurtcagle @johnrlive

http://jsmodules.io
https://blog.whatwg.org/js-modules
http://2ality.com/2017/05/es-module-specifiers.html
(@tmornini you thought THIS PR was Epic. lol) Add <script type="module"> and module resolution/fetching/evaluation whatwg/html#443

tmornini · 2018-02-20T19:36:44Z

@snuggs WOW! Huge piece of work!

snuggs · 2018-02-22T05:16:32Z

@tmornini indeed! And VERY well tested :-)

mrbernnz · 2018-03-08T21:48:03Z

@snuggs did this PR really need 250+ commits?

tmornini · 2018-03-08T22:44:38Z

@mrbernnz Strange question to ask when staring at a 250+ commit PR. 😄

Of course, @snuggs may well decide to do some rebasing before going to master.

brandondees · 2018-03-08T23:19:57Z

what is this, github golf? ⛳️

tmornini · 2018-03-09T04:57:25Z

@brandondees If so, we're going to have to start calling @snuggs "Tiger" 😄

snuggs · 2018-03-09T13:53:41Z

There's also an added 150-200 tests as well @mrbernnz 😄. there are over 100+ different permutations for matching routes to endpoints with HTTP verbs. Let alone content negotiation for the types. Imagine that truth table. Truth be told (pun intended) surprised it's not 1,000 commits knowing me. I just want this thing DONE. I'm Also working with MDN, WHATWG, and W3C on doing SOUND HTTP Documentation for the web. Let's just say it doesn't move as fast and this PR is a cumulation of much to do with #71. As of this year we are in a better place to implement HTTP Resource Variants (ne. Content Negotiation) which is EXTREMELY IMPORTANT related to preloading. (Preload. HUH?! What is it good for?)

Just be readin' up on the description links about HTTP variants and resources....That is unless you want to slang commits too! ;-)

You will see that (finally) we are moving in a direction towards HTTP that @tmornini and I been talking about for damn near a decade. HTTP > MVC. @brandondees @janz93 @pachonk I came across and added some contribution to a nice little Ruby library Flame is based on many of the same principles of this PR.

The intent of this PR is because i'm sick of having a "God directory" for static assets and endpoints when that's not how HTTP works. Also the "front end" paradigm is now a grey area as Links, Resources, preloading, prefetching IS REAL AND READY TODAY. Specs are just relying on people to implement...(since 1999) /cc @btakita @codeSnorter @kurtcagle

foo/index.es

class { // this is a resource. You "route" to it.
  // You get GET, HEAD, OPTIONS(I think?) for free because the HTTP spec says so

  options (context) // this should be handled by CORS. Resource knows supported methods ;-)
    { context.status = 204; context.body = 'Or you could monkeypatch but not advisable' }

  // Usually non-safe methods go here
  put (context) {}

  post (context) {}

  delete (context) {}

  // all other 50+ HTTP verb methods not defined issue a 405 Not Allowed on the resource.
  // **** It never was 404 but framework authors got lazy. ****
  // any GLOBAL HTTP method returns a 501 Not Implemented
  // and can be configured from server. (As the spec states)
  // @tmornini GET / PURGE is where caching is done YES?! ;-) 
  // The spec allows custom verbs/codes like PURGE & 420. Can easily be defined here.
  // e.g. GET /foo/bar.html doesn't need implementation. Just send bar.html from /foo dir.
  // https://stackoverflow.com/questions/35632607/is-the-http-method-purge-idempotent-in-varnish
}

server code somewhere

mount `foo` // or
route `/some/other/endpoint/{id}` (Resource `foo`)
// etc. or some other api. But that's all I wanna do. Rise repeat for next route
// so far haven't seen more terse/lazy code to perform what we want to do.
// An ENDPOINT is nothing more than route + method on resource.

Amazes me we took this long to realize there are no Controllers....only Resources...The http spec says so.

To be clear this is for internal use only and allows me to be dry with the many projects that all require this boilerplate BS. (including yours @mrbernnz). I refuse to use YAAF (Yet Another (wrong) Abstraction Framework).

P.S. There's a test for every single one of those permutations as well 😎:

We got route tokenization (e.g./foo/{id}/bar/{baz}) of routes for free from the code that does tokenization of custom elements. @tmornini @mrbernnz I never knew how cool / painful string manipulation was lol.

snuggs · 2018-03-07T00:56:23Z

index.html

@@ -4,13 +4,19 @@

 <meta
  name=viewport
-  content='width=device-width,initial-scale=1'
+  content='user-scalable=no,initial-scale=1.0,minimum-scale=1.0,maximum-scale=1.0,width=device-width,height=device-height'


thanks @mrbernnz

snuggs · 2018-03-11T00:27:19Z

@tmornini @mrbernnz @brandondees interesting...
Feeling even better about this PR.

How many frameworks you think would go 💥 💣 if you went back in retrospect and implemented the HTTP spec correctly....

gin-gonic/gin#20
julienschmidt/httprouter#51
julienschmidt/httprouter#30
julienschmidt/httprouter#52

brandondees · 2018-03-11T04:31:32Z

@snuggs hard to say i could expect any of them to work with it fully without taking it all back to the drawing board. Looks like the example cited is the closest thing I know of, and still doesn't have all the kinks worked out yet.

Imagine if you could run a standard suite of tests that could assert spec compliance for a given implementation...

snuggs · 2018-03-14T19:32:08Z

Imagine if you could run a standard suite of tests that could assert spec compliance for a given implementation... @brandondees

Interesting you say this. About a year ago @tmornini and I had a discussion about versions/testing "User land" tools based off the living specs (As opposed to canonical apps that end-use-developers create that don't have failing tests appear out of nowhere). I mentioned to him how even I don't know if the suite passes and that's great we have CI. The specs change frequently and implementation can break just from an update to WPT (Web Platform Tests. The test suite that ALL browsers MUST adhere to.) And we use JSDOM which basically creates an in memory "browser" that is merely a stubbed implementation of said WPT/WHATWG/W3C specs.

Browser vendors are constantly breaking tests (that were previously passing). An update to specs (therefore update to WPT) triggers failing tests and tell the browsers what they MUST implement immediately. They then implement bugs with respective vendors. None of the spec's concern either....an implementation detail. Personal problem of the browser vendors if you will. We are also under the same scrutiny which I think is a great thing.

That being said, to make a long story longer your thoughts may not be too out of reach in 2018. We're definitely on that path already (intentionally). #YouHeardItHereFirst

snuggs · 2018-03-15T03:54:16Z

@brandondees @tmornini PERFECT example of the spec being changed (which now breaks our implementation theoretically). This would be a nightmare for us 1 year from now. Luckily we didn't expose CustomElement.upgrade (element) but internally we will need to implement this algorithm. @brandondees feel like making a issue here? Dunno what to call it as it's not really a bug but will be in the future.

Issue Request from W3C: WICG/webcomponents#710
PR Response from WHATWG: whatwg/html#3535
LOC in question (@tmornini when you're TOO good at #CATAT Call A Thing A Thing): https://github.com/devpunks/snuggsi/blame/master/custom-element-registry/index.es#L32
Tests to pass: web-platform-tests/wpt#9869
Browser Bugs they are now forced to implement ASAP: whatwg/html#3535 (comment)

We got time before the browsers get there LOLZ but doesn't mean we can't implement today.

P.S. @brandondees still steering CLEAR AWAY from Shadow DOMmy stuff. YAGNI (yet). We support <slot> tho. Please respond with an issue not comment on unrelated PR that's becomming long in the tooth if not merged this week hehe.

/cc @tmornini

snuggs · 2018-03-29T23:51:23Z

In defense of `.js`

@brandondees we'll need to update Koa, (at minimum), Rails, Sinatra, Express, for the new .mjs and existing .es extensions. Many servers respond with application/octetstream for .mjs and .es. This is problematic and causes a Content-Disposition due to the spec stating to NEVER run that media type (for obvious security reasons). Luckily we can just update the core libs for respective platforms and cure most of the (sane) frameworks in the meantime. Anything that prevents the downloading of these files.

This prevents people from linking to unpkg. As a band aid we create a .es.js file 💩
For example >>> http://unpkg.com/snuggsi/dist/snuggsi.min.es vs http://unpkg.com/snuggsi/dist/snuggsi.min.es.js

Same holds true for .mjs files. Therefore a TON of servers are "broken" with this feature/bug (including ourselves).

I'm on it this weekend. This list and Resource.assets will be just about it for this PR. https://snuggsi.es can be active then and unpkg.com will work properly for people migrating to JS modules which will be very shortly here. Lastly, web/WHATWG/W3C/IETF/HTML/Fetch/DOM specs are fairly in order as media types have been an ancient 💩 show for about a decade now.

@tmornini may need your guidance on the protocol to update golang source code. text/x-javascript is on the MUST NOT USE list as far as the platform is concerned. What should we do to notify them? https://golang.org/src/mime/type.go

References

ES6 Modules & Client/Server support

Anyone touching an app in the browser should at least get familiar.
/cc @mrbernnz @btakita @halorium @janz93 @pachonk @robcole @scottmacdowell @kurtcagle @johnrlive

http://jsmodules.io
https://blog.whatwg.org/js-modules
http://2ality.com/2017/05/es-module-specifiers.html
(@tmornini you thought THIS PR was Epic. lol) Add <script type="module"> and module resolution/fetching/evaluation whatwg/html#443

tmornini · 2018-04-02T17:57:22Z

middleware/negotiator.es

@@ -36,7 +36,7 @@ module.exports = async (context, next) =>

  await next ()

-  console.log (extension, resource)
+  //console.log (extension, resource)


Not a fan of commented-out code.

@tmornini dammit you're right. This file actually is gonna go bye bye. Literally does nothing now (But has some good practices that need importing.) Thanks for the update. I just now saw this. /cc @brandondees

tmornini · 2018-04-02T17:57:51Z

middleware/route.test

@@ -109,7 +109,7 @@ test ('decoding route URI encoded {token} parameters', async t => {
 })


-test.only ('', async t => {
+test.only ('Testin bitch', async t => {


HTTP 406 - not acceptable

tmornini · 2018-04-02T17:59:08Z

middleware/route.test

@@ -125,7 +125,7 @@ test.only ('', async t => {
  , server =
      (new Server (stack)).serve ``

-  , response = await fetch (`${URI}${encoded}`)
+  , response = await fetch (`${URI}${value}`)


Why code more-generic? The original name is so much more clear...

Which name? @tmornini? I've made the changes you requested. Pruning the bonzai bush. Only Resource and route concepts are i'd say 90% of the work. 10% is crap from shitty implementation. Wanted to get it working "the right way" then take out the trash. (Starting to get stinky in the old parts of the code now).

@tmornini @brandondees wrapping this one up just making sure I covered everything asked for. encoded in retrospect is better off. "token/value" pairs are the most abstract. The automobile to the skateboard.

#FunFact @tmornini they don't call it HTTP method in the spec they call it method *token* ;-)

@domenic

I have discussed this in the past with @domenic. He seems to think all requests/responses should default to `text/javascript` Unfortunately...ALL(?) browsers seem to request JS with `application/javascript` And in response servers respond with `application/javascript` In order to abide by his convention we need to update a myriad of libraries in Ruby AND Javascript for these tests to pass. Pros: Luckily most browsers and servers are "broken" therefore our "false positives" with `application/javascript` work. ¯\_(ツ)_/¯ Cons: If only one extension related per mime there will be blood! Although `text/javascript` may be more correct... the web (as we know it) still uses `application/javascript` convention. See References: - #164 Next steps: Determine if issue needs to be created in WHATWG HTML / DOM / Fetch ¯\_(ツ)_/¯ This will be the only way we can get user land mime libraries updated without an IETF (pending) blessing for `text/javascript`. /cc @tmornini @brandondees

snuggs · 2018-06-06T18:35:14Z

@brandondees @tmornini irony. At the WHATWG/W3C we've come to consensus that the defacto mime for JS is text/javascript However, in Ruby, Node, and I believe Go the main mime libraries do not reflect for text/javascript (But do wrongly assume application/javascript as the old spec from 2003 that we are updating has always lead devs down the wrong mimetype path (pun intended).

So the last remaining tests in suite are failing until this bug/feature gets added.

snuggs · 2018-06-08T07:45:57Z

@tmornini as per your recommendation to skip for time being. Have created issue #174 to track progress.

Addressed in 00dc115

I think that's a wrap!!

Please accept.

tmornini · 2018-06-10T16:08:53Z

@tmornini may need your guidance on the protocol to update golang source code. text/x-javascript is on the MUST NOT USE list as far as the platform is concerned. What should we do to notify them? https://golang.org/src/mime/type.go

https://github.com/golang/go/issues

@domenic

I have discussed this in the past with @domenic. He seems to think all requests/responses should default to `text/javascript` Unfortunately...ALL(?) browsers seem to request JS with `application/javascript` And in response servers respond with `application/javascript` In order to abide by his convention we need to update a myriad of libraries in Ruby AND Javascript for these tests to pass. Pros: Luckily most browsers and servers are "broken" therefore our "false positives" with `application/javascript` work. ¯\_(ツ)_/¯ Cons: If only one extension related per mime there will be blood! Although `text/javascript` may be more correct... the web (as we know it) still uses `application/javascript` convention. See References: - #164 Next steps: Determine if issue needs to be created in WHATWG HTML / DOM / Fetch ¯\_(ツ)_/¯ This will be the only way we can get user land mime libraries updated without an IETF (pending) blessing for `text/javascript`. /cc @tmornini @brandondees

snuggs added the 🚧 WIP 🚧 label Jan 25, 2018

snuggs self-assigned this Jan 25, 2018

snuggs force-pushed the content-negotiation branch from f43303c to c9c427a Compare January 25, 2018 05:02

snuggs changed the title ~~Content negotiation~~ Content Negotiation Jan 25, 2018

snuggs force-pushed the content-negotiation branch from 3073a44 to abe5e5b Compare January 25, 2018 20:03

snuggs changed the title ~~Content Negotiation~~ HTTP Representation Variants / Content Negotiation Jan 26, 2018

snuggs force-pushed the content-negotiation branch from a2c196c to 86e9912 Compare January 29, 2018 19:30

snuggs force-pushed the content-negotiation branch from 611a285 to c62404e Compare February 17, 2018 02:07

snuggs commented Mar 9, 2018

View reviewed changes

snuggs changed the title ~~HTTP Representation Variants / Content Negotiation~~ HTTP Representation Variants Mar 15, 2018

snuggs changed the title ~~HTTP Representation Variants~~ HTTP Resource Representation Variants Mar 15, 2018

This was referenced Mar 15, 2018

Tape does not pass require.extensions to resolve tape-testing/tape#395

Closed

Sinatra doesn't seem to like a trailing / fastlane/ci#15

Closed

tmornini requested changes Apr 2, 2018

View reviewed changes

snuggs mentioned this pull request Apr 7, 2018

Add TRACE to Safe Methods #168

Closed

snuggs added 2 commits June 6, 2018 14:23

Migrate fixture path to resources/fixtures/

12d364e

Update path to add current working directory

1ee7d52

snuggs added 15 commits June 6, 2018 14:25

Add examples route and resource to serve

45f0871

Add caching ranges comment

12b4ef8

Add link to readFile being far slower

2c3e923

Add link to readFile being far slower

303f21c

Add link to koa-stream

dedc259

Add link to AWS requests using streams

6ec9ecb

Change resolve to then

ae72aad

Migrate join buffer

64d784f

Refactor method signature for module export

b9a482c

Refactor buffer out of read module and into join function

a20c8b1

Remove bug from join source

63676ba

Refactor join algorithm for stream buffer reading

87c3518

Add whitespace to join

a635878

Add only test

f4a4cd7

Remove package lock

5586336

snuggs force-pushed the content-negotiation branch from edf9197 to 5586336 Compare June 6, 2018 18:31

snuggs added 5 commits June 7, 2018 09:58

Use backticks for context.accepts

40a797f

Update whitespace

496585d

Add comment for preload

45169e9

Add comment for preload

f942161

Skip tests as per @tmornini recommendation /cc @brandondees

00dc115

snuggs removed the 🚧 WIP 🚧 label Jun 9, 2018

tmornini approved these changes Jun 10, 2018

View reviewed changes

snuggs merged commit 74f15c5 into master Jun 11, 2018

snuggs deleted the content-negotiation branch June 11, 2018 16:36

snuggs mentioned this pull request Jul 11, 2018

Add test suite coverage to deployment #175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP Resource Representation Variants #164

HTTP Resource Representation Variants #164

snuggs commented Jan 25, 2018 •

edited

Loading

tmornini commented Feb 20, 2018

snuggs commented Feb 22, 2018

mrbernnz commented Mar 8, 2018

tmornini commented Mar 8, 2018 •

edited

Loading

brandondees commented Mar 8, 2018

tmornini commented Mar 9, 2018

snuggs commented Mar 9, 2018 •

edited

Loading

snuggs Mar 7, 2018

snuggs commented Mar 11, 2018

brandondees commented Mar 11, 2018

snuggs commented Mar 14, 2018 •

edited

Loading

snuggs commented Mar 15, 2018 •

edited

Loading

snuggs commented Mar 29, 2018 •

edited

Loading

tmornini Apr 2, 2018

snuggs Apr 13, 2018

tmornini Apr 2, 2018

tmornini Apr 2, 2018

snuggs Apr 13, 2018

snuggs May 9, 2018

snuggs commented Jun 6, 2018

snuggs commented Jun 8, 2018 •

edited

Loading

tmornini commented Jun 10, 2018

HTTP Resource Representation Variants #164

HTTP Resource Representation Variants #164

Conversation

snuggs commented Jan 25, 2018 • edited Loading

Installation

server.js

index.html

References:

Resource Representations

Resource Default HTML GET / Accept: text/html

Resource Default CSS GET / Accept: text/css

Resource Default Javascript GET / Accept: */*

GET /snuggsi

GET /snuggsi.es

GET /snuggsi.js

References

ES6 Modules & Client/Server support

tmornini commented Feb 20, 2018

snuggs commented Feb 22, 2018

mrbernnz commented Mar 8, 2018

tmornini commented Mar 8, 2018 • edited Loading

brandondees commented Mar 8, 2018

tmornini commented Mar 9, 2018

snuggs commented Mar 9, 2018 • edited Loading

foo/index.es

server code somewhere

Choose a reason for hiding this comment

snuggs commented Mar 11, 2018

brandondees commented Mar 11, 2018

snuggs commented Mar 14, 2018 • edited Loading

snuggs commented Mar 15, 2018 • edited Loading

snuggs commented Mar 29, 2018 • edited Loading

Related

References

ES6 Modules & Client/Server support

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snuggs commented Jun 6, 2018

snuggs commented Jun 8, 2018 • edited Loading

tmornini commented Jun 10, 2018

snuggs commented Jan 25, 2018 •

edited

Loading

Resource Default HTML `GET / Accept: text/html`

Resource Default CSS `GET / Accept: text/css`

Resource Default Javascript `GET / Accept: /`

`GET /snuggsi`

`GET /snuggsi.es`

`GET /snuggsi.js`

tmornini commented Mar 8, 2018 •

edited

Loading

snuggs commented Mar 9, 2018 •

edited

Loading

snuggs commented Mar 14, 2018 •

edited

Loading

snuggs commented Mar 15, 2018 •

edited

Loading

snuggs commented Mar 29, 2018 •

edited

Loading

snuggs commented Jun 8, 2018 •

edited

Loading