Pain-points with existing http module

[jasnell]

@nodejs/http ... While the existing HTTP support in node works, it's not perfect and it's not easy to build on from a user land perspective. Let's use this issue to begin documenting the pain points. To help keep things organized, let's try to differentiate user pain from module-developer pain...

[ritch]

As a user of node's http server (and client)... I want node (or something) to guarantee 100% support of the http spec, so I can be sure my http server (and client) is compatible with other systems.

[ritch]

As a node module-developer... I want specific documentation around what I should and should not expect from the http parser, so that I can design and implement my module in a way that works well with node as it exists now.

Example: I want to be able to completely stop parsing an http request after a specific header has been parsed. AFAICT node v4 does not include a header event, and at some point in the past it did IIRC. This sort of information should be made clear somewhere for module developers.

[jasnell]

From a Node Core perspective, there is uncertainty on what exactly Node is expected to provide as far as HTTP is concerned. @dougwilson made the comment yesterday that the http module in node currently is really a framework in it's own right. If you look at Java by comparison... the Java core library provides many of the low level pieces, and you can build a working http server using those but you really need to look to the ecosystem for the actual functionality.

[mikeal]

So, we know that we can't remove, or even effectively change, the API that is there today because it would break too much of the ecosystem. With that in mind it seems like there are two things people could do:

1. Improve the existing implementation, either by allowing for an alternative implementation of the same API outside of core or simply by patching the current one.
2. Provide a way for users of core http to provide their own improved versions that are compatible but extended in some way.

Off the top of my head, this is what would need to happen for this to be possible:

• Separate the abstract base classes for incoming/outgoing so that they aren't shared between client and server.
• For each object that is instantiated use a createMethod function that is publicly accessible and could be overrided. This would allow people to build their own: Parser, ServerRequest, ServerResponse, ClientRequest and ClientResponse objects. Of course, they'd need to implement the existing API for them to work but this would allow for alternative implementations.

Thoughts?

[jasnell]

@mikeal brings up the good point that, currently, the HTTP client and server pieces overlap each other a bit too much. Sharing code between the two is great, but they can and should be teased apart quite a bit more.

[mikeal]

having them shared is an interesting point of code re-use but it makes them impossible to reason about, and probably slows them down tbh.

[indutny]

@jasnell I think they should overlap even more :) Why should they be separate? Will it help anyone?

[ritch]

For each object that is instantiated use a createMethod function that is publicly accessible and could be overrided. This would allow people to build their own: Parser, ServerRequest, ServerResponse, ClientRequest and ClientResponse objects. Of course, they'd need to implement the existing API for them to work but this would allow for alternative implementations.

So, make the http module hookable essentially. Overall LGTM...I think the important part is:

• agreeing on the whats hookable, (IMO above seems like a reasonable place to start)
• documenting the hooks well, so the stuff outside of core can be built in a way that core expects
• ensure some reasonable stability of the hooking interface(s), so the userland modules can focus on the improvments instead of keeping up with core changes to the hook api

I think they should overlap even more :) Why should they be separate? Will it help anyone?

As long as you can hook the server stuff and leave client stuff alone (and vice versa)... then it should be fine to share code. At least from the hookability perspective. Making it easier to reason about the code aside of course.

[jasnell]

I created a new issue we can use to deep dive on the hookability point -> #6