util: add MIME utilities

[bmeck]

Checklist

• make -j4 test (UNIX), or vcbuild test (Windows) passes
• tests and/or benchmarks are included
• documentation is changed or added
• commit message follows commit guidelines

This adds a new module for handling MIME types. It is standalone based upon the WHATWG MIME standards , but is more intended to open up APIs that wish to use MIME for coordinating content types. I have left out the various sniffing for content bodies until a future date as I believe that is a longer discussion than just parsing and serializing. The API is based upon URL and URLSearchParams and intentionally allows us to expand MIMEParams for dealing with multiple parameters with the same name (.getAll / .append etc.).

This PR does not introduce any MIME databases for file paths.

The MIMEParams are not exposed as a string getter / setter, unlike url.search. These manipulations can be done using the map interface provided, and accessors can be added in a follow on if seen as desirable.

Test cases were pulled from web platform tests , but we don't have a well documented process for keeping such fixtures in sync so I have not done anything except vendor them for now, similar to https://github.com/nodejs/node/blob/master/test/fixtures/url-tests.js .

[addaleax]

Did you reach out to the owners of http://npm.im/mime?

[bmeck]

@addaleax nope, I don't plan to merge this immediately and think all the names on npm cause problems. Kind of leaning on nodejs/TSC#389 , but am also open to changing the name, all simple ones seem to conflict though.

[devsnek]

this looks pretty sweet :D

you'll need to modify the type parser and add MIMEParams

[jdalton]

Possibly related work (and contributors) https://github.com/jsdom/whatwg-mimetype.

[Trott]

I hate to do this (and if other people think it's the wrong thing to do, go ahead and undo it), but I'm going to add the blocked label on this because, as alluded above, we're going to want to get resolution on nodejs/TSC#389 before landing this, if at all possible.

[jdalton]

In addition, with this functionality covered by user-land and something that isn't a standardized constructor (like WHATWG URL) I think there needs to be more thought on whether this belongs in core.

[jasnell]

@bradleymeck ... Can you please go into some of the motivation behind this? It's been something that you and I have spoken about but others would benefit from the context.

I think that a new top level module for this likely is not the best approach. Dropping this off util would work, I think... And would allow it to side step the module namespacing issue.

[targos]

What are the reasons for implementing this in C++ instead of JS?

[bmeck]

@targos C++ is easier to put robust code into and this has potential usage for things like http server header parsing where things need a bit more scrutiny I believe as there can be vulnerable to DOS.

@jdalton Lack of an standard API is exactly why this is being proposed. If there was a standard API, we could follow it. I've had brief talks in the past about standardizing a MIME JS API, but do not see WHATWG as a place that I've had success standardizing things. Even if this module is overtaken by a WHATWG API, I will still see this PR as a success. If you will note, the package you linked is not currently on standards track and there are also a variety of other modules with similar purpose. I'd note that the package you have chosen as an example has a design extremely similar to this API, and I have done some digging to see what other packages provide as well. If you don't think this API is valid without a standard, would Node creating a separate standards document be sufficient?

@jasnell This PR has many motivations:

• HTTP header parsing
• Enabling implementing APIs that use MIMEs, particularly: Blob and potentially Response
  • The usage of these APIs is increasingly important to me as Service Workers are seeing increased usage and writing code between them and Node has a lot of friction.
• Allow exploring a more robust path for ESM than a custom one off format type
  • One less format is probably a good thing and matches web loading
• Allow adding the MIME sniffing algorithms once we can decide on and API to expose for those
  • This would allow simpler integrations for uploads seeking to obtain MIMEs for the various formats
• Allow exploring MIME database integrations for file paths. Even though MIME Sniffing does not use file extensions, browsers do however and are called out in handling of files for input.
  • this would allow a consistent experience for both configuration and API usage. It also alleviates a major package download https://npm-stat.com/charts.html?package=mime-db&from=2018-05-05&to=2018-06-05

This PR just introduces the base structure for those and is not trying to overburden itself by trying to satisfy all those use case workflows at once. This allows those workflows to be explored based upon an API rooted in the concepts of URL and written in a web compatible way using getters etc. I am also following the same convention of adding it as a module. I do not think hanging it off util is appropriate as the use cases grow.

[apapirovski]

C++ is easier to put robust code into and this has potential usage for things like http server header parsing where things need a bit more scrutiny I believe as there can be vulnerable to DOS.

This might not be the PR for this conversation but I would really like to have a discussion at some point re: choices being made for C++ vs JS implementations. There are upsides to C++ without a doubt but it's incredibly hard to get contributors that cover all 3 of domain expertise, JS and C++. IMO that is something that sort of gets ignored when we add new C++ code.

If our goal is to have the project be sustainable in the long-term then the right balance needs to be found. Otherwise we're making it more and more difficult for a large portion of the ecosystem to contribute anything back.

[bmeck]

@apapirovski I'd agree. This PR mostly requires a non-flexible parsing algorithm be exposed. I think keeping as much as possible in JS is fine, but still requires more effort than C++ in order to achieve robust code that doesn't suffer from things like prototype pollution or reflection to get to internals. If you have specific comments about this PR I will do my best to address them, but would not like to discuss that topic as a whole in this PR.

[bmeck]

@annevk on IRC has noted that in https://tools.ietf.org/html/rfc2045#section-5 the parameters are called a set, so we should not expose a .getAll in the future. This PR currently does not expose duplicates to users and does not need to be changed.

[jdalton]

@bmeck

I do not think hanging it off util is appropriate as the use cases grow.

I'm with @jasnell that it might be easier with it off something like util or was even just internal for now. Since this does exist in user-land and Node's small core preference I would tend to favor punting in core until it's explicitly needed and then maybe landing it as an internal helper first.

[bmeck]

@jdalton in order to expose any of the use cases I listed above we can't have it be entirely internal.

Per hanging off util it does not seem that hanging classes of util has a well defined meaning because this is a general purpose data structure, not just some utility functionality. We could claim that everything could be hung off util, but I don't think adding discrete classes around standards makes sense. We wouldn't add things like http to util either.

Per userland solutions, as you pointed to there are some, but the most popular one is not the one you pointed to and this is a means to define a singular point for solutions using MIMEs to rely on and we can gather input as we move forward such that the reasons for all the differing implementations can be alleviated. The fact that there are so many downloads and so many implementations makes this area ripe for standardizing on a solution so that people can reliably use the same APIs. An API spaced with millions of downloads a day seems like something we should look at providing out of the box, especially since it relates to other parts of core such as http headers.

In addition, per my comment above, there are many things that could be implemented using this. Punting it until we have a specific use case doesn't make sense to me as we would design the API specifically for a use case rather than implement a generic low level solution that can be used by all use cases. I do not want to conflate this with other use cases as it is valuable as a lower level primitive on its own. We should design this around being a reusable primitive rather than trying to carve out a primitive from a specific use case only.

[jdalton]

Per userland solutions, as you pointed to there are some, but the most popular one is not the one you pointed to and this is a means to define a singular point for solutions using MIMEs to rely on

There's always choices in user-land and that hasn't stopped Node from deferring to user-land. Popular solutions tend to shake out. We're now aware of the API (or similar) so if the need comes as part of some Node feature down the line then we'll be able to revisit and decide on internal or external, builtin module or bolted on some place else. At the moment though it feels like we're getting into these discussions a bit early.

[bmeck]

@jdalton I only brought up those discussions in response to yours. I'm not sure why you think this use case of providing the standardized parser for MIMEs is invalid on its own. I can only address the reasons why this is a valid API to alleviate concerns given the current framing. I don't think there is wiggle room on the parsing aspect of this, and would appreciate explanation of why we should not provide the parser primitives to JS.

[jdalton]

When prioritizing scenarios this may be needed for I'm OK deferring those that can be handled with user-land packages to user-land (so lower priority) and moving those that would require core integration higher. I'm fine with an internal helper to enable whatever scenario it's needed for at the time it's needed. I'm not okay adding a new built-in for a bunch of maybe-might-be scenarios esp. when you've shown resistance to bolting it on anywhere else. If you're not open to internal or bolting it on somewhere else I feel like the PR is going to get nixed.

[bmeck]

@jdalton I cannot have it be internal for any of those uses, which I don't see being claimed as invalid. I also am unclear on why bolting it onto util makes it somehow better, can you explain why putting it on util makes it more valuable? Also, can you explain why allowing users to create solutions using this and using user land to explore expanding this primitive to other parts of Core APIs makes it less appealing? I do not understand the resistance to the functionality being provided in your comments, and do not understand if you are against any new primitives being added to Node given that your comments seem to apply to all possible features in general that can be built upon rather than providing a specific use case that is deemed worthy of a PR. I generally approach APIs by building upon primitives rather than trying to carve primitives out of monolithic solutions.

[annevk]

FWIW, as I noted to @bmeck on IRC, there is interest in adding a standardized API that also works for Node.js. I've previously pinged @jasnell about this: whatwg/mimesniff#43.