test flight before publish

[jdalton]

I think it would be awesome to be able to test-flight an update before publishing a package. For example before I npm publish an update to lodash say 2.4.3 it would offer to do a test flight (also invokable independently of publishing) which would pull down a number of its most used dependents and, using their package rules, attempt to run their unit tests ,npm test, with the pre-published version of lodash. It would then report back the pass/fails of the dependents. This would help give devs a heads up to potential breaking changes, even on undocumented/pseudo private functionality, before they publish.

Thoughts?

[othiym23]

This is a cool idea, but needs significant refinement before it'll be ready to implement. It's almost a fully-fledged service instead of a feature. I don't have time to dig into this right now, but if you (and others) want to start figuring out how this would work, I can provide feedback later.

[jdalton]

Rock. I know there's at least one project that has attempted this. I can't think of it off the top of my head but I've seen it mentioned a time or two in npm issues. I tried tweeting to see if I can get them pulled in to the discussion.

[anvaka]

I would love this feature too!

Also could be interesting to let developers subscribe to events stream from npm, and run their own scripts based on events.

E.g. If package A depends on B I would love to get notified when minor version of B is published, so that I can rebuild A.

[terinjokes]

@anvaka npm exposes a CouchDB replication endpoint that could be used with follow, or friends to build something like this.

How are you thinking this would work? I assume the most popular is a daemon you can run that triggers some CI action? A tool a developer runs in a package directory is probably better suited to tooling around something like npm outdated or David.

[dylang]

To see what packages have your module in their devDependencies might require npm/npm-registry-couchapp#122 to be merged in.

[sindresorhus]

👍 This would be so helpful. I don't care if it's in npm or an independent service (happy to pay). I currently do this manually for my most popular packages, but it's time consuming and boring.

[kevva]

👍 would love this.

[bahmutov]

I wrote a simple CLI tool dont-break if anyone is interested. Installs given list of projects from .dont-break text file, copies your current code into dependency and runs npm test.

[sindresorhus]

@bahmutov Nice, but I don't want to maintain a list of dependents manually. Maybe get the 10 most downloaded dependents or something.

[bahmutov]

@sindresorhus I added --top-downloads <n> and --top-starred <n> to fetch and overwrite .dont-break file.

[othiym23]

This is still an interesting idea, and is something that I could see npm, or perhaps another team, like Greenkeeper, building. I completely agree that it would be useful, in much the same way that citgm has been useful for the Node.js developer team.

However! This is something that’s much larger than the CLI, and as it’s too big for the CLI team to scope, I’m going to close this feature request. I would love to have something like this for my own projects, so I encourage interested parties to keep banging on it (and this seems like a valuable service for npm to provide at some point in the future, so who knows?). Thanks to all for their ideas and time!