Auto merge of #21248 - brson:feature-staging, r=alexcrichton

This implements the remaining bits of 'feature staging', as described in [RFC 507](https://github.com/rust-lang/rfcs/blob/master/text/0507-release-channels.md).

This is not quite done, but the substance of the work is complete so submitting for early review.

Key changes:
* `unstable`, `stable` and `deprecated` attributes all require 'feature' and 'since', and support an optional 'reason'.
* The `unstable` lint is removed.
* A new 'stability checking' pass warns when a used unstable library feature has not been activated with the `feature` attribute. At 1.0 beta this will become an error.
* A new 'unused feature checking' pass emits a lint ('unused_feature', renamed from 'unknown_feature') for any features that were activated but not used.
* A new tidy script `featureck.py` performs some global sanity checking, particularly that 'since' numbers agree, and also prints out a summary of features.

Differences from RFC:
* As implemented `unstable` requires a `since` attribute. I do not know if this is useful. I included it in the original sed script and just left it.
* RFC didn't specify the name of the optional 'reason' attribute.
* This continues to use 'unstable', 'stable' and 'deprecated' names (the 'nice' names) instead of 'staged_unstable', but only activates them with the crate-level 'staged_api' attribute.

I intend to update the RFC based on the outcome of this PR.

Issues:
* The unused feature check doesn't account for language features - i.e. you can activate a language feature, not use it, and not get the error.

Open questions:
* All unstable and deprecated features are named 'unnamed_feature', which i picked just because it is uniquely greppable. This is the 'catch-all' feature. What should it be?
* All stable features are named 'grandfathered'. What should this be?

TODO:
* Add check that all `deprecated` attributes are paired with a `stable` attribute in order to preserve the knowledge about when a feature became stable.
* Update rustdoc in various ways.
* Remove obsolete stability discussion from reference.
* Add features for 'path', 'io', 'os', 'hash' and 'rand'.

cc #20445 @alexcrichton @aturon

mk/tests.mk

@@ -301,6 +301,7 @@ tidy:
 		| grep '^$(S)src/rust-installer' -v \
 		| xargs $(CFG_PYTHON) $(S)src/etc/check-binaries.py
 		$(Q) $(CFG_PYTHON) $(S)src/etc/errorck.py $(S)src/
+		$(Q) $(CFG_PYTHON) $(S)src/etc/featureck.py $(S)src/
 
 
 endif

src/compiletest/compiletest.rs

@@ -13,7 +13,15 @@
 #![feature(slicing_syntax, unboxed_closures)]
 #![feature(box_syntax)]
 #![feature(int_uint)]
-#![allow(unstable)]
+#![feature(test)]
+#![feature(rustc_private)]
+#![feature(std_misc)]
+#![feature(path)]
+#![feature(io)]
+#![feature(core)]
+#![feature(collections)]
+#![feature(os)]
+#![feature(unicode)]
 
 #![deny(warnings)]
 

src/doc/reference.md

@@ -2359,77 +2359,6 @@ Supported traits for `derive` are:
 * `Show`, to format a value using the `{}` formatter.
 * `Zero`, to create a zero instance of a numeric data type.
 
-### Stability
-
-One can indicate the stability of an API using the following attributes:
-
-* `deprecated`: This item should no longer be used, e.g. it has been
-  replaced. No guarantee of backwards-compatibility.
-* `experimental`: This item was only recently introduced or is
-  otherwise in a state of flux. It may change significantly, or even
-  be removed. No guarantee of backwards-compatibility.
-* `unstable`: This item is still under development, but requires more
-  testing to be considered stable. No guarantee of backwards-compatibility.
-* `stable`: This item is considered stable, and will not change
-  significantly. Guarantee of backwards-compatibility.
-* `frozen`: This item is very stable, and is unlikely to
-  change. Guarantee of backwards-compatibility.
-* `locked`: This item will never change unless a serious bug is
-  found. Guarantee of backwards-compatibility.
-
-These levels are directly inspired by
-[Node.js' "stability index"](http://nodejs.org/api/documentation.html).
-
-Stability levels are inherited, so an item's stability attribute is the default
-stability for everything nested underneath it.
-
-There are lints for disallowing items marked with certain levels: `deprecated`,
-`experimental` and `unstable`. For now, only `deprecated` warns by default, but
-this will change once the standard library has been stabilized. Stability
-levels are meant to be promises at the crate level, so these lints only apply
-when referencing items from an _external_ crate, not to items defined within
-the current crate. Items with no stability level are considered to be unstable
-for the purposes of the lint. One can give an optional string that will be
-displayed when the lint flags the use of an item.
-
-For example, if we define one crate called `stability_levels`:
-
-```{.ignore}
-#[deprecated="replaced by `best`"]
-pub fn bad() {
-    // delete everything
-}
-
-pub fn better() {
-    // delete fewer things
-}
-
-#[stable]
-pub fn best() {
-    // delete nothing
-}
-```
-
-then the lints will work as follows for a client crate:
-
-```{.ignore}
-#![warn(unstable)]
-extern crate stability_levels;
-use stability_levels::{bad, better, best};
-
-fn main() {
-    bad(); // "warning: use of deprecated item: replaced by `best`"
-
-    better(); // "warning: use of unmarked item"
-
-    best(); // no warning
-}
-```
-
-> **Note:** Currently these are only checked when applied to individual
-> functions, structs, methods and enum variants, *not* to entire modules,
-> traits, impls or enums themselves.
-
 ### Compiler Features
 
 Certain aspects of Rust may be implemented in the compiler, but they're not

src/driver/driver.rs

@@ -8,7 +8,9 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.
 
-#![allow(unstable)]
+#![allow(unknown_features)]
+#![cfg_attr(rustc, feature(rustc_private))]
+#![cfg_attr(rustdoc, feature(rustdoc))]
 
 #[cfg(rustdoc)]
 extern crate "rustdoc" as this;

src/etc/featureck.py

@@ -0,0 +1,243 @@
+# Copyright 2015 The Rust Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution and at
+# http://rust-lang.org/COPYRIGHT.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+# This script does a tree-wide sanity checks against stability
+# attributes, currently:
+#   * For all feature_name/level pairs the 'since' field is the same
+#   * That no features are both stable and unstable.
+#   * That lib features don't have the same name as lang features
+#     unless they are on the 'joint_features' whitelist
+#   * That features that exist in both lang and lib and are stable
+#     since the same version
+#   * Prints information about features
+
+import sys, os, re
+
+src_dir = sys.argv[1]
+
+# Features that are allowed to exist in both the language and the library
+joint_features = [ ]
+
+# Grab the list of language features from the compiler
+language_gate_statuses = [ "Active", "Deprecated", "Removed", "Accepted" ]
+feature_gate_source = os.path.join(src_dir, "libsyntax", "feature_gate.rs")
+language_features = []
+language_feature_names = []
+with open(feature_gate_source, 'r') as f:
+    for line in f:
+        original_line = line
+        line = line.strip()
+        is_feature_line = False
+        for status in language_gate_statuses:
+            if status in line and line.startswith("("):
+                is_feature_line = True
+
+        if is_feature_line:
+            line = line.replace("(", "").replace("),", "").replace(")", "")
+            parts = line.split(",")
+            if len(parts) != 3:
+                print "error: unexpected number of components in line: " + original_line
+                sys.exit(1)
+            feature_name = parts[0].strip().replace('"', "")
+            since = parts[1].strip().replace('"', "")
+            status = parts[2].strip()
+            assert len(feature_name) > 0
+            assert len(since) > 0
+            assert len(status) > 0
+
+            language_feature_names += [feature_name]
+            language_features += [(feature_name, since, status)]
+
+assert len(language_features) > 0
+
+errors = False
+
+lib_features = { }
+lib_features_and_level = { }
+for (dirpath, dirnames, filenames) in os.walk(src_dir):
+    # Don't look for feature names in tests
+    if "src/test" in dirpath:
+        continue
+
+    # Takes a long time to traverse LLVM
+    if "src/llvm" in dirpath:
+        continue
+
+    for filename in filenames:
+        if not filename.endswith(".rs"):
+            continue
+
+        path = os.path.join(dirpath, filename)
+        with open(path, 'r') as f:
+            line_num = 0
+            for line in f:
+                line_num += 1
+                level = None
+                if "[unstable(" in line:
+                    level = "unstable"
+                elif "[stable(" in line:
+                    level = "stable"
+                else:
+                    continue
+
+                # This is a stability attribute. For the purposes of this
+                # script we expect both the 'feature' and 'since' attributes on
+                # the same line, e.g.
+                # `#[unstable(feature = "foo", since = "1.0.0")]`
+
+                p = re.compile('(unstable|stable).*feature *= *"(\w*)"')
+                m = p.search(line)
+                if not m is None:
+                    feature_name = m.group(2)
+                    since = None
+                    if re.compile("\[ *stable").search(line) is not None:
+                        pp = re.compile('since *= *"([\w\.]*)"')
+                        mm = pp.search(line)
+                        if not mm is None:
+                            since = mm.group(1)
+                        else:
+                            print "error: misformed stability attribute"
+                            print "line " + str(line_num) + " of " + path + ":"
+                            print line
+                            errors = True
+
+                    lib_features[feature_name] = feature_name
+                    if lib_features_and_level.get((feature_name, level)) is None:
+                        # Add it to the observed features
+                        lib_features_and_level[(feature_name, level)] = \
+                            (since, path, line_num, line)
+                    else:
+                        # Verify that for this combination of feature_name and level the 'since'
+                        # attribute matches.
+                        (expected_since, source_path, source_line_num, source_line) = \
+                            lib_features_and_level.get((feature_name, level))
+                        if since != expected_since:
+                            print "error: mismatch in " + level + " feature '" + feature_name + "'"
+                            print "line " + str(source_line_num) + " of " + source_path + ":"
+                            print source_line
+                            print "line " + str(line_num) + " of " + path + ":"
+                            print line
+                            errors = True
+
+                    # Verify that this lib feature doesn't duplicate a lang feature
+                    if feature_name in language_feature_names:
+                        print "error: lib feature '" + feature_name + "' duplicates a lang feature"
+                        print "line " + str(line_num) + " of " + path + ":"
+                        print line
+                        errors = True
+
+                else:
+                    print "error: misformed stability attribute"
+                    print "line " + str(line_num) + " of " + path + ":"
+                    print line
+                    errors = True
+
+# Merge data about both lists
+# name, lang, lib, status, stable since
+
+language_feature_stats = {}
+
+for f in language_features:
+    name = f[0]
+    lang = True
+    lib = False
+    status = "unstable"
+    stable_since = None
+
+    if f[2] == "Accepted":
+        status = "stable"
+    if status == "stable":
+        stable_since = f[1]
+
+    language_feature_stats[name] = (name, lang, lib, status, stable_since)
+
+lib_feature_stats = {}
+
+for f in lib_features:
+    name = f
+    lang = False
+    lib = True
+    status = "unstable"
+    stable_since = None
+
+    is_stable = lib_features_and_level.get((name, "stable")) is not None
+    is_unstable = lib_features_and_level.get((name, "unstable")) is not None
+
+    if is_stable and is_unstable:
+        print "error: feature '" + name + "' is both stable and unstable"
+        errors = True
+
+    if is_stable:
+        status = "stable"
+        stable_since = lib_features_and_level[(name, "stable")][0]
+    elif is_unstable:
+        status = "unstable"
+
+    lib_feature_stats[name] = (name, lang, lib, status, stable_since)
+
+# Check for overlap in two sets
+merged_stats = { }
+
+for name in lib_feature_stats:
+    if language_feature_stats.get(name) is not None:
+        if not name in joint_features:
+            print "error: feature '" + name + "' is both a lang and lib feature but not whitelisted"
+            errors = True
+        lang_status = lang_feature_stats[name][3]
+        lib_status = lib_feature_stats[name][3]
+        lang_stable_since = lang_feature_stats[name][4]
+        lib_stable_since = lib_feature_stats[name][4]
+
+        if lang_status != lib_status and lib_status != "deprecated":
+            print "error: feature '" + name + "' has lang status " + lang_status + \
+                  " but lib status " + lib_status
+            errors = True
+
+        if lang_stable_since != lib_stable_since:
+            print "error: feature '" + name + "' has lang stable since " + lang_stable_since + \
+                  " but lib stable since " + lib_stable_since
+            errors = True
+
+        merged_stats[name] = (name, True, True, lang_status, lang_stable_since)
+
+        del language_feature_stats[name]
+        del lib_feature_stats[name]
+
+if errors:
+    sys.exit(1)
+
+# Finally, display the stats
+stats = {}
+stats.update(language_feature_stats)
+stats.update(lib_feature_stats)
+stats.update(merged_stats)
+lines = []
+for s in stats:
+    s = stats[s]
+    type_ = "lang"
+    if s[1] and s[2]:
+        type_ = "lang/lib"
+    elif s[2]:
+        type_ = "lib"
+    line = "{: <32}".format(s[0]) + \
+           "{: <8}".format(type_) + \
+           "{: <12}".format(s[3]) + \
+           "{: <8}".format(str(s[4]))
+    lines += [line]
+
+lines.sort()
+
+print
+print "Rust feature summary:"
+print
+for line in lines:
+    print line
+print
+