Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce Custom Test Frameworks #53410

Merged
merged 4 commits into from
Sep 5, 2018

Conversation

djrenren
Copy link
Contributor

@djrenren djrenren commented Aug 15, 2018

Introduces #[test_case] and #[test_runner] and re-implements #[test] and #[bench] in terms of them.

Details found here: https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html

@rust-highfive
Copy link
Collaborator

r? @alexcrichton

(rust_highfive has picked a reviewer for you, use r? to override)

@rust-highfive
Copy link
Collaborator

⚠️ Warning ⚠️

  • These commits modify submodules.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 15, 2018
@djrenren djrenren force-pushed the custom-test-frameworks branch 2 times, most recently from 2c23431 to 1eae7e9 Compare August 15, 2018 23:08
@rust-highfive

This comment has been minimized.

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
[00:48:19] ....................................................................................................
[00:48:22] .................................................................................................i..
[00:48:25] ....................................................................................................
[00:48:28] ....................................................................................................
[00:48:30] ..............................................iiiiiiiii.............................................
[00:48:36] ....................................................................................................
[00:48:40] ....................................................................................................
[00:48:42] ..........................i.........................................................................
[00:48:45] ............................i...............................................i.i..ii.................
---
travis_time:start:test_incremental
Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[00:55:49] 
[00:55:49] running 90 tests
[00:56:03] .........................................................F................................
[00:56:03] 
[00:56:03] ---- [incremental] incremental/issue-49595/issue_49595.rs stdout ----
[00:56:03] 
[00:56:03] 
[00:56:03] error in revision `cfail2`: test compilation failed although it shouldn't!
[00:56:03] status: exit code: 101
[00:56:03] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/incremental/issue-49595/issue_49595.rs" "--target=x86_64-unknown-linux-gnu" "--cfg" "cfail2" "-C" "incremental=/checkout/obj/build/x86_64-unknown-linux-gnu/test/incremental/issue-49595/issue_49595/issue_4-89d4-8b8521d6bb01 kernel: [    0.000000] ACPI: WAET 0x00000000BFFF5CE0 000028 (v01 Google GOOGWAET 00000001 GOOG 00000001)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] ACPI: SRAT 0x00000000BFFF4C30 0000E8 (v01 Google GOOGSRAT 00000001 GOOG 00000001)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] ACPI: Local APIC address 0xfee00000
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: PXM 0 -> APIC 0x00 -> Node 0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: PXM 0 -> APIC 0x01 -> Node 0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: PXM 0 -> APIC 0x02 -> Node 0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: PXM 0 -> APIC 0x03 -> Node 0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: Node 0 PXM 0 [mem 0x00000000-0x0009ffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: Node 0 PXM 0 [mem 0x00100000-0xbfffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] SRAT: Node 0 PXM 0 [mem 0x100000000-0x3ffffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] NUMA: Node 0 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0xbfffffff] -> [mem 0x00000000-0xbfffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x10000000 ranges
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   node   0: [mem 0x0000000000100000-0x00000000bfff2fff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   node   0: [mem 0x0000000100000000-0x00000003ffffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x00000003ffffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] On node 0 totalpages: 3932049
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   DMA zone: 64 pages used for memmap
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   DMA zone: 21 pages reserved
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   DMA zone: 3998 pages, LIFO batch:0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   DMA32 zone: 12224 pages used for memmap
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   DMA32 zone: 782323 pages, LIFO batch:31
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   Normal zone: 49152 pages used for memmap
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000]   Normal zone: 3145728 pages, LIFO batch:31
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4ug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:4 nr_node_ids:1
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] PERCPU: Embedded 34 pages/cpu @ffff8803ffc00000 s98392 r8192 d32680 u524288
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] pcpu-alloc: s98392 r8192 d32680 u524288 alloc=1*2097152
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] pcpu-alloc: [0] 0 1 2 3 
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 3870588
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Policy zone: Normal
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-101-generic root=UUID=752b4ef1-0512-4cae-b541-f03ffd29be1b ro cgroup_enable=memory swapaccount=1 apparmor=0 console=ttyS0 console=ttyS0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Calgary: detecting Calgary via BIOS EBDA area
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.000000] Memory: 153kernel: [    0.640599] ftrace: allocating 32185 entries in 126 pages
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.700148] smpboot: APIC(0) Converting physical 0 to logical package 0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.702319] smpboot: Max logical packages: 2
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.704219] x2apic enabled
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.706306] Switched APIC routing to physical x2apic.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.711936] ..TIMER: vector=0x30 apic1=0 pin1=0 apic2=-1 pin2=-1
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.820872] smpboot: CPU0: Intel(R) Xeon(R) CPU @ 2.30GHz (family: 0x6, model: 0x3f, stepping: 0x0)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.826305] Performance Events: unsupported p6 CPU model 63 no PMU driver, software events only.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.832102] x86: Booting SMP configuration:
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.833754] .... node  #0, CPUs:      #1
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.835668] kvm-clock: cpu 1, msr 3:ffff1041, secondary cpu clock
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.842394]  #2
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    0.843194] kvm-clock: cpu 2, msr 3:ffff1081,travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.264930] usbcore: registered new interface driver usbfs
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.267286] usbcore: registered new interface driver hub
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.270450] usbcore: registered new device driver usb
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.273634] ioremap error for 0xbfffd000-0xc0000000, requested 0x2, got 0x0
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.276083] dmi: Firmware registration failed.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.278115] PCI: Using ACPI for IRQ routing
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.279954] PCI: pci_cache_line_size set to 64 bytes
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.280064] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.280067] e820: reserve RAM buffer [mem 0xbfff3000-0xbfffffff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.280216] NetLabel: Initializing
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.282187] NetLabel:  domain hash size = 128
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.284227] NetLabel:  protocols = UNLABELED CIPSOv4
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.285766] NetLabel:  unlab16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.336775] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.339867] PCI: CLS 0 bytes, default 64
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    1.339955] Unpacking initramfs...
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.560234] Freeing initrd memory: 21432K
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.562221] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.565536] software IO TLB [mem 0xbbff3000-0xbfff3000] (64MB) mapped at [ffff8800bbff3000-ffff8800bfff2fff]
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.569827] RAPL PMU detected, API unit is 2^-32 Joules, 3 fixed counters 10737418240 ms ovfl timer
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.572936] hw unit of domain pp0-core 2^-0 Joules
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.574467] hw unit of domain package 2^-0 Joules
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.575957] hw unit of domain dram 2^-16 Joules
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.577789] Scanning for low memory corruption every 60 seconds
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.581146] audit: initializing netlink subsys (disabled)
Aug 16 00:16:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.798447] tun: Universal TUN/TAP device driver, 1.6
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.800089] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.870872] PPP generic driver version 2.4.2
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.873821] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.878152] ehci-pci: EHCI PCI platform driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.879932] ehci-platform: EHCI generic platform driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.882127] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.884704] ohci-pci: OHCI PCI platform driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.886392] ohci-platform: OHCI generic platform driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.889036] uhci_hcd: USB Universal Host Controller Interface driver
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.892520] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    3.896710] i8042: Warning: Keylock active
Aug 16 00:16:03 travis-job-d2cf025c-62be-00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.810049] raid6: avx2x4   gen() 20572 MB/s
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.812446] raid6: using algorithm avx2x4 gen() 20572 MB/s
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.815412] raid6: using avx2x2 recovery algorithm
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.819969] xor: automatically using best checksumming function:
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.862074]    avx       : 21594.000 MB/sec
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.879175] Btrfs loaded
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.941397] EXT4-fs (sda1): INFO: recovery required on readonly filesystem
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    8.944369] EXT4-fs (sda1): write access will be enabled during recovery
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    9.020774] EXT4-fs (sda1): orphan cleanup on readonly fs
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    9.030492] EXT4-fs (sda1): 6 orphan inodes deleted
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    9.032372] EXT4-fs (sda1): recovery complete
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [    9.041385] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel:andom read (6 bytes read, 50 bits of entropy available)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   11.329049] random: cloud-init: uninitialized urandom read (32 bytes read, 50 bits of entropy available)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   11.535307] random: cloud-init: uninitialized urandom read (32 bytes read, 50 bits of entropy available)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   11.884909] random: mktemp: uninitialized urandom read (12 bytes read, 53 bits of entropy available)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   11.989594] random: mktemp: uninitialized urandom read (6 bytes read, 54 bits of entropy available)
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   12.096291] EXT4-fs (sda1): resizing filesystem from 3931904 to 7864064 blocks
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   12.158055] EXT4-fs (sda1): resized filesystem to 7864064
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   12.663881] init: failsafe main process (1096) killed by TERM signal
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 instance-setup: INFO Running set_multiqueue.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 instance-setup: INFO Set channels for eth0 to 4.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 instance-setup: INFO Setting /proc/irq/25/smp_affinity_list to 0 for device virtio1.
Aug 16 00:16:03 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 instance-d6bb01 google-accounts: INFO Created user account me.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-accounts: INFO Creating a new user account for aj.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-accounts: INFO Created user account aj.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-accounts: INFO Creating a new user account for carmen.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-clock-skew: INFO Synced system time with hardware clock.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-accounts: INFO Created user account carmen.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 google-accounts: INFO Removing user packer.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   13.938166] floppy0: no floppy controllers found
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 pollinate: system was previously seeded at [2017-12-05 19:31:29.715998981 +0000]
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 pollinate: To re-seed this system again, use the -r|--reseed option
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   14.176047] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   14.179225] Bridge firewalling registered
Aug 16 00:16:04 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   14.190467] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
0:16:12 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 ec2: -----END SSH HOST KEY FINGERPRINTS-----
Aug 16 00:16:12 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 ec2: #############################################################
Aug 16 00:16:19 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 ntpdate[2146]: the NTP socket is in use, exiting
Aug 16 00:16:54 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [   64.484917] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Aug 16 00:17:01 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 CRON[4128]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Aug 16 00:18:36 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.567938] device veth270ce62 entered promiscuous mode
Aug 16 00:18:36 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.665319] cgroup: docker-runc (4848) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
Aug 16 00:18:36 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.665322] cgroup: "memory" requires setting use_hierarchy to 1 on the root
Aug 16 00:18:37 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.732517] eth0: renamed from veth3b757f6
Aug 16 00:18:37 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.769765] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Aug 16 00:18:37 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.771137] docker0: port 1(veth270ce62) entered forwarding state
Aug 16 00:18:37 travis-job-d2cf025c-62be-47f5-89d4-8b8521d6bb01 kernel: [  166.771160] doc\(.*\)|\1|;y|!|/|'); if [ -f "$EXE" ]; then printf travis_fold":start:crashlog\n\033[31;1m%s\033[0m\n" "$CORE"; gdb -q -c "$CORE" "$EXE" -iex 'set auto-load off' -iex 'dir src/' -iex 'set sysroot .' -ex bt -ex q; echo travis_fold":"end:crashlog; fi; done || true
travis_fold:end:after_failure.5
travis_fold:start:after_failure.6
travis_time:start:10a2bca8
travis_time:start:10a2bca8
$ cat ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers || true
cat: ./obj/build/x86_64-unknown-linux-gnu/native/asan/build/lib/asan/clang_rt.asan-dynamic-i386.vers: No such file or directory
travis_fold:end:after_failure.6
travis_fold:start:after_failure.7
travis_time:start:1ea01aa8
$ dmesg | grep -i kill

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@alexcrichton
Copy link
Member

Aha! I think this may be a bug in the testing attributes there, this line may need to be renamed from __test to test perhaps?

@rust-highfive
Copy link
Collaborator

The job x86_64-gnu-llvm-5.0 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/6b/fa/89c248eaacccd816fdea88206060a7cd221f227855782ff7b0ffb80d725a/awscli-1.15.81-py2.py3-none-any.whl (1.3MB)
    0% |▎                               | 10kB 13.6MB/s eta 0:00:01
    1% |▌                               | 20kB 1.9MB/s eta 0:00:01
    2% |▊                               | 30kB 2.2MB/s eta 0:00:01
    3% |█                               | 40kB 2.0MB/s eta 0:00:01
---
    100% |████████████████████████████████| 61kB 8.0MB/s 
Collecting botocore==1.10.80 (from awscli)
/usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading https://files.pythonhosted.org/packages/5e/cf/b97f44993766af17bf64aeddadf66f63b6ebf3d700565cc7ee7b13cd0067/botocore-1.10.80-py2.py3-none-any.whl (4.5MB)
    0% |                                | 10kB 36.6MB/s eta 0:00:01
    0% |▏                               | 20kB 29.6MB/s eta 0:00:01
    0% |▏                               | 30kB 35.3MB/s eta 0:00:01
    0% |▎                               | 40kB 23.4MB/s eta 0:00:01
---
[00:51:41] ....................................................................................................
[00:51:45] ....................................................................................................
[00:51:47] .i..................................................................................................
[00:51:51] ....................................................................................................
[00:51:53] ..................................................iiiiiiiii.........................................
[00:51:59] ....................................................................................................
[00:52:03] ....................................................................................................
[00:52:06] ...............................i....................................................................
[00:52:08] .................................i...............................................i.i..ii............
---
travis_time:start:test_run-make-fulldeps
Check compiletest suite=run-make-fulldeps mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
[01:21:42] 
[01:21:42] running 189 tests
[01:22:13] .......................................................................................F............
[01:23:07] ........................................................................................test [run-make] run-make-fulldeps/long-linker-command-lines has been running for over 60 seconds
[01:23:47] failures:
[01:23:47] 
[01:23:47] ---- [run-make] run-make-fulldeps/libtest-json stdout ----
[01:23:47] 
[01:23:47] 
[01:23:47] error: make failed
[01:23:47] status: exit code: 2
[01:23:47] command: "make"
[01:23:47] stdout:
[01:23:47] ------------------------------------------
[01:23:47] make[1]: Entering directory '/checkout/src/test/run-make-fulldeps/libtest-json'
[01:23:47] LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json  --test f.rs
[01:23:47] RUST_BACKTRACE=0 LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib:" /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/f -Z unstable-options --test-threads=1 --format=json > /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json || true
[01:23:47] cat /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json | "/usr/bin/python2.7" validate_json.py
[01:23:47] # Compare to output file
[01:23:47] diff output.json /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/libtest-json/libtest-json/libtest-json-output.json
[01:23:47] 2,9c2,9
[01:23:47] < { "type": "test", "event": "started", "name": "a" }
[01:23:47] < { "type": "test", "name": "a", "event": "ok" }
[01:23:47] < { "type": "test", "event": "started", "name": "b" }
[01:23:47] < { "type": "test", "name": "b", "event": "failed", "stdout": "thread 'b' panicked at 'assertion failed: false', f.rs:18:5\nnote: Run with `RUST_BACKTRACE=1` for a backtrace.\n" }
[01:23:47] < { "type": "test", "event": "started", "name": "c" }
[01:23:47] < { "type": "test", "name": "c", "event": "ok" }
[01:23:47] < { "type": "test", "event": "started", "name": "d" }
[01:23:47] < { "type": "test", "name": "d", "event": "ignored" }
[01:23:47] ---
[01:23:47] > { "type": "test", "event": "started", "name": "f::a" }
[01:23:47] > { "type": "test", "name": "f::a", "event": "ok" }
[01:23:47] > { "type": "test", "event": "started", "name": "f::b" }
[01:23:47] > { "type": "test", "name": "f::b", "event": "failed", "stdout": "thread 'f::b' panicked at 'assertion failed: false', f.rs:18:5\nnote: Run with `RUST_BACKTRACE=1` for a backtrace.\n" }
[01:23:47] > { "type": "test", "event": "started", "name": "f::c" }
[01:23:47] > { "type": "test", "name": "f::c", "event": "ok" }
[01:23:47] > { "type": "test", "event": "started", "name": "f::d" }
[01:23:47] > { "type": "test", "name": "f::d", "event": "ignored" }
[01:23:47] Makefile:8: recipe for target 'all' failed
[01:23:47] make[1]: Leaving directory '/checkout/src/test/run-make-fulldeps/libtest-json'
[01:23:47] ------------------------------------------
[01:23:47] stderr:
[01:23:47] ------------------------------------------
[01:23:47] ------------------------------------------
[01:23:47] make[1]: *** [all] Error 1
[01:23:47] ------------------------------------------
[01:23:47] 
[01:23:47] 
[01:23:47] thread '[run-make] run-make-fulldeps/libtest-json' panicked at 'explicit panic', tools/compiletest/src/runtest.rs:3166:9
[01:23:47] 
[01:23:47] 
[01:23:47] failures:
[01:23:47]     [run-make] run-make-fulldeps/libtest-json
[01:23:47]     [run-make] run-make-fulldeps/libtest-json
[01:23:47] 
[01:23:47] test result: FAILED. 188 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out
[01:23:47] 
[01:23:47] 
[01:23:47] 
[01:23:47] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/run-make-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-5.0/bin/FileCheck" "--host-rustcflags" "-Crpath -O -Zunstable-options " "--target-rustcflags" "-Crpath -O -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "5.0.0\n" "--system-llvm" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--llvm-components" "aarch64 aarch64asmparser aarch64asmprinter aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils all all-targets amdgpu amdgpuasmparser amdgpuasmprinter amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armasmprinter armcodegen armdesc armdisassembler arminfo asmparser asmprinter binaryformat bitreader bitwriter bpf bpfasmprinter bpfcodegen bpfdesc bpfdisassembler bpfinfo codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfomsf debuginfopdb demangle dlltooldriver engine executionengine globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader lanai lanaiasmparser lanaiasmprinter lanaicodegen lanaidesc lanaidisassembler lanaiinfo libdriver lineeditor linker lto mc mcdisassembler mcjit mcparser mips mipsasmparser mipsasmprinter mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmprinter msp430codegen msp430desc msp430info native nativecodegen nvptx nvptxasmprinter nvptxcodegen nipe for target 'check' failed

The command "stamp sh -x -c "$RUN_SCRIPT"" exited with 2.
travis_time:start:062c3400
$ date && (curl -fs --head https://google.com | grep ^Date: | sed 's/Date: //g' || true)

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

@djrenren djrenren changed the title WIP: Introduce Custom Test Frameworks Introduce Custom Test Frameworks Aug 18, 2018
@djrenren
Copy link
Contributor Author

@alexcrichton That was definitely the issue. Should be all fixed up!

@bors
Copy link
Contributor

bors commented Aug 21, 2018

☔ The latest upstream changes (presumably #53471) made this pull request unmergeable. Please resolve the merge conflicts.

rustc_target = { path = "../librustc_target" }
log = "0.4"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing newline

Copy link
Member

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty awesome, thanks so much for all this! I'm pretty excited in eventually stabilizing low-level tools like #[test_case] to help custom test frameworks/runners execute.

Some overall comments I have are:

  • Could documentation for these attributes be added to the unstable book? This'll help keep track of their current implementation as well as their intended directions.
  • One risk I see here is that if you accidentally mix #[test] with a custom test runner that isn't expecting the data structures in the test crate it'll give some wonky errors, right? Or if not, could a UI test be added for this to see what it looks like?
  • I think this'll definitely solve wasm-bindgen's use case in the sense that wasm-bindgen largely just needs the ability to aggregate tests, and I think we could work with this. Definitely something I'd like to make sure to game out before stabilization, but we've got quite some time to do that!
  • This is touching a very heavily used aspect of rustc, so we should definitely have a crater run before landing. If you want to rebase I'll kick off the try run so we can get that going!
  • Could some more tests be added perhaps exercising different sorts of test frameworks, mixing them together, expanding to different data structures, etc? It'd also be great to have a "pretty complete" example which involves usage of a procedural macro as well to help ensure that all the plumbing is hooked up

let path = {
let attr = attr.as_ref().unwrap();

// A hack to make #[test] not shadowable. This is in line with legacy behavior.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check seems a bit suspect to me in the sense that we're in general going towards a world where it's fine to shadow built-in attributes (for the most part). Instead of this though I'd prefer to make it an error to define or import a macro called test rather than forcing resolution always go to the macro prelude.

Could you elaborate a bit though on how this came up?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing, there's this phenomenon where people have defined macro_rules macros called test! that they use as some part of generating tests. For example:

macro_rules! test {
(case: $test_name:ident,
text: $text:expr,
source_file_start_pos: $source_file_start_pos:expr,
lines: $lines:expr,
multi_byte_chars: $multi_byte_chars:expr,
non_narrow_chars: $non_narrow_chars:expr,) => (
#[test]
fn $test_name() {

We see this in many places even just within the compiler but a cursory github search also reveals there are many.

No one would be happier than me to remove this hack, but I figured it best to start conservative and remove later if we can.

self.tests.push(item.ident);

// Ensure this item is capable of being reexported
item.vis = respan(item.vis.span, ast::VisibilityKind::Public);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems slightly suspect to me in the sense that it messes with the module system in a user-visible fashion, but I haven't finished reading this PR yet so I'll keep going!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah this is actually a mistake. This is already handled in expand.rs where we both gensym and make public so it's not user-visible. This line essentially a no-op currently.

@@ -907,7 +907,8 @@ mod tests {

#[cfg(test)]
mod bench {
use Bencher;
extern crate test;
use self::test::Bencher;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a little odd, but this is the whole "how does test test itself" problem, right? If so, could these tests be moved to an external location like src/libtest/benches/stats.rs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah so now libtest is being tested by the compiler's libtest.

The first two are actual benchmarks of the stats sum method, but I think all 3 are really just smoke tests and may make most sense as a compiletest.

@djrenren
Copy link
Contributor Author

djrenren commented Aug 22, 2018

Just pushed a rebase, so we can get the crater run going. (also removed the redundant publicizing). I'll work on moving the libtest tests and producing more tests next.

@alexcrichton
Copy link
Member

@bors: try

I'll take a bit closer look responses soon

@bors
Copy link
Contributor

bors commented Aug 22, 2018

⌛ Trying commit 44db252 with merge 424c902...

bors added a commit that referenced this pull request Aug 22, 2018
Introduce Custom Test Frameworks

Introduces `#[test_case]` and `#[test_runner]` and re-implements `#[test]` and `#[bench]` in terms of them.

Details found here: https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html
Copy link
Member

@alexcrichton alexcrichton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So just to make sure I understand everything that's going on here as well, there's currently logic which uses a "clever series" of modules to make all the #[test]-annotated functions available at the root of the crate. We then generate an entry point which takes all these functions, assembles some structures, and then passes a list of them to the test crate.

After this change that is somewhat different. Instead all #[test_case] annotated items are reexported to the root of the crate verbatim. Then a mutable reference to all #[test_case] items is passed to the "test runner", which takes over. The #[test] attribute generates const values which are the struct that libtest expects.

Does that sound like what's going on?

@@ -475,6 +475,10 @@ impl<'a, 'cl> Resolver<'a, 'cl> {
return def;
}

if kind == MacroKind::Attr && *&path[0].as_str() == "test" {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @petrochenkov, do you know how this might be best implemented?

The gist of the change here is that #[test] is becoming a procedural macro attribute, but this is in practice conflicting with a lot of macro_rules! macros called test. I believe the intention here is that test as an attribute is always resolved to the macro attribute, but otherwise it goes through normal resolution rules.

@djrenren I think what we probably want, long term, here is to have a forward-compatibility warning issued. Crates which require this overriding behavior of "#[test] trumps test!" will want to get a warning that eventually we'd like to remove this hack in the future, and the macro should be renamed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm all for it 👍

vec![field("desc", desc_expr),
field("testfn", testfn_expr)])
fn get_test_runner(sd: &errors::Handler, krate: &ast::Crate) -> Option<ast::Path> {
if let Some(test_attr) = attr::find_by_name(&krate.attrs, "test_runner") {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this use ? and early returns on invalid cases to reduce the indentation a bit?

Copy link
Contributor Author

@djrenren djrenren Aug 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah totally could (and will)

src/libsyntax_ext/test.rs Show resolved Hide resolved
src/libsyntax_ext/test.rs Show resolved Hide resolved
src/libsyntax/test.rs Show resolved Hide resolved
@bors
Copy link
Contributor

bors commented Aug 22, 2018

☀️ Test successful - status-travis
State: approved= try=True

@djrenren
Copy link
Contributor Author

After this change that is somewhat different...

Yeah that's what's going on. Though further reflection and talking to @nrc, makes me think we might want to use immutable references at first and see if we need mutability

@alexcrichton
Copy link
Member

@craterbot ping

@craterbot
Copy link
Collaborator

🔒 Error: you're not allowed to interact with this bot.

🔑 If you are a member of the Rust team and need access, add yourself to the whitelist.
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

@alexcrichton
Copy link
Member

@rust-lang/infra mind scheduling a crater run for this PR?

@alexcrichton
Copy link
Member

@bors: r+ delegate+

@bors
Copy link
Contributor

bors commented Sep 5, 2018

✌️ @djrenren can now approve this pull request

@bors
Copy link
Contributor

bors commented Sep 5, 2018

📌 Commit 0593dc7 has been approved by alexcrichton

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 5, 2018
@bors
Copy link
Contributor

bors commented Sep 5, 2018

⌛ Testing commit 0593dc7 with merge 3f13b27...

bors added a commit that referenced this pull request Sep 5, 2018
Introduce Custom Test Frameworks

Introduces `#[test_case]` and `#[test_runner]` and re-implements `#[test]` and `#[bench]` in terms of them.

Details found here: https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html
@bors
Copy link
Contributor

bors commented Sep 5, 2018

☀️ Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing 3f13b27 to master...

@bors bors merged commit 0593dc7 into rust-lang:master Sep 5, 2018
@ehuss ehuss mentioned this pull request Sep 8, 2018
bors added a commit that referenced this pull request Sep 10, 2018
resolve: Split macro prelude into built-in and user-defined parts

This is a refactoring that will help to remove `unshadowable_attrs` when #53410 lands.

UPDATE: The second commit actually removes `unshadowable_attrs`.
bors added a commit that referenced this pull request Sep 14, 2018
resolve: Introduce two sub-namespaces in macro namespace

Two sub-namespaces are introduced in the macro namespace - one for bang macros and one for attribute-like macros (attributes, derives).

"Sub-namespace" means this is not a newly introduced full namespace, the single macro namespace is still in place.
I.e. you still can't define/import two macros with the same name in a single module, `use` imports still import only one name in macro namespace (from any sub-namespace) and not possibly two.

However, when we are searching for a name used in a `!` macro call context (`my_macro!()`) we skip attribute names in scope, and when we are searching for a name used in attribute context (`#[my_macro]`/`#[derive(my_macro)]`) we are skipping bang macro names in scope.
In other words, bang macros cannot shadow attribute macros and vice versa.

For a non-macro analogy, we could e.g. skip non-traits when searching for `MyTrait` in `impl MyTrait for Type { ... }`.
However we do not do it in non-macro namespaces because we don't have practical issues with e.g. non-traits shadowing traits with the same name, but with macros we do, especially after macro modularization.

For `#[test]` and `#[bench]` we have a hack in the compiler right now preventing their shadowing by `macro_rules! test` and similar things. This hack was introduced after making `#[test]`/`#[bench]` built-in macros instead of built-in attributes (#53410), something that needed to be done from the start since they are "active" attributes transforming their inputs.
Now they are passed through normal name resolution and can be shadowed, but that's a breaking change, so we have  a special hack basically applying this PR for `#[test]` and `#[bench]` only.

Soon all potentially built-in attributes will be passed through normal name resolution (#53913) and that uncovers even more cases where the strict "macro namespace is a single namespace" rule needs to be broken.
For example, with strict rules, built-in macro `cfg!(...)` would shadow built-in attribute `#[cfg]` (they are different things), standard library macro `thread_local!(...)` would shadow built-in attribute `#[thread_local]` - both of these cases are covered by special hacks in #53913 as well.
Crater run uncovered more cases of attributes being shadowed by user-defined macros (`warn`, `doc`, `main`, even `deprecated`), we cannot add exceptions in the compiler for all of them.

Regressions with user-defined attributes like #53583 and #53898 also appeared after enabling macro modularization.

People are also usually confused (#53205 (comment), #53583 (comment)) when they see conflicts between attributes and non-attribute macros for the first time.

So my proposed solution is to solve this issue by introducing two sub-namespaces and thus skipping resolutions of the wrong kind and preventing more error-causing cases of shadowing.

Fixes #53583
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request May 19, 2021
…ou-se

Expose `Concurrent` (private type in public i'face)

rust-lang#53410 introduced experimental support for custom test frameworks.

Such frameworks may wish to build upon `library/test` by calling into its publicly exposed API (which I entirely understand is wholly unstable).  However, any that wish to call `test::run_test` cannot currently do so because `test::options::Concurrent` (the type of its `concurrent` parameter) is not publicly exposed.
Dylan-DPC-zz pushed a commit to Dylan-DPC-zz/rust that referenced this pull request May 19, 2021
…ou-se

Expose `Concurrent` (private type in public i'face)

rust-lang#53410 introduced experimental support for custom test frameworks.

Such frameworks may wish to build upon `library/test` by calling into its publicly exposed API (which I entirely understand is wholly unstable).  However, any that wish to call `test::run_test` cannot currently do so because `test::options::Concurrent` (the type of its `concurrent` parameter) is not publicly exposed.
RalfJung added a commit to RalfJung/rust that referenced this pull request May 19, 2021
…ou-se

Expose `Concurrent` (private type in public i'face)

rust-lang#53410 introduced experimental support for custom test frameworks.

Such frameworks may wish to build upon `library/test` by calling into its publicly exposed API (which I entirely understand is wholly unstable).  However, any that wish to call `test::run_test` cannot currently do so because `test::options::Concurrent` (the type of its `concurrent` parameter) is not publicly exposed.
@crlf0710 crlf0710 added the F-custom_test_frameworks `#![feature(custom_test_frameworks)]` label Dec 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
F-custom_test_frameworks `#![feature(custom_test_frameworks)]` S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.