Debugging help through showing pipeline intermediates.

[ca8IiN4oow]

Describe the bug
A clear and concise description of what the bug is.

Debugging longer jq pipes is opaque and time consuming in part because the intermediate pipe line stage results are not available for inspection.

To Reproduce

echo '[ "a" , "b" , "c" , "d" ] ' | jq '. | map(ascii_upcase) | reverse | .[] | [ match("[ACD]").string ] | map(ascii_downcase) | reverse | add '

Produces :
"d"
"c"
null
"a"

Where did the null come from? People new to jq will take a while to figure it out.

Expected behavior
jq --show_your_work_please ...{ the rest of it. }

For each stage in the pipeline produce 3 artifacts, the raw before, the raw after and a diff between the two. Optionally wrapping it all up into some HTML would be nice tie the entire pipeline together both visually and logically.

Environment (please complete the following information):
jq -V ; uname -a ; cat /etc/os-release
jq-1.5
Linux localhost.localdomain 4.18.0-193.19.1.el8_2.x86_64 #1 SMP Mon Sep 14 14:37:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

Additional context
JQ is the epoxy keeping the clouds together, thank you for it.

[wtlangford]

So, this is a feature that already exists! You can place the debug/0 filter anywhere you'd like in a pipeline and jq will print the filter's input to stderr as debug output:

echo '[ "a" , "b" , "c" , "d" ]' | jq 'map(ascii_upcase) | reverse[] | [ match("[ACD]").string ] | map(ascii_downcase) | reverse | debug | add'
["DEBUG:",{"offset":0,"length":1,"string":"D","captures":[]}]
["DEBUG:",["d"]]
"d"
["DEBUG:",{"offset":0,"length":1,"string":"C","captures":[]}]
["DEBUG:",["c"]]
"c"
["DEBUG:",[]]
null
["DEBUG:",{"offset":0,"length":1,"string":"A","captures":[]}]
["DEBUG:",["a"]]
"a"

For the reason we don't have a more general flag- by the time we're executing your program, the idea of the "pipeline" is gone and replaced with a program we run in a little custom-purpose VM (much in the same way python and ruby compile to bytecode). We'd have a difficult time figuring out the right place to print debug output, and in a sufficiently complex pipeline, it would be too hard to understand, because you wouldn't know where we'd placed the debug outputs.

[pkoppstein]

@wtlangford- it would really be quite helpful (and easy to implement) if debug/1 defined as follows were always available:

def debug(message): (message|debug)as $debug | .;

The documentation could give an example showing how message can include . or some interesting bit thereof.

[wtlangford]

I've actually wanted such a debug/1 in the past to print more complex debug details without changing my ..

debug/0 is actually implemented in C, and while we could define debug/1 in jq like that, I'd probably just define it in C as well to skip the variable binding step.

If you're interested in giving it a go yourself, you're more than welcome to! Otherwise, I'll probably add it soon.

[pkoppstein]

@wtlangford - Please go ahead!

Since debug's output is both rigid and often difficult to read, do you think that debug/2 would be helpful so that (for example) the input and the message can be easily distinguished? E.g.

def debug(E;F): (E|debug) as $d | (F|debug) as $d | .;

[yertto]

@wtlangford / @pkoppstein - any chance you could get behind the jq defined implementation in #2112 and we just get that merged?