Add backtrace-mode and use it in the debugger, ERT and Edebug

* doc/lispref/debugging.texi (Using Debugger): Remove explanation of
backtrace buffer.  Refer to new node.
(Backtraces): New node.
(Debugger Commands): Refer to new node.  Remove 'v'.
* doc/lispref/edebug.texi (Edebug Misc): Refer to new node.
* doc/misc/ert.texi (Running Tests Interactively): Refer to new node.

* lisp/emacs-lisp-backtrace.el: New file.
* test/lisp/emacs-lisp/backtrace-tests.el: New file.

* lisp/emacs-lisp/debug.el: (debugger-buffer-state): New cl-defstruct.
(debugger--restore-buffer-state): New function.
(debug): Use a debugger-buffer-state object to save and restore buffer
state.  Fix bug#15749 by leaving an unused buffer in debugger-mode,
empty, instead of in fundamental-mode, and then when reusing a buffer,
not calling debugger-mode if the buffer is already in debugger-mode.
(debugger-insert-backtrace): Remove.
(debugger-setup-buffer): Use backtrace-mode.
(debugger--insert-header): New function.
(debugger-continue, debugger-return-value): Change check for flags to
use backtrace-frames.
(debugger-frame-number): Determine backtrace frame number from
backtrace-frames.
(debugger--locals-visible-p, debugger--insert-locals)
(debugger--show-locals, debugger--hide-locals)
(debugger-toggle-locals): Remove.
(debugger-mode-map): Make a child of backtrace-mode-map.  Move
navigation commands to backtrace-mode-map.  Bind 'q' to debugger-quit
instead of top-level.  Make Help Follow menu item call
backtrace-help-follow-symbol.
(debugger-mode): Derive from backtrace-mode.
(debug-help-follow): Remove.  Move body of this function to
'backtrace-help-follow-symbol' in backtrace.el.
(debugger-quit): New function.

* lisp/emacs-lisp/edebug.el (edebug-unwrap-results): Remove warning
in docstring about circular results.
(edebug-unwrap): Use pcase.
(edebug-unwrap1): New function to unwrap circular objects.
(edebug-unwrap*): Use it.
(edebug--frame): New cl-defstruct.
(edebug-backtrace): Call the buffer *Edebug Backtrace* and use
backtrace-mode.  Get the frames from edebug--backtrace-frames.
(edebug--backtrace-frames, edebug--unwrap-and-add-info)
(edebug--symbol-not-prefixed-p): New functions.

* lisp/emacs-lisp/lisp-mode.el
(lisp-el-font-lock-keywords-for-backtraces)
(lisp-el-font-lock-keywords-for-backtraces-1)
(lisp-el-font-lock-keywords-for-backtraces-2): New constants.

* lisp/emacs-lisp/ert.el (ert--print-backtrace): Remove.
(ert--run-test-debugger): Use backtrace-get-frames.
(ert-run-tests-batch): Use backtrace-to-string.
(ert-results-pop-to-backtrace-for-test-at-point): Use backtrace-mode.
(ert--insert-backtrace-header): New function.

* tests/lisp/emacs-lisp/ert-tests.el (ert-test--which-file):
Use backtrace-frame slot accessor.

doc/lispref/debugging.texi

@@ -81,7 +81,8 @@ debugger recursively.  @xref{Recursive Editing}.
 * Function Debugging::    Entering it when a certain function is called.
 * Variable Debugging::    Entering it when a variable is modified.
 * Explicit Debug::        Entering it at a certain point in the program.
-* Using Debugger::        What the debugger does; what you see while in it.
+* Using Debugger::        What the debugger does.
+* Backtraces::            What you see while in the debugger.
 * Debugger Commands::     Commands used while in the debugger.
 * Invoking the Debugger:: How to call the function @code{debug}.
 * Internals of Debugger:: Subroutines of the debugger, and global variables.
@@ -392,32 +393,79 @@ this is not what you want, you can either set
 @code{eval-expression-debug-on-error} to @code{nil}, or set
 @code{debug-on-error} to @code{nil} in @code{debugger-mode-hook}.
 
+  The debugger itself must be run byte-compiled, since it makes
+assumptions about the state of the Lisp interpreter.  These
+assumptions are false if the debugger is running interpreted.
+
+@node Backtraces
+@subsection Backtraces
+@cindex backtrace buffer
+
+Debugger mode is derived from Backtrace mode, which is also used to
+show backtraces by Edebug and ERT.  (@pxref{Edebug} and @ref{Top,the
+ERT manual,, ert, ERT: Emacs Lisp Regression Testing})
+
+@cindex stack frame
+The backtrace buffer shows you the functions that are executing and
+their argument values.  When a backtrace buffer is created, it shows
+each stack frame on one, possibly very long, line.  (A stack frame is
+the place where the Lisp interpreter records information about a
+particular invocation of a function.)  The most recently called
+function will be at the top.
+
 @cindex current stack frame
-  The backtrace buffer shows you the functions that are executing and
-their argument values.  It also allows you to specify a stack frame by
-moving point to the line describing that frame.  (A stack frame is the
-place where the Lisp interpreter records information about a particular
-invocation of a function.)  The frame whose line point is on is
-considered the @dfn{current frame}.  Some of the debugger commands
-operate on the current frame.  If a line starts with a star, that means
-that exiting that frame will call the debugger again.  This is useful
-for examining the return value of a function.
-
-  If a function name is underlined, that means the debugger knows
-where its source code is located.  You can click with the mouse on
-that name, or move to it and type @key{RET}, to visit the source code.
+In a backtrace you can specify a stack frame by moving point to a line
+describing that frame.  The frame whose line point is on is considered
+the @dfn{current frame}.
+
+If a function name is underlined, that means Emacs knows where its
+source code is located.  You can click with the mouse on that name, or
+move to it and type @key{RET}, to visit the source code.  You can also
+type @key{RET} while point is on any name of a function or variable
+which is not underlined, to see help information for that symbol in a
+help buffer, if any exists. The @code{xref-find-definitions} command,
+bound to @key{M-.}, can also be used on any identifier in a backtrace
+(@pxref{Looking Up Identifiers,,,emacs,Emacs manual}).
+
+In backtraces, the tails of long lists and the ends of long strings,
+vectors or structures, as well as objects which are deeply nested,
+will be printed as underlined ``...''.  You can click with the mouse
+on a ``...'', or type @key{RET} while point is on it, to show the part
+of the object that was hidden.  To control how much abbreviation is
+done, customize @code{backtrace-line-length}.
+
+Here is a list of commands for navigating and viewing backtraces:
 
-  The debugger itself must be run byte-compiled, since it makes
-assumptions about how many stack frames are used for the debugger
-itself.  These assumptions are false if the debugger is running
-interpreted.
+@table @kbd
+@item v
+Toggle the display of local variables of the current stack frame.
+
+@item p
+Move to the beginning of the frame, or to the beginning
+of the previous frame.
+
+@item n
+Move to the beginning of the next frame.
+
+@item +
+Add line breaks and indentation to the top-level Lisp form at point to
+make it more readable.
+
+@item =
+Collapse the top-level Lisp form at point back to a single line.
+
+@item #
+Toggle @code{print-circle} for the frame at point.
+
+@end table
 
 @node Debugger Commands
 @subsection Debugger Commands
 @cindex debugger command list
 
   The debugger buffer (in Debugger mode) provides special commands in
-addition to the usual Emacs commands.  The most important use of
+addition to the usual Emacs commands and to the Backtrace mode commands
+described in the previous section.  The most important use of
 debugger commands is for stepping through code, so that you can see
 how control flows.  The debugger can step through the control
 structures of an interpreted function, but cannot do so in a
@@ -427,6 +475,11 @@ the same function.  (To do this, visit the source for the function and
 type @kbd{C-M-x} on its definition.)  You cannot use the Lisp debugger
 to step through a primitive function.
 
+Some of the debugger commands operate on the current frame.  If a
+frame starts with a star, that means that exiting that frame will call the
+debugger again.  This is useful for examining the return value of a
+function.
+
 @c FIXME: Add @findex for the following commands?  --xfq
   Here is a list of Debugger mode commands:
 
@@ -502,8 +555,6 @@ Display a list of functions that will invoke the debugger when called.
 This is a list of functions that are set to break on entry by means of
 @code{debug-on-entry}.
 
-@item v
-Toggle the display of local variables of the current stack frame.
 @end table
 
 @node Invoking the Debugger

doc/lispref/edebug.texi

@@ -442,8 +442,8 @@ Redisplay the most recently known expression result in the echo area
 Display a backtrace, excluding Edebug's own functions for clarity
 (@code{edebug-backtrace}).
 
-You cannot use debugger commands in the backtrace buffer in Edebug as
-you would in the standard debugger.
+@xref{Debugging,, Backtraces, elisp}, for the commands which work
+in a backtrace buffer.
 
 The backtrace buffer is killed automatically when you continue
 execution.

doc/misc/ert.texi

@@ -273,9 +273,11 @@ moving point to it and typing @kbd{@key{RET}} jumps to its definition.
 @cindex backtrace of a failed test
 Pressing @kbd{r} re-runs the test near point on its own.  Pressing
 @kbd{d} re-runs it with the debugger enabled.  @kbd{.} jumps to the
-definition of the test near point (@kbd{@key{RET}} has the same effect if
-point is on the name of the test).  On a failed test, @kbd{b} shows
-the backtrace of the failure.
+definition of the test near point (@kbd{@key{RET}} has the same effect
+if point is on the name of the test).  On a failed test, @kbd{b} shows
+the backtrace of the failure.  @xref{Debugging,, Backtraces, elisp,
+the Emacs Lisp Reference Manual}, for more information about
+backtraces.
 
 @kindex l@r{, in ert results buffer}
 @kbd{l} shows the list of @code{should} forms executed in the test.

etc/NEWS

@@ -466,6 +466,14 @@ the shift key.
 *** Isearch now remembers the regexp-based search mode for words/symbols
 and case-sensitivity together with search strings in the search ring.
 
+** Debugger
+
++++
+*** The Lisp Debugger is now based on 'backtrace-mode'.
+Backtrace mode adds fontification and commands for changing the
+appearance of backtrace frames. See the node "Backtraces" in the Elisp
+manual for documentation of the new mode and its commands.
+
 ** Edebug
 
 +++
@@ -475,14 +483,18 @@ using the new variables 'edebug-behavior-alist',
 'edebug-new-definition-function'.  Edebug's behavior can be changed
 globally or for individual definitions.
 
++++
+*** Edebug's backtrace buffer now uses 'backtrace-mode'.
+Backtrace mode adds fontification, links and commands for changing the
+appearance of backtrace frames. See the node "Backtraces" in the Elisp
+manual for documentation of the new mode and its commands.
+
 ** Enhanced xterm support
 
 *** New variable 'xterm-set-window-title' controls whether Emacs sets
 the XTerm window title.  This feature is experimental and is disabled
 by default.
 
-** Gamegrid
-
 ** grep
 
 +++
@@ -499,6 +511,14 @@ The abbreviation can be disabled by the new option
 *** New variable 'ert-quiet' allows to make ERT output in batch mode
 less verbose by removing non-essential information.
 
++++
+*** ERT's backtrace buffer now uses 'backtrace-mode'.
+Backtrace mode adds fontification and commands for changing the
+appearance of backtrace frames. See the node "Backtraces" in the Elisp
+manual for documentation of the new mode and its commands.
+
+** Gamegrid
+
 ---
 *** Gamegrid now determines its default glyph size based on display
 dimensions, instead of always using 16 pixels.  As a result, Tetris,
@@ -669,6 +689,13 @@ transport strategies as well as a separate API to use them.  A
 transport implementation for process-based communication, such as is
 used by the Language Server Protocol (LSP), is readily available.
 
++++
+** Backtrace mode improves viewing of Elisp backtraces.
+Backtrace mode adds pretty printing, fontification and ellipsis
+expansion to backtrace buffers produced by the Lisp debugger, Edebug
+and ERT. See the node "Backtraces" in the Elisp manual for
+documentation of the new mode and its commands.
+
 
 * Incompatible Lisp Changes in Emacs 27.1