Merge branch 'feature/redrawhook'

* feature/redrawhook:
  docs: Track making the new codepath conditional upon the 'memo=' feature.
  On the feature/redrawhook branch, changelog: Add entries for issues fixed by this branch.
  On the feature/redrawhook branch, change the detection of the 'memo=' feature to avoid a catch-22.
  driver: Make the redrawhook codepath conditional upon the memo= feature.
  On the feature/redrawhook branch, move the changelog entry to the current release's section.
  driver: Fix a bug that prevented subsequent, third-party zle-line-pre-redraw hooks from running.
  driver: Do not pass widget arguments to _zsh_highlight
  driver: Clarify comment.  No functional change.
  driver: Allow for -U in autoloaded function definition
  driver: Use idiomatic module check
  driver: Make the shadowing $WIDGET read only.
  driver: Avoid a fork in the common case.
  test harness: Actually test the new code.
  driver: Rewrite without a state variable
  noop: Make a whitespace-only change to reduce noise in the next commit.
  docs: Rewrap.
  docs: Update FAQ answer per changes on this branch.
  redo _zsh_highlight__function_callable_p
  driver: Use a different way of checking whether add-zle-hook-widget is present.
  changelog: Use a more specific link.
  changelog: Note the effect of fixing #245/#90 and an alternative.
  driver: Pass zle-line-finish arguments on to _zsh_highlight.
  driver: Hook zle-line-finish.
  driver: Reimplement using 'add-zle-hook-widget zle-line-pre-redraw'
  wrappers: Reimplement using Mikachu's zle-line-pre-redraw hook (workers/36650).

README.md

@@ -44,11 +44,23 @@ FAQ
 
 ### Why must `zsh-syntax-highlighting.zsh` be sourced at the end of the `.zshrc` file?
 
-`zsh-syntax-highlighting.zsh` wraps ZLE widgets.  It must be sourced after all
+zsh-syntax-highlighting works by hooking into the Zsh Line Editor (ZLE) and
-custom widgets have been created (i.e., after all `zle -N` calls and after
+computing syntax highlighting for the command-line buffer as it stands at the
-running `compinit`).  Widgets created later will work, but will not update the
+time z-sy-h's hook is invoked.
+
+In zsh 5.2 and older,
+`zsh-syntax-highlighting.zsh` hooks into ZLE by wrapping ZLE widgets.  It must
+be sourced after all custom widgets have been created (i.e., after all `zle -N`
+calls and after running `compinit`) in order to be able to wrap all of them.
+Widgets created after z-sy-h is sourced will work, but will not update the
 syntax highlighting.
 
+In zsh newer than 5.8 (not including 5.8 itself),
+zsh-syntax-highlighting uses the `add-zle-hook-widget` facility to install
+a `zle-line-pre-redraw` hook.  Hooks are run in order of registration,
+therefore, z-sy-h must be sourced (and register its hook) after anything else
+that adds hooks that modify the command-line buffer.
+
 ### Does syntax highlighting work during incremental history search?
 
 Highlighting the command line during an incremental history search (by default bound to

changelog.md

@@ -1,6 +1,57 @@
 # Changes in HEAD
 
 
+## Changes fixed as part of the switch to zle-line-pre-redraw
+
+The changes in this section were fixed by switching to a `zle-line-pre-redraw`-based
+implementation.
+
+Note: The new implementation will only be used on future zsh releases,
+numbered 5.8.0.3 and newer, due to interoperability issues with other plugins
+(issues #418 and #579).  The underlying zsh feature has been available since
+zsh 5.2.
+
+Whilst under development, the new implementation was known as the
+"feature/redrawhook" topic branch.
+
+- Fixed: Highlighting not triggered after popping a buffer from the buffer stack
+  (using the `push-line` widget, default binding: `M-q`)
+  [#40]
+
+- Fixed: Invoking completion when there were no matches removed highlighting
+  [#90, #470]
+
+- Fixed: Two successive deletes followed by a yank only yanked the latest
+  delete, rather than both of them
+  [#150, #151, #160; cf. #183]
+
+- Presumed fixed: Completing `$(xsel)` results in an error message from `xsel`,
+  with pre-2017 versions of `xsel`.  (For 2017 vintage and newer, see the issue
+  for details.)
+  [#154]
+
+- Fixed: When the standard `bracketed-paste-magic` widget is in use, pastes were slow
+  [#295]
+
+- Fixed: No way to prevent a widget from being wrapped
+  [#324]
+
+- Fixed: No highlighting while cycling menu completion
+  [#375]
+
+- Fixed: Does not coexist with the `IGNORE_EOF` option
+  [#377]
+
+- Fixed: The `undefined-key` widget was wrapped
+  [#421]
+
+- Fixed: Does not coexist with the standard `surround` family of widgets
+  [#520]
+
+- Fixed: First completed filename doesn't get `path` highlighting
+  [#632]
+
+
 # Changes in 0.8.0-alpha1-pre-redrawhook
 
 ## Notice about an improbable-but-not-impossible forward incompatibility
@@ -19,6 +70,25 @@ added to zsh at z-sy-h's initiative.  The new feature is used in the fix
 to issue #418.
 
 
+## Incompatible changes:
+
+- An unsuccessful completion (a <kbd>⮀ Tab</kbd> press that doesn't change the
+  command line) no longer causes highlighting to be lost.  Visual feedback can
+  alternatively be achieved by setting the `format` zstyle under the `warnings`
+  tag, for example,
+
+        zstyle ':completion:*:warnings' format '%F{red}No matches%f'
+
+    Refer to the [description of the `format` style in `zshcompsys(1)`]
+    [zshcompsys-Standard-Styles-format].
+
+    (#90, part of #245 (feature/redrawhook))
+
+[zshcompsys-Standard-Styles]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html#Standard-Styles
+[zshcompsys-Standard-Styles-format]: http://zsh.sourceforge.net/Doc/Release/Completion-System.html#index-format_002c-completion-style
+  
+
+
 ## Other changes:
 
 - Document `$ZSH_HIGHLIGHT_MAXLENGTH`.

tests/generate.zsh

@@ -31,6 +31,9 @@
 emulate -LR zsh
 setopt localoptions extendedglob
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 # Argument parsing.
 if (( $# * $# - 7 * $# + 12 )) || [[ $1 == -* ]]; then
   print -r -- >&2 "$0: usage: $0 BUFFER HIGHLIGHTER BASENAME [PREAMBLE]"

tests/test-highlighting.zsh

@@ -31,6 +31,9 @@
 
 setopt NO_UNSET WARN_CREATE_GLOBAL
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 local -r root=${0:h:h}
 local -a anon_argv; anon_argv=("$@")
 

tests/test-perfs.zsh

@@ -29,6 +29,9 @@
 # -------------------------------------------------------------------------------------------------
 
 
+# Required for add-zle-hook-widget.
+zmodload zsh/zle
+
 # Check an highlighter was given as argument.
 [[ -n "$1" ]] || {
   echo >&2 "Bail out! You must provide the name of a valid highlighter as argument."

zsh-syntax-highlighting.zsh

@@ -49,6 +49,52 @@ if true; then
   fi
 fi
 
+# This function takes a single argument F and returns True iff F is an autoload stub.
+_zsh_highlight__function_is_autoload_stub_p() {
+  if zmodload -e zsh/parameter; then
+    #(( ${+functions[$1]} )) &&
+    [[ "$functions[$1]" == *"builtin autoload -X"* ]]
+  else
+    #[[ $(type -wa -- "$1") == *'function'* ]] &&
+    [[ "${${(@f)"$(which -- "$1")"}[2]}" == $'\t'$histchars[3]' undefined' ]]
+  fi
+  # Do nothing here: return the exit code of the if.
+}
+
+# Return True iff the argument denotes a function name.
+_zsh_highlight__is_function_p() {
+  if zmodload -e zsh/parameter; then
+    (( ${+functions[$1]} ))
+  else
+    [[ $(type -wa -- "$1") == *'function'* ]]
+  fi
+}
+
+# This function takes a single argument F and returns True iff F denotes the
+# name of a callable function.  A function is callable if it is fully defined
+# or if it is marked for autoloading and autoloading it at the first call to it
+# will succeed.  In particular, if a function has been marked for autoloading
+# but is not available in $fpath, then this function will return False therefor.
+#
+# See users/21671 http://www.zsh.org/cgi-bin/mla/redirect?USERNUMBER=21671
+_zsh_highlight__function_callable_p() {
+  if _zsh_highlight__is_function_p "$1" &&
+     ! _zsh_highlight__function_is_autoload_stub_p "$1"
+  then
+    # Already fully loaded.
+    return 0 # true
+  else
+    # "$1" is either an autoload stub, or not a function at all.
+    #
+    # Use a subshell to avoid affecting the calling shell.
+    #
+    # We expect 'autoload +X' to return non-zero if it fails to fully load
+    # the function.
+    ( autoload -U +X -- "$1" 2>/dev/null )
+    return $?
+  fi
+}
+
 # -------------------------------------------------------------------------------------------------
 # Core highlighting update system
 # -------------------------------------------------------------------------------------------------
@@ -351,72 +397,116 @@ _zsh_highlight_call_widget()
   _zsh_highlight
 }
 
-# Rebind all ZLE widgets to make them invoke _zsh_highlights.
+# Decide whether to use the zle-line-pre-redraw codepath (colloquially known as
-_zsh_highlight_bind_widgets()
+# "feature/redrawhook", after the topic branch's name) or the legacy "bind all
-{
+# widgets" codepath.
-  setopt localoptions noksharrays
+#
-  typeset -F SECONDS
+# We use the new codepath under two conditions:
-  local prefix=orig-s$SECONDS-r$RANDOM # unique each time, in case we're sourced more than once
+#
-
+# 1. If it's available, which we check by testing for add-zle-hook-widget's availability.
-  # Load ZSH module zsh/zleparameter, needed to override user defined widgets.
+# 
-  zmodload zsh/zleparameter 2>/dev/null || {
+# 2. If zsh has the memo= feature, which is required for interoperability reasons.
-    print -r -- >&2 'zsh-syntax-highlighting: failed loading zsh/zleparameter.'
+#    See issues #579 and #735, and the issues referenced from them.
-    return 1
+#
+#    We check this with a plain version number check, since a functional check,
+#    as done by _zsh_highlight, can only be done from inside a widget
+#    function — a catch-22.
+#
+#    See _zsh_highlight for the magic version number.  (The use of 5.8.0.2
+#    rather than 5.8.0.3 as in the _zsh_highlight is deliberate.)
+if is-at-least 5.8.0.2 && _zsh_highlight__function_callable_p add-zle-hook-widget
+then
+  autoload -U add-zle-hook-widget
+  _zsh_highlight__zle-line-finish() {
+    # Reset $WIDGET since the 'main' highlighter depends on it.
+    #
+    # Since $WIDGET is declared by zle as read-only in this function's scope,
+    # a nested function is required in order to shadow its built-in value;
+    # see "User-defined widgets" in zshall.
+    () {
+      local -h -r WIDGET=zle-line-finish
+      _zsh_highlight
+    }
   }
+  _zsh_highlight__zle-line-pre-redraw() {
+    # Set $? to 0 for _zsh_highlight.  Without this, subsequent
+    # zle-line-pre-redraw hooks won't run, since add-zle-hook-widget happens to
+    # call us with $? == 1 in the common case.
+    true && _zsh_highlight "$@"
+  }
+  _zsh_highlight_bind_widgets(){}
+  if [[ -o zle ]]; then
+    add-zle-hook-widget zle-line-pre-redraw _zsh_highlight__zle-line-pre-redraw
+    add-zle-hook-widget zle-line-finish _zsh_highlight__zle-line-finish
+  fi
+else
+  # Rebind all ZLE widgets to make them invoke _zsh_highlights.
+  _zsh_highlight_bind_widgets()
+  {
+    setopt localoptions noksharrays
+    typeset -F SECONDS
+    local prefix=orig-s$SECONDS-r$RANDOM # unique each time, in case we're sourced more than once
 
-  # Override ZLE widgets to make them invoke _zsh_highlight.
+    # Load ZSH module zsh/zleparameter, needed to override user defined widgets.
-  local -U widgets_to_bind
+    zmodload zsh/zleparameter 2>/dev/null || {
-  widgets_to_bind=(${${(k)widgets}:#(.*|run-help|which-command|beep|set-local-history|yank|yank-pop)})
+      print -r -- >&2 'zsh-syntax-highlighting: failed loading zsh/zleparameter.'
+      return 1
+    }
 
-  # Always wrap special zle-line-finish widget. This is needed to decide if the
+    # Override ZLE widgets to make them invoke _zsh_highlight.
-  # current line ends and special highlighting logic needs to be applied.
+    local -U widgets_to_bind
-  # E.g. remove cursor imprint, don't highlight partial paths, ...
+    widgets_to_bind=(${${(k)widgets}:#(.*|run-help|which-command|beep|set-local-history|yank|yank-pop)})
-  widgets_to_bind+=(zle-line-finish)
 
-  # Always wrap special zle-isearch-update widget to be notified of updates in isearch.
+    # Always wrap special zle-line-finish widget. This is needed to decide if the
-  # This is needed because we need to disable highlighting in that case.
+    # current line ends and special highlighting logic needs to be applied.
-  widgets_to_bind+=(zle-isearch-update)
+    # E.g. remove cursor imprint, don't highlight partial paths, ...
+    widgets_to_bind+=(zle-line-finish)
 
-  local cur_widget
+    # Always wrap special zle-isearch-update widget to be notified of updates in isearch.
-  for cur_widget in $widgets_to_bind; do
+    # This is needed because we need to disable highlighting in that case.
-    case ${widgets[$cur_widget]:-""} in
+    widgets_to_bind+=(zle-isearch-update)
 
-      # Already rebound event: do nothing.
+    local cur_widget
-      user:_zsh_highlight_widget_*);;
+    for cur_widget in $widgets_to_bind; do
+      case ${widgets[$cur_widget]:-""} in
 
-      # The "eval"'s are required to make $cur_widget a closure: the value of the parameter at function
+        # Already rebound event: do nothing.
-      # definition time is used.
+        user:_zsh_highlight_widget_*);;
-      #
-      # We can't use ${0/_zsh_highlight_widget_} because these widgets are always invoked with
-      # NO_function_argzero, regardless of the option's setting here.
 
-      # User defined widget: override and rebind old one with prefix "orig-".
+        # The "eval"'s are required to make $cur_widget a closure: the value of the parameter at function
-      user:*) zle -N $prefix-$cur_widget ${widgets[$cur_widget]#*:}
+        # definition time is used.
-              eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget ${(q)prefix}-${(q)cur_widget} -- \"\$@\" }"
+        #
-              zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
+        # We can't use ${0/_zsh_highlight_widget_} because these widgets are always invoked with
+        # NO_function_argzero, regardless of the option's setting here.
 
-      # Completion widget: override and rebind old one with prefix "orig-".
+        # User defined widget: override and rebind old one with prefix "orig-".
-      completion:*) zle -C $prefix-$cur_widget ${${(s.:.)widgets[$cur_widget]}[2,3]} 
+        user:*) zle -N $prefix-$cur_widget ${widgets[$cur_widget]#*:}
-                    eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget ${(q)prefix}-${(q)cur_widget} -- \"\$@\" }"
+                eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget ${(q)prefix}-${(q)cur_widget} -- \"\$@\" }"
-                    zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
+                zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
 
-      # Builtin widget: override and make it call the builtin ".widget".
+        # Completion widget: override and rebind old one with prefix "orig-".
-      builtin) eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget .${(q)cur_widget} -- \"\$@\" }"
+        completion:*) zle -C $prefix-$cur_widget ${${(s.:.)widgets[$cur_widget]}[2,3]}
-               zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
+                      eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget ${(q)prefix}-${(q)cur_widget} -- \"\$@\" }"
+                      zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
 
-      # Incomplete or nonexistent widget: Bind to z-sy-h directly.
+        # Builtin widget: override and make it call the builtin ".widget".
-      *) 
+        builtin) eval "_zsh_highlight_widget_${(q)prefix}-${(q)cur_widget}() { _zsh_highlight_call_widget .${(q)cur_widget} -- \"\$@\" }"
-         if [[ $cur_widget == zle-* ]] && (( ! ${+widgets[$cur_widget]} )); then
+                 zle -N $cur_widget _zsh_highlight_widget_$prefix-$cur_widget;;
-           _zsh_highlight_widget_${cur_widget}() { :; _zsh_highlight }
+
-           zle -N $cur_widget _zsh_highlight_widget_$cur_widget
+        # Incomplete or nonexistent widget: Bind to z-sy-h directly.
-         else
+        *)
-      # Default: unhandled case.
+           if [[ $cur_widget == zle-* ]] && (( ! ${+widgets[$cur_widget]} )); then
-           print -r -- >&2 "zsh-syntax-highlighting: unhandled ZLE widget ${(qq)cur_widget}"
+             _zsh_highlight_widget_${cur_widget}() { :; _zsh_highlight }
-           print -r -- >&2 "zsh-syntax-highlighting: (This is sometimes caused by doing \`bindkey <keys> ${(q-)cur_widget}\` without creating the ${(qq)cur_widget} widget with \`zle -N\` or \`zle -C\`.)"
+             zle -N $cur_widget _zsh_highlight_widget_$cur_widget
-         fi
+           else
-    esac
+        # Default: unhandled case.
-  done
+             print -r -- >&2 "zsh-syntax-highlighting: unhandled ZLE widget ${(qq)cur_widget}"
-}
+             print -r -- >&2 "zsh-syntax-highlighting: (This is sometimes caused by doing \`bindkey <keys> ${(q-)cur_widget}\` without creating the ${(qq)cur_widget} widget with \`zle -N\` or \`zle -C\`.)"
+           fi
+      esac
+    done
+  }
+fi
 
 # Load highlighters from directory.
 #