Use Capture.ByteRange() from v2 instead of a hand-rolled rune to byte
offset mapping. The old mapping required capture offsets to be monotonic
across matches, so a capture inside a lookaround overlapping a later match
in non-ASCII text caused all matches to be silently dropped.
Also propagate match errors instead of discarding them. Drops the v1
dependency, v2 was already pulled in by chroma.
- Guard against the quotes post processor producing start > end for a match
consisting of just a single ASCII quote
- Use char_before() in the URL trailing punctuation loop and avoid computing
the closing bracket twice
- Add PR reference to changelog entry
The post-processors index the text by byte, so when looking at the
character before a position they decoded only the last byte of
multi-byte quotes such as ‘’ and “” and never matched them. URLs kept
the closing quote and paths kept both quotes. When the closing quote was
just outside the match, the start was also advanced by a single byte,
splitting the opening quote and producing invalid UTF-8, for example,
for the file name in wget's "Saving to: ‘index.html’".
This worked before the hints kitten was ported to Go.
Use Unicode word boundaries with the stdlib regexp engine
Go's regexp only treats ASCII characters as word characters for \b, while
regexp2 uses Unicode ones, so switching the builtin path pattern to the
stdlib engine changed which paths were marked next to non-ASCII text, for
example config.yaml文件 marked config.yaml. Match against a proxy of the
text in which every non-ASCII rune is replaced by a single ASCII byte that
is a word character iff the rune is one, then map the offsets back.
regexp2 backtracks on the path pattern, retrying it at every position
of a long line that has no spaces, so the path and linenum types took
seconds on a screen full of such text (about 36s for a 380x100 window
of hex). The builtin patterns dont need lookarounds, so use the linear
time stdlib engine for them. User supplied --regex patterns stay on
regexp2 so lookarounds keep working (#6265).
Note that we loose unicode char matching for --type=word because of
https://github.com/dlclark/regexp2/issues/65 and of course user regexps
cant use \p{N} escapes any more. Hopefully regexp2 will add support for
these soon-ish. IMO lookaround is more important than \p.
Fixes#6265