Use Capture.ByteRange() from v2 instead of a hand-rolled rune to byte
offset mapping. The old mapping required capture offsets to be monotonic
across matches, so a capture inside a lookaround overlapping a later match
in non-ASCII text caused all matches to be silently dropped.
Also propagate match errors instead of discarding them. Drops the v1
dependency, v2 was already pulled in by chroma.
- Guard against the quotes post processor producing start > end for a match
consisting of just a single ASCII quote
- Use char_before() in the URL trailing punctuation loop and avoid computing
the closing bracket twice
- Add PR reference to changelog entry
The post-processors index the text by byte, so when looking at the
character before a position they decoded only the last byte of
multi-byte quotes such as ‘’ and “” and never matched them. URLs kept
the closing quote and paths kept both quotes. When the closing quote was
just outside the match, the start was also advanced by a single byte,
splitting the opening quote and producing invalid UTF-8, for example,
for the file name in wget's "Saving to: ‘index.html’".
This worked before the hints kitten was ported to Go.
Use Unicode word boundaries with the stdlib regexp engine
Go's regexp only treats ASCII characters as word characters for \b, while
regexp2 uses Unicode ones, so switching the builtin path pattern to the
stdlib engine changed which paths were marked next to non-ASCII text, for
example config.yaml文件 marked config.yaml. Match against a proxy of the
text in which every non-ASCII rune is replaced by a single ASCII byte that
is a word character iff the rune is one, then map the offsets back.
regexp2 backtracks on the path pattern, retrying it at every position
of a long line that has no spaces, so the path and linenum types took
seconds on a screen full of such text (about 36s for a 380x100 window
of hex). The builtin patterns dont need lookarounds, so use the linear
time stdlib engine for them. User supplied --regex patterns stay on
regexp2 so lookarounds keep working (#6265).
* Switches to "positional notation" to fix the bug.
* Consolidates `encode_hint` and `generate_prefix_free_hint` into `encode_hint`.
* Moves prefix_free_hints.go to hints_generator.go; `encode_hint` and `decode_hint` were moved here too.
* The "prefix-free" logic was made 0-index based instead of 1-index based for two reasons:
1. To simplify 1-off discrepancies.
2. To keep backwards compatibility with the old usage of `encode_hint`.
* Slight perfomance improvement added by reusing the runt-to-index map instead of rebuilding for each call to `decode_hint`.
* Unit tests added; for all new logic and old affected logic.
Fixes#10231
- Declares --prefix-free CLI flag in main.py.
- Implements dynamic index offset shifting in marks.go to prevent
generating hints that prefix other hints.
- Integrates prefix-free encoding in main.go.
- Includes tests.
Implements #10195
Note that we loose unicode char matching for --type=word because of
https://github.com/dlclark/regexp2/issues/65 and of course user regexps
cant use \p{N} escapes any more. Hopefully regexp2 will add support for
these soon-ish. IMO lookaround is more important than \p.
Fixes#6265