mirror of
https://github.com/PCRE2Project/pcre2.git
synced 2025-10-20 04:25:35 +08:00
Updates to the README and some documentation (#681)
This commit is contained in:
@@ -23,7 +23,7 @@ please consult the man page, in case the conversion went wrong.
|
||||
<li><a name="TOC8" href="#SEC8">REVISION</a>
|
||||
</ul>
|
||||
<br><a name="SEC1" href="#TOC1">PCRE2 MATCHING ALGORITHMS</a><br>
|
||||
<P>
|
||||
<p>
|
||||
This document describes the two different algorithms that are available in
|
||||
PCRE2 for matching a compiled regular expression against a given subject
|
||||
string. The "standard" algorithm is the one provided by the <b>pcre2_match()</b>
|
||||
@@ -32,14 +32,14 @@ Perl-compatible matching operation. The just-in-time (JIT) optimization that is
|
||||
described in the
|
||||
<a href="pcre2jit.html"><b>pcre2jit</b></a>
|
||||
documentation is compatible with this function.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
An alternative algorithm is provided by the <b>pcre2_dfa_match()</b> function;
|
||||
it operates in a different way, and is not Perl-compatible. This alternative
|
||||
has advantages and disadvantages compared with the standard algorithm, and
|
||||
these are described below.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
When there is only one possible way in which a given subject string can match a
|
||||
pattern, the two algorithms give the same answer. A difference arises, however,
|
||||
when there are multiple possibilities. For example, if the anchored pattern
|
||||
@@ -52,18 +52,18 @@ is matched against the string
|
||||
</pre>
|
||||
there are three possible answers. The standard algorithm finds only one of
|
||||
them, whereas the alternative algorithm finds all three.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC2" href="#TOC1">REGULAR EXPRESSIONS AS TREES</a><br>
|
||||
<P>
|
||||
<p>
|
||||
The set of strings that are matched by a regular expression can be represented
|
||||
as a tree structure. An unlimited repetition in the pattern makes the tree of
|
||||
infinite size, but it is still a tree. Matching the pattern to a given subject
|
||||
string (from a given starting point) can be thought of as a search of the tree.
|
||||
There are two ways to search a tree: depth-first and breadth-first, and these
|
||||
correspond to the two matching algorithms provided by PCRE2.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC3" href="#TOC1">THE STANDARD MATCHING ALGORITHM</a><br>
|
||||
<P>
|
||||
<p>
|
||||
In the terminology of Jeffrey Friedl's book "Mastering Regular Expressions",
|
||||
the standard algorithm is an "NFA algorithm". It conducts a depth-first search
|
||||
of the pattern tree. That is, it proceeds along a single path through the tree,
|
||||
@@ -73,23 +73,23 @@ fail, it backs up to the previous branch point in the tree, and tries the next
|
||||
alternative branch at that level. This often involves backing up (moving to the
|
||||
left) in the subject string as well. The order in which repetition branches are
|
||||
tried is controlled by the greedy or ungreedy nature of the quantifier.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
If a leaf node is reached, a matching string has been found, and at that point
|
||||
the algorithm stops. Thus, if there is more than one possible match, this
|
||||
algorithm returns the first one that it finds. Whether this is the shortest,
|
||||
the longest, or some intermediate length depends on the way the alternations
|
||||
and the greedy or ungreedy repetition quantifiers are specified in the
|
||||
pattern.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
Because it ends up with a single path through the tree, it is relatively
|
||||
straightforward for this algorithm to keep track of the substrings that are
|
||||
matched by portions of the pattern in parentheses. This provides support for
|
||||
capturing parentheses and backreferences.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC4" href="#TOC1">THE ALTERNATIVE MATCHING ALGORITHM</a><br>
|
||||
<P>
|
||||
<p>
|
||||
This algorithm conducts a breadth-first search of the tree. Starting from the
|
||||
first matching point in the subject, it scans the subject string from left to
|
||||
right, once, character by character, and as it does this, it remembers all the
|
||||
@@ -97,14 +97,14 @@ paths through the tree that represent valid matches. In Friedl's terminology,
|
||||
this is a kind of "DFA algorithm", though it is not implemented as a
|
||||
traditional finite state machine (it keeps multiple states active
|
||||
simultaneously).
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
Although the general principle of this matching algorithm is that it scans the
|
||||
subject string only once, without backtracking, there is one exception: when a
|
||||
lookaround assertion is encountered, the characters following or preceding the
|
||||
current point have to be independently inspected.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
The scan continues until either the end of the subject is reached, or there are
|
||||
no more unterminated paths. At this point, terminated paths represent the
|
||||
different matching possibilities (if there are none, the match has failed).
|
||||
@@ -112,14 +112,14 @@ Thus, if there is more than one possible match, this algorithm finds all of
|
||||
them, and in particular, it finds the longest. The matches are returned in
|
||||
the output vector in decreasing order of length. There is an option to stop the
|
||||
algorithm after the first match (which is necessarily the shortest) is found.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
Note that the size of vector needed to contain all the results depends on the
|
||||
number of simultaneous matches, not on the number of capturing parentheses in
|
||||
the pattern. Using <b>pcre2_match_data_create_from_pattern()</b> to create the
|
||||
match data block is therefore not advisable when doing DFA matching.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
Note also that all the matches that are found start at the same point in the
|
||||
subject. If the pattern
|
||||
<pre>
|
||||
@@ -129,8 +129,8 @@ is matched against the string "the caterpillar catchment", the result is the
|
||||
three strings "caterpillar", "cater", and "cat" that start at the fifth
|
||||
character of the subject. The algorithm does not automatically move on to find
|
||||
matches that start at later positions.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
PCRE2's "auto-possessification" optimization usually applies to character
|
||||
repeats at the end of a pattern (as well as internally). For example, the
|
||||
pattern "a\d+" is compiled as if it were "a\d++" because there is no point
|
||||
@@ -138,13 +138,13 @@ even considering the possibility of backtracking into the repeated digits. For
|
||||
DFA matching, this means that only one possible match is found. If you really
|
||||
do want multiple matches in such cases, either use an ungreedy repeat
|
||||
("a\d+?") or set the PCRE2_NO_AUTO_POSSESS option when compiling.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
There are a number of features of PCRE2 regular expressions that are not
|
||||
supported or behave differently in the alternative matching function. Those
|
||||
that are not supported cause an error if encountered.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
1. Because the algorithm finds all possible matches, the greedy or ungreedy
|
||||
nature of repetition quantifiers is not relevant (though it may affect
|
||||
auto-possessification, as just described). During matching, greedy and ungreedy
|
||||
@@ -158,14 +158,14 @@ This pattern matches "aaab!" but not "aaa!", which would be matched by a
|
||||
non-possessive quantifier. Similarly, if an atomic group is present, it is
|
||||
matched as if it were a standalone pattern at the current point, and the
|
||||
longest match is then "locked in" for the rest of the overall pattern.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
2. When dealing with multiple paths through the tree simultaneously, it is not
|
||||
straightforward to keep track of captured substrings for the different matching
|
||||
possibilities, and PCRE2's implementation of this algorithm does not attempt to
|
||||
do this. This means that no captured substrings are available.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
3. Because no substrings are captured, a number of related features are not
|
||||
available:
|
||||
<br>
|
||||
@@ -181,78 +181,78 @@ for a specific group recursion;
|
||||
<br>
|
||||
<br>
|
||||
(d) Scan substring assertions.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
4. Because many paths through the tree may be active, the \K escape sequence,
|
||||
which resets the start of the match when encountered (but may be on some paths
|
||||
and not on others), is not supported.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
5. Callouts are supported, but the value of the <i>capture_top</i> field is
|
||||
always 1, and the value of the <i>capture_last</i> field is always 0.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
6. The \C escape sequence, which (in the standard algorithm) always matches a
|
||||
single code unit, even in a UTF mode, is not supported in UTF modes because
|
||||
the alternative algorithm moves through the subject string one character (not
|
||||
code unit) at a time, for all active paths through the tree.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
7. Except for (*FAIL), the backtracking control verbs such as (*PRUNE) are not
|
||||
supported. (*FAIL) is supported, and behaves like a failing negative assertion.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
8. The PCRE2_MATCH_INVALID_UTF option for <b>pcre2_compile()</b> is not
|
||||
supported by <b>pcre2_dfa_match()</b>.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC5" href="#TOC1">ADVANTAGES OF THE ALTERNATIVE ALGORITHM</a><br>
|
||||
<P>
|
||||
<p>
|
||||
The main advantage of the alternative algorithm is that all possible matches
|
||||
(at a single point in the subject) are automatically found, and in particular,
|
||||
the longest match is found. To find more than one match at the same point using
|
||||
the standard algorithm, you have to do kludgy things with callouts.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
Partial matching is possible with this algorithm, though it has some
|
||||
limitations. The
|
||||
<a href="pcre2partial.html"><b>pcre2partial</b></a>
|
||||
documentation gives details of partial matching and discusses multi-segment
|
||||
matching.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC6" href="#TOC1">DISADVANTAGES OF THE ALTERNATIVE ALGORITHM</a><br>
|
||||
<P>
|
||||
<p>
|
||||
The alternative algorithm suffers from a number of disadvantages:
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
1. It is substantially slower than the standard algorithm. This is partly
|
||||
because it has to search for all possible matches, but is also because it is
|
||||
less susceptible to optimization.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
2. Capturing parentheses and other features such as backreferences that rely on
|
||||
them are not supported.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
3. Matching within invalid UTF strings is not supported.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
4. Although atomic groups are supported, their use does not provide the
|
||||
performance advantage that it does for the standard algorithm.
|
||||
</P>
|
||||
<P>
|
||||
</p>
|
||||
<p>
|
||||
5. JIT optimization is not supported.
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC7" href="#TOC1">AUTHOR</a><br>
|
||||
<P>
|
||||
<p>
|
||||
Philip Hazel
|
||||
<br>
|
||||
Retired from University Computing Service
|
||||
<br>
|
||||
Cambridge, England.
|
||||
<br>
|
||||
</P>
|
||||
</p>
|
||||
<br><a name="SEC8" href="#TOC1">REVISION</a><br>
|
||||
<P>
|
||||
<p>
|
||||
Last updated: 30 August 2024
|
||||
<br>
|
||||
Copyright © 1997-2024 University of Cambridge.
|
||||
|
Reference in New Issue
Block a user