Regex-Toolkit provides tools for creating RE and RE2 expressions.
Regex-Toolkit requires Python 3.10 or higher, is platform independent, and has no outside dependencies.
If you discover an issue with Regex-Toolkit, please report it at https://github.com/Phosmic/regex-toolkit/issues.
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.
Requirements Installing Usage Library
Most stable version from PyPi:
python3 -m pip install regex-toolkit
Development version from GitHub:
git clone git+https://github.com/Phosmic/regex-toolkit.git
cd regex-toolkit
python3 -m pip install -e .
To harness the toolkit's capabilities, you should import the necessary packages:
import re
# and/or
import re2
import regex_toolkit as rtk
Regex definitions vary across languages and versions. By using the toolkit, you can achieve a more consistent and comprehensive representation of unicode support. It is especially useful to supplement base unicode sets with the latest definitions from other languages and standards.
RE2 focuses on safely processing regular expressions, particularly from untrusted inputs. It ensures both linear match time and efficient memory usage. Although it might not always surpass other engines in speed, it intentionally omits features that depend solely on backtracking, like backreferences and look-around assertions.
A brief rundown of RE2 terminology:
- BitState: An execution engine that uses backtracking search.
- bytecode: The set of instructions that form an automaton.
- DFA: The engine for Deterministic Finite Automaton searches.
- NFA: Implements the Nondeterministic Finite Automaton search method.
- OnePass: A one-pass search execution engine.
- pattern: The textual form of a regex.
- Prog: The compiled version of a regex.
- Regexp: The parsed version of a regex.
- Rune: A character in terms of encoding, essentially a code point.
For an in-depth exploration, please refer to the RE2 documentation.
def resolve_flavor(potential_flavor: int | RegexFlavor | None) -> RegexFlavor
Resolve a regex flavor.
If the flavor is an integer, it is validated and returned.
If the flavor is a RegexFlavor, it is returned.
If the flavor is None, the default flavor is returned. To change the default flavor, set default_flavor
.
import regex_toolkit as rtk
rtk.base.default_flavor = 2
assert rtk.utils.resolve_flavor(None) == rtk.enums.RegexFlavor.RE2
Arguments:
potential_flavor
int | RegexFlavor | None - Potential regex flavor.
Returns:
- RegexFlavor - Resolved regex flavor.
Raises:
ValueError
- Invalid regex flavor.
def iter_sort_by_len(texts: Iterable[str],
*,
reverse: bool = False) -> Generator[str, None, None]
Iterate strings sorted by length.
Arguments:
texts
Iterable[str] - Strings to sort.reverse
bool, optional - Sort in descending order (longest to shortest). Defaults to False.
Yields:
- str - Strings sorted by length.
def sort_by_len(texts: Iterable[str],
*,
reverse: bool = False) -> tuple[str, ...]
Sort strings by length.
Arguments:
texts
Iterable[str] - Strings to sort.reverse
bool, optional - Sort in descending order (longest to shortest). Defaults to False.
Returns:
- tuple[str, ...] - Strings sorted by length.
def ord_to_cpoint(ordinal: int) -> str
Character ordinal to character codepoint.
The codepoint is always 8 characters long (zero-padded).
Example:
ord_to_cpoint(97)
# Output: '00000061'
Arguments:
ordinal
int - Character ordinal.
Returns:
- str - Character codepoint.
def cpoint_to_ord(cpoint: str) -> int
Character codepoint to character ordinal.
Arguments:
cpoint
str - Character codepoint.
Returns:
- int - Character ordinal.
def char_to_cpoint(char: str) -> str
Character to character codepoint.
Example:
char_to_cpoint("a")
# Output: '00000061'
Arguments:
char
str - Character.
Returns:
- str - Character codepoint.
def to_nfc(text: str) -> str
Normalize a Unicode string to NFC form C.
Form C favors the use of a fully combined character.
Example:
to_nfc("e\\u0301") == "é"
# Output: True
Arguments:
text
str - String to normalize.
Returns:
- str - Normalized string.
def iter_char_range(first_char: str,
last_char: str) -> Generator[str, None, None]
Iterate all characters within a range of characters (inclusive).
Example:
char_range("a", "c")
# Output: ('a', 'b', 'c')
char_range("c", "a")
# Output: ('c', 'b', 'a')
Arguments:
first_char
str - Starting (first) character.last_char
str - Ending (last) character.
Yields:
- str - Characters within a range of characters.
def char_range(first_char: str, last_char: str) -> tuple[str, ...]
Tuple of all characters within a range of characters (inclusive).
Example:
char_range("a", "d")
# Output: ('a', 'b', 'c', 'd')
char_range("d", "a")
# Output: ('d', 'c', 'b', 'a')
Arguments:
first_char
str - Starting (first) character.last_char
str - Ending (last) character.
Returns:
- tuple[str, ...] - Characters within a range of characters.
def mask_span(text: str,
span: list[int] | tuple[int, int],
mask: str | None = None) -> str
Slice and mask a string using a single span.
Arguments:
text
str - String to slice.span
list[int] | tuple[int, int] - Domain of index positions (start, end) to mask.mask
str, optional - Mask to insert after slicing. Defaults to None.
Returns:
- str - String with span replaced with the mask text.
def mask_spans(text: str,
spans: Iterable[list[int] | tuple[int, int]],
masks: Iterable[str] | None = None) -> str
Slice and mask a string using multiple spans.
Todo: Add support for overlapping (and unordered?) spans.
Arguments:
text
str - String to slice.spans
Iterable[list[int] | tuple[int, int]] - Domains of index positions (x1, x2) to mask within the text.masks
Iterable[str], optional - Masks to insert when slicing. Defaults to None.
Returns:
- str - String with all spans replaced with the mask text.
def escape(char: str, flavor: int | None = None) -> str
Create a regex expression that exactly matches a character.
Arguments:
char
str - Character to match.flavor
int | None, optional - Regex flavor (1 for RE, 2 for RE2). Defaults to None.
Returns:
- str - Expression that exactly matches the original character.
Raises:
ValueError
- Invalid regex flavor.
def string_as_exp(text: str, flavor: int | None = None) -> str
Create a regex expression that exactly matches a string.
Arguments:
text
str - String to match.flavor
int | None, optional - Regex flavor (1 for RE, 2 for RE2). Defaults to None.
Returns:
- str - Expression that exactly matches the original string.
Raises:
ValueError
- Invalid regex flavor.
def strings_as_exp(texts: Iterable[str], flavor: int | None = None) -> str
Create a regex expression that exactly matches any one string.
Arguments:
texts
Iterable[str] - Strings to match.flavor
int | None, optional - Regex flavor (1 for RE, 2 for RE2). Defaults to None.
Returns:
- str - Expression that exactly matches any one of the original strings.
Raises:
ValueError
- Invalid regex flavor.
def make_exp(chars: Iterable[str], flavor: int | None = None) -> str
Create a regex expression that exactly matches a list of characters.
The characters are sorted and grouped into ranges where possible. The expression is not anchored, so it can be used as part of a larger expression.
Example:
exp = "[" + make_exp(["a", "b", "c", "z", "y", "x"]) + "]"
# Output: '[a-cx-z]'
Arguments:
chars
Iterable[str] - Characters to match.flavor
int | None, optional - Regex flavor (1 for RE, 2 for RE2). Defaults to None.
Returns:
- str - Expression that exactly matches the original characters.
Raises:
ValueError
- Invalid regex flavor.
Enums.
class RegexFlavor(int, Enum)
Regex flavors.
Attributes:
RE
int - Standard Python regex flavor.RE2
int - Google RE2 regex flavor.