Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate pygments_lexer_pseudocode2/utils.py @ 291:b09873e4df58

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Unit test: enhance some test for \ttX
author Franz Glasner <fzglas.hg@dom66.de>
date Fri, 22 May 2026 12:57:49 +0200
parents 298841bc4dee
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
85
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8; -*-
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # SPDX-License-Identifier: MIT
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Some helper utility functions and data.
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
8 """
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
288
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
10 __all__ = [
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
11 "REVERSED_STANDARD_TYPES",
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
12 "string_to_defined_tokentype",
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
13 ]
85
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16 import pygments.token
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
18
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
19 REVERSED_STANDARD_TYPES = {}
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
20
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
21 # Side-effect on import
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
22 for _toktype, _cssstyle in pygments.token.STANDARD_TYPES.items():
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
23 REVERSED_STANDARD_TYPES[_cssstyle] = _toktype
ae5e741d2a9b Optimize op_explicit_tokentype(): use a pref-computed reversed pygments.token.STANDARD_TYPES
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
24 del _toktype, _cssstyle
288
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
25
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
26
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
27 def string_to_defined_tokentype(s):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
28 """Determine whether the token type `s` given as string is defined.
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
29
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
30 :param str s: A token type string as in
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
31 :py:func:`pygments.token.string_to_tokentype`.
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
32 :returns: An **existing** token if
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
33 :py:func:`pygments.token.string_to_tokentype`
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
34 would return an already existing token type,
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
35 :py:obj:`None` otherwise.
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
36 :rtype: :py:class:`pygments.token._TokenType` or :py:obj:`None`
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
37
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
38 This implementation is needed because
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
39 :py:func:`pygments.token.string_to_tokentype` synthesizes a new token
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
40 on not yet existing token types.
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
41 And :py:func:`is_token_subtype` works only on token instances.
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
42
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
43 """
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
44 ttype = pygments.token.Token
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
45 ttype_prefix = "Token."
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
46 if not s:
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
47 return ttype
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
48 for part in s.split("."):
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
49 for subtype in ttype.subtypes:
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
50 subtypename = str(subtype)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
51 # Remove prefix
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
52 assert subtypename.startswith(ttype_prefix)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
53 subtypename = subtypename[len(ttype_prefix):]
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
54 if subtypename == part:
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
55 ttype = subtype
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
56 ttype_prefix += "%s." % (part,)
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
57 break
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
58 else:
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
59 return None
298841bc4dee Allow "normal" Pygments token names in "\ttX" ("Error", "Text.Whitespace", ...)
Franz Glasner <fzglas.hg@dom66.de>
parents: 85
diff changeset
60 return ttype