Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate pygments_lexer_pseudocode2/bases.py @ 83:cd79d2c76347

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
If a Pygments callback gets a "Context" it must set the new position explicitely. This is **not** done automatically.
author Franz Glasner <fzglas.hg@dom66.de>
date Thu, 30 Apr 2026 13:21:38 +0200
parents a2a56d08b860
children e1663ac707b0
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # SPDX-License-Identifier: MIT
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Some common bases for the lexers."""
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
8 __all__ = ["LexBase", "uni_name", "py_innerstring_rules", "py_name_rules"]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10
73
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
11 import sys
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
12
52
5bfa9113d3c4 First tests with "py-name": names from the Python lexer
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
13 from pygments import unistring
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
14 from pygments.lexer import RegexLexer, combined, bygroups, include
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
15 from pygments.token import (Comment, Error, Name, Number, Other, String)
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
17
73
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
18 PY2 = sys.version_info[0] <= 2
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
19
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
20
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
21 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
22 # SPDX-SnippetBegin
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
23 # SPDX-License-Identifier: BSD-2-Clause
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
24 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
25 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
26 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
27
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
28 uni_name = "[%s][%s]*" % (unistring.xid_start, unistring.xid_continue)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
29
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
30
73
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
31 """PY3 allows no @staticmethod but PY2 needs it."""
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
32 if PY2:
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
33 _staticmethod = staticmethod
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
34 else:
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
35 def _staticmethod(fn):
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
36 return fn
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
37
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
38
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
39 def py_innerstring_rules(ttype):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
40 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
41 # the old style '%s' % (...) string formatting (still valid in Py3)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
42 (r'%(\(\w+\))?[-#0 +]*([0-9]+|[*])?(\.([0-9]+|[*]))?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
43 '[hlL]?[E-GXc-giorsaux%]', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
44 # the new style '{}'.format(...) string formatting
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
45 (r'\{'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
46 r'((\w+)((\.\w+)|(\[[^\]]+\]))*)?' # field name
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
47 r'(\![sra])?' # conversion
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
48 r'(\:(.?[<>=\^])?[-+ ]?#?0?(\d+)?,?(\.\d+)?[E-GXb-gnosx%]?)?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
49 r'\}', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
50 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
51 # backslashes, quotes and formatting signs must be parsed
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
52 # one at a time
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
53 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
54 (r'[^\\\'"%{\n]+', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
55 (r'[\'"\\]', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
56 # unhandled string formatting sign
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
57 (r'%|(\{{1,2})', ttype)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
58 # newlines are an error (use "nl" state)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
59 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
60
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
61
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
62 def py_name_rules(ttype, deco_ttype=Name.Decorator):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
63 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
64 # We recognize decorator syntax here
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
65 (r'@' + uni_name, deco_ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
66 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
67 # Python's new matrix multiplication operator:
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
68 # not used here in pseudocode
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
69 # (r'@', Operator),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
70 (uni_name, ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
71 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
72
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
73 # SPDX-SnippetEnd
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
74
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
75
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
76 class LexBase(RegexLexer):
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
77
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
78 """A base that defines some common lexer states.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
79
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
80 Default flags are not important.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
81
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
82 """
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
83
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
84 def op_ignore(lexer, match, ctx=None):
71
2ea86269e84e Doc string
Franz Glasner <fzglas.hg@dom66.de>
parents: 65
diff changeset
85 """Unconditionally ignore the match."""
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
86 if False:
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
87 yield match.start(), Other, ""
83
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely.
Franz Glasner <fzglas.hg@dom66.de>
parents: 73
diff changeset
88 if ctx:
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely.
Franz Glasner <fzglas.hg@dom66.de>
parents: 73
diff changeset
89 ctx.pos = match.end()
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
90
73
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase.
Franz Glasner <fzglas.hg@dom66.de>
parents: 72
diff changeset
91 @_staticmethod
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
92 def op_fixed(toktype, value):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
93 """Unconditionally yield a given token type and value."""
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
94
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
95 def _op_fixed(lexer, match, ctx=None):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
96 yield match.start(), toktype, value
83
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely.
Franz Glasner <fzglas.hg@dom66.de>
parents: 73
diff changeset
97 if ctx:
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely.
Franz Glasner <fzglas.hg@dom66.de>
parents: 73
diff changeset
98 ctx.pos = match.end()
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
99
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
100 return _op_fixed
65
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
101
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
102 tokens = {
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
103 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
104 # These states are borrowed from Pygment's Python lexer.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
105 # Their names have been prefixed with `py-'.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
106 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
107 # SPDX-SnippetBegin
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
108 # SPDX-License-Identifier: BSD-2-Clause
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
109 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
110 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
111 #
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
112 'py-numbers': [
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
113 (r'(\d(?:_?\d)*\.(?:\d(?:_?\d)*)?|(?:\d(?:_?\d)*)?\.\d(?:_?\d)*)'
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
114 r'([eE][+-]?\d(?:_?\d)*)?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
115 (r'\d(?:_?\d)*[eE][+-]?\d(?:_?\d)*j?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
116 (r'0[oO](?:_?[0-7])+', Number.Oct),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
117 (r'0[bB](?:_?[01])+', Number.Bin),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
118 (r'0[xX](?:_?[a-fA-F0-9])+', Number.Hex),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
119 (r'\d(?:_?\d)*', Number.Integer),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
120 ],
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
121 'py-strings': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
122 # non-raw strings
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
123 ('([uU]?)(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
124 combined('py-stringescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
125 ("([uU]?)(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
126 combined('py-stringescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
127 ('([uU]?)(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
128 combined('py-stringescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
129 ("([uU]?)(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
130 combined('py-stringescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
131 # non-raw bytes
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
132 ('([bB])(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
133 combined('py-bytesescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
134 ("([bB])(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
135 combined('py-bytesescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
136 ('([bB])(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
137 combined('py-bytesescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
138 ("([bB])(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
139 combined('py-bytesescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
140 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
141 'py-stringescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
142 (r'\\(N\{.*?\}|u[a-fA-F0-9]{4}|U[a-fA-F0-9]{8})', String.Escape),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
143 include('py-bytesescape')
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
144 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
145 'py-bytesescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
146 (r'\\([\\abfnrtv"\']|\n|x[a-fA-F0-9]{2}|[0-7]{1,3})',
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
147 String.Escape)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
148 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
149 'py-dqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
150 (r'"', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
151 (r'\\\\|\\"|\\\n', String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
152 include('py-strings-double'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
153 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
154 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
155 'py-sqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
156 (r"'", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
157 (r"\\\\|\\'|\\\n", String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
158 include('py-strings-single'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
159 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
160 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
161 'py-tdqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
162 (r'"""', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
163 include('py-strings-double'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
164 (r'\n', String.Double)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
165 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
166 'py-tsqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
167 (r"'''", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
168 include('py-strings-single'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
169 (r'\n', String.Single)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
170 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
171 'py-strings-single': py_innerstring_rules(String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
172 'py-strings-double': py_innerstring_rules(String.Double),
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
173 'py-name': py_name_rules(Name.Entity),
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
174 # SPDX-SnippetEnd
42
193ee1eb5013 Document where "multiline-nested-comment" is taken from
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
175 # This snippet is from the Pygments' documentation "Write your own lexer"
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
176 'multiline-nested-comment': [
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
177 (r'[^*/]+', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
178 (r'/\*', Comment.Multiline, '#push'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
179 (r'\*/', Comment.Multiline, '#pop'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
180 (r'[*/]', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
181 ]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
182 }