Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate pygments_lexer_pseudocode2/bases.py @ 72:206017a08ed7

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
author Franz Glasner <fzglas.hg@dom66.de>
date Tue, 28 Apr 2026 19:14:17 +0200
parents 2ea86269e84e
children a2a56d08b860
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # SPDX-License-Identifier: MIT
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Some common bases for the lexers."""
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
8 __all__ = ["LexBase", "uni_name", "py_innerstring_rules", "py_name_rules"]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10
52
5bfa9113d3c4 First tests with "py-name": names from the Python lexer
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
11 from pygments import unistring
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
12 from pygments.lexer import RegexLexer, combined, bygroups, include
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
13 from pygments.token import (Comment, Error, Name, Number, Other, String)
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
14
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
16 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
17 # SPDX-SnippetBegin
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
18 # SPDX-License-Identifier: BSD-2-Clause
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
19 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
20 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
21 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
22
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
23 uni_name = "[%s][%s]*" % (unistring.xid_start, unistring.xid_continue)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
24
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
25
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
26 def py_innerstring_rules(ttype):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
27 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
28 # the old style '%s' % (...) string formatting (still valid in Py3)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
29 (r'%(\(\w+\))?[-#0 +]*([0-9]+|[*])?(\.([0-9]+|[*]))?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
30 '[hlL]?[E-GXc-giorsaux%]', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
31 # the new style '{}'.format(...) string formatting
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
32 (r'\{'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
33 r'((\w+)((\.\w+)|(\[[^\]]+\]))*)?' # field name
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
34 r'(\![sra])?' # conversion
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
35 r'(\:(.?[<>=\^])?[-+ ]?#?0?(\d+)?,?(\.\d+)?[E-GXb-gnosx%]?)?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
36 r'\}', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
37 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
38 # backslashes, quotes and formatting signs must be parsed
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
39 # one at a time
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
40 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
41 (r'[^\\\'"%{\n]+', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
42 (r'[\'"\\]', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
43 # unhandled string formatting sign
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
44 (r'%|(\{{1,2})', ttype)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
45 # newlines are an error (use "nl" state)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
46 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
47
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
48
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
49 def py_name_rules(ttype, deco_ttype=Name.Decorator):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
50 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
51 # We recognize decorator syntax here
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
52 (r'@' + uni_name, deco_ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
53 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
54 # Python's new matrix multiplication operator:
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
55 # not used here in pseudocode
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
56 # (r'@', Operator),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
57 (uni_name, ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
58 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
59
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
60 # SPDX-SnippetEnd
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
61
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
62
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
63 class LexBase(RegexLexer):
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65 """A base that defines some common lexer states.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67 Default flags are not important.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69 """
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
71 def op_ignore(lexer, match, ctx=None):
71
2ea86269e84e Doc string
Franz Glasner <fzglas.hg@dom66.de>
parents: 65
diff changeset
72 """Unconditionally ignore the match."""
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
73 if False:
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
74 yield match.start(), Other, ""
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
75
72
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
76 def op_fixed(toktype, value):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
77 """Unconditionally yield a given token type and value."""
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
78
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
79 def _op_fixed(lexer, match, ctx=None):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
80 yield match.start(), toktype, value
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
81
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value
Franz Glasner <fzglas.hg@dom66.de>
parents: 71
diff changeset
82 return _op_fixed
65
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
83
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
84 tokens = {
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
85 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
86 # These states are borrowed from Pygment's Python lexer.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
87 # Their names have been prefixed with `py-'.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
88 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
89 # SPDX-SnippetBegin
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
90 # SPDX-License-Identifier: BSD-2-Clause
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
91 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
92 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
93 #
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 'py-numbers': [
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 (r'(\d(?:_?\d)*\.(?:\d(?:_?\d)*)?|(?:\d(?:_?\d)*)?\.\d(?:_?\d)*)'
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 r'([eE][+-]?\d(?:_?\d)*)?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 (r'\d(?:_?\d)*[eE][+-]?\d(?:_?\d)*j?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 (r'0[oO](?:_?[0-7])+', Number.Oct),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 (r'0[bB](?:_?[01])+', Number.Bin),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
100 (r'0[xX](?:_?[a-fA-F0-9])+', Number.Hex),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
101 (r'\d(?:_?\d)*', Number.Integer),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
102 ],
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
103 'py-strings': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
104 # non-raw strings
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
105 ('([uU]?)(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
106 combined('py-stringescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
107 ("([uU]?)(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
108 combined('py-stringescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
109 ('([uU]?)(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
110 combined('py-stringescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
111 ("([uU]?)(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
112 combined('py-stringescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
113 # non-raw bytes
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
114 ('([bB])(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
115 combined('py-bytesescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
116 ("([bB])(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
117 combined('py-bytesescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
118 ('([bB])(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
119 combined('py-bytesescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
120 ("([bB])(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
121 combined('py-bytesescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
122 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
123 'py-stringescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
124 (r'\\(N\{.*?\}|u[a-fA-F0-9]{4}|U[a-fA-F0-9]{8})', String.Escape),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
125 include('py-bytesescape')
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
126 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
127 'py-bytesescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
128 (r'\\([\\abfnrtv"\']|\n|x[a-fA-F0-9]{2}|[0-7]{1,3})',
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
129 String.Escape)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
130 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
131 'py-dqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
132 (r'"', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
133 (r'\\\\|\\"|\\\n', String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
134 include('py-strings-double'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
135 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
136 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
137 'py-sqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
138 (r"'", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
139 (r"\\\\|\\'|\\\n", String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
140 include('py-strings-single'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
141 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
142 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
143 'py-tdqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
144 (r'"""', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
145 include('py-strings-double'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
146 (r'\n', String.Double)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
147 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
148 'py-tsqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
149 (r"'''", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
150 include('py-strings-single'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
151 (r'\n', String.Single)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
152 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
153 'py-strings-single': py_innerstring_rules(String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
154 'py-strings-double': py_innerstring_rules(String.Double),
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
155 'py-name': py_name_rules(Name.Entity),
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
156 # SPDX-SnippetEnd
42
193ee1eb5013 Document where "multiline-nested-comment" is taken from
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
157 # This snippet is from the Pygments' documentation "Write your own lexer"
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
158 'multiline-nested-comment': [
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
159 (r'[^*/]+', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
160 (r'/\*', Comment.Multiline, '#push'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
161 (r'\*/', Comment.Multiline, '#pop'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
162 (r'[*/]', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
163 ]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
164 }