Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate pygments_lexer_pseudocode2/bases.py @ 66:9ba1584d4b4a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Also add a test that completely highlights a document with Pygments
author Franz Glasner <fzglas.hg@dom66.de>
date Tue, 28 Apr 2026 15:02:15 +0200
parents 3f4223a79d2b
children 2ea86269e84e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
1 # -*- coding: utf-8 -*-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
2 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
4 # SPDX-License-Identifier: MIT
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
5 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
6 r"""Some common bases for the lexers."""
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
7
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
8 __all__ = ["LexBase", "uni_name", "py_innerstring_rules", "py_name_rules"]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
9
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
10
52
5bfa9113d3c4 First tests with "py-name": names from the Python lexer
Franz Glasner <fzglas.hg@dom66.de>
parents: 43
diff changeset
11 from pygments import unistring
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
12 from pygments.lexer import RegexLexer, combined, bygroups, include
65
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
13 from pygments.token import (Comment, Error, Name, Number, Other, String,
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
14 Whitespace)
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
15
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
16
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
17 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
18 # SPDX-SnippetBegin
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
19 # SPDX-License-Identifier: BSD-2-Clause
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
20 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
21 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
22 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
23
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
24 uni_name = "[%s][%s]*" % (unistring.xid_start, unistring.xid_continue)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
25
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
26
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
27 def py_innerstring_rules(ttype):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
28 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
29 # the old style '%s' % (...) string formatting (still valid in Py3)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
30 (r'%(\(\w+\))?[-#0 +]*([0-9]+|[*])?(\.([0-9]+|[*]))?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
31 '[hlL]?[E-GXc-giorsaux%]', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
32 # the new style '{}'.format(...) string formatting
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
33 (r'\{'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
34 r'((\w+)((\.\w+)|(\[[^\]]+\]))*)?' # field name
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
35 r'(\![sra])?' # conversion
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
36 r'(\:(.?[<>=\^])?[-+ ]?#?0?(\d+)?,?(\.\d+)?[E-GXb-gnosx%]?)?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
37 r'\}', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
38 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
39 # backslashes, quotes and formatting signs must be parsed
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
40 # one at a time
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
41 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
42 (r'[^\\\'"%{\n]+', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
43 (r'[\'"\\]', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
44 # unhandled string formatting sign
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
45 (r'%|(\{{1,2})', ttype)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
46 # newlines are an error (use "nl" state)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
47 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
48
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
49
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
50 def py_name_rules(ttype, deco_ttype=Name.Decorator):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
51 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
52 # We recognize decorator syntax here
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
53 (r'@' + uni_name, deco_ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
54 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
55 # Python's new matrix multiplication operator:
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
56 # not used here in pseudocode
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
57 # (r'@', Operator),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
58 (uni_name, ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
59 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
60
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
61 # SPDX-SnippetEnd
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
62
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
63
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
64 class LexBase(RegexLexer):
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
65
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
66 """A base that defines some common lexer states.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
67
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
68 Default flags are not important.
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
69
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
70 """
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
71
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
72 def op_ignore(lexer, match, ctx=None):
65
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
73 """Ignore the match."""
62
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
74 if False:
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
75 yield match.start(), Other, ""
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts
Franz Glasner <fzglas.hg@dom66.de>
parents: 56
diff changeset
76
65
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
77 def op_space(lexer, match, ctx=None):
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
78 """Unconditionally yield a single whitespace ' '."""
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
79 yield match.start(), Whitespace, " "
3f4223a79d2b Normalize whitespace handling for entity names
Franz Glasner <fzglas.hg@dom66.de>
parents: 62
diff changeset
80
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
81 tokens = {
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
82 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
83 # These states are borrowed from Pygment's Python lexer.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
84 # Their names have been prefixed with `py-'.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
85 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
86 # SPDX-SnippetBegin
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
87 # SPDX-License-Identifier: BSD-2-Clause
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
88 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
89 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
90 #
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
91 'py-numbers': [
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
92 (r'(\d(?:_?\d)*\.(?:\d(?:_?\d)*)?|(?:\d(?:_?\d)*)?\.\d(?:_?\d)*)'
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
93 r'([eE][+-]?\d(?:_?\d)*)?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
94 (r'\d(?:_?\d)*[eE][+-]?\d(?:_?\d)*j?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
95 (r'0[oO](?:_?[0-7])+', Number.Oct),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
96 (r'0[bB](?:_?[01])+', Number.Bin),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
97 (r'0[xX](?:_?[a-fA-F0-9])+', Number.Hex),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
98 (r'\d(?:_?\d)*', Number.Integer),
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
99 ],
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
100 'py-strings': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
101 # non-raw strings
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
102 ('([uU]?)(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
103 combined('py-stringescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
104 ("([uU]?)(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
105 combined('py-stringescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
106 ('([uU]?)(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
107 combined('py-stringescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
108 ("([uU]?)(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
109 combined('py-stringescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
110 # non-raw bytes
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
111 ('([bB])(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
112 combined('py-bytesescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
113 ("([bB])(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
114 combined('py-bytesescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
115 ('([bB])(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
116 combined('py-bytesescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
117 ("([bB])(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
118 combined('py-bytesescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
119 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
120 'py-stringescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
121 (r'\\(N\{.*?\}|u[a-fA-F0-9]{4}|U[a-fA-F0-9]{8})', String.Escape),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
122 include('py-bytesescape')
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
123 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
124 'py-bytesescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
125 (r'\\([\\abfnrtv"\']|\n|x[a-fA-F0-9]{2}|[0-7]{1,3})',
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
126 String.Escape)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
127 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
128 'py-dqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
129 (r'"', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
130 (r'\\\\|\\"|\\\n', String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
131 include('py-strings-double'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
132 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
133 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
134 'py-sqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
135 (r"'", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
136 (r"\\\\|\\'|\\\n", String.Escape), # included here for raw strings
43
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
137 include('py-strings-single'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed
Franz Glasner <fzglas.hg@dom66.de>
parents: 42
diff changeset
138 (r'\n', Error), # added by fag
34
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
139 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
140 'py-tdqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
141 (r'"""', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
142 include('py-strings-double'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
143 (r'\n', String.Double)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
144 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
145 'py-tsqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
146 (r"'''", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
147 include('py-strings-single'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
148 (r'\n', String.Single)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
149 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
150 'py-strings-single': py_innerstring_rules(String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python
Franz Glasner <fzglas.hg@dom66.de>
parents: 33
diff changeset
151 'py-strings-double': py_innerstring_rules(String.Double),
56
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type.
Franz Glasner <fzglas.hg@dom66.de>
parents: 52
diff changeset
152 'py-name': py_name_rules(Name.Entity),
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
153 # SPDX-SnippetEnd
42
193ee1eb5013 Document where "multiline-nested-comment" is taken from
Franz Glasner <fzglas.hg@dom66.de>
parents: 39
diff changeset
154 # This snippet is from the Pygments' documentation "Write your own lexer"
39
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
155 'multiline-nested-comment': [
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
156 (r'[^*/]+', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
157 (r'/\*', Comment.Multiline, '#push'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
158 (r'\*/', Comment.Multiline, '#pop'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
159 (r'[*/]', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks"
Franz Glasner <fzglas.hg@dom66.de>
parents: 34
diff changeset
160 ]
33
db1bc740a201 FIX: ADD: Forgotten bases.py.
Franz Glasner <fzglas.hg@dom66.de>
parents:
diff changeset
161 }