Python/libs/pygments-lexer-pseudocode2: pygments_lexer

annotate pygments_lexer_pseudocode2/bases.py @ 160:b4028838e0c8

Implement lexer option "prohibit_raiseonerror_filter". Sphinx raises by default when an Error token is seen (by means of the "raiseonerror" filter that is applied by default to lexers in Sphinx). This option skips this and allows error locations to be seen and highlighted properly. While there convert most Generic.Error tokens to Error tokens because now they can be handled by a lexer with "prohibit_raiseonerror_filter=True".

author	Franz Glasner <fzglas.hg@dom66.de>
date	Fri, 08 May 2026 17:46:28 +0200
parents	e1663ac707b0
children

rev	line source
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	1 # -- coding: utf-8 --
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	2 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	3 # SPDX-FileCopyrightText: © 2026 Franz Glasner
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	4 # SPDX-License-Identifier: MIT
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	5 # :-
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	6 r"""Some common bases for the lexers."""
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	7
56 661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	8 __all__ = ["LexBase", "uni_name", "py_innerstring_rules", "py_name_rules"]
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	9
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	10
73 a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	11 import sys
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	12
52 5bfa9113d3c4 First tests with "py-name": names from the Python lexer Franz Glasner <fzglas.hg@dom66.de> parents: 43 diff changeset	13 from pygments import unistring
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	14 from pygments.lexer import RegexLexer, combined, bygroups, include
72 206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	15 from pygments.token import (Comment, Error, Name, Number, Other, String)
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	16
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	17
73 a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	18 PY2 = sys.version_info[0] <= 2
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	19
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	20
56 661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	21 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	22 # SPDX-SnippetBegin
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	23 # SPDX-License-Identifier: BSD-2-Clause
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	24 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	25 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	26 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	27
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	28 uni_name = "[%s][%s]*" % (unistring.xid_start, unistring.xid_continue)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	29
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	30
73 a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	31 """PY3 allows no @staticmethod but PY2 needs it."""
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	32 if PY2:
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	33 _staticmethod = staticmethod
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	34 else:
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	35 def _staticmethod(fn):
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	36 return fn
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	37
a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	38
56 661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	39 def py_innerstring_rules(ttype):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	40 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	41 # the old style '%s' % (...) string formatting (still valid in Py3)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	42 (r'%(\(\w+\))?[-#0 +]([0-9]+\|[])?(\.([0-9]+\|[*]))?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	43 '[hlL]?[E-GXc-giorsaux%]', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	44 # the new style '{}'.format(...) string formatting
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	45 (r'\{'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	46 r'((\w+)((\.\w+)\|(\[[^\]]+\]))*)?' # field name
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	47 r'(\![sra])?' # conversion
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	48 r'(\:(.?[<>=\^])?[-+ ]?#?0?(\d+)?,?(\.\d+)?[E-GXb-gnosx%]?)?'
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	49 r'\}', String.Interpol),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	50 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	51 # backslashes, quotes and formatting signs must be parsed
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	52 # one at a time
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	53 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	54 (r'[^\\\'"%{\n]+', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	55 (r'[\'"\\]', ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	56 # unhandled string formatting sign
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	57 (r'%\|(\{{1,2})', ttype)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	58 # newlines are an error (use "nl" state)
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	59 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	60
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	61
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	62 def py_name_rules(ttype, deco_ttype=Name.Decorator):
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	63 return [
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	64 # We recognize decorator syntax here
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	65 (r'@' + uni_name, deco_ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	66 #
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	67 # Python's new matrix multiplication operator:
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	68 # not used here in pseudocode
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	69 # (r'@', Operator),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	70 (uni_name, ttype),
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	71 ]
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	72
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	73 # SPDX-SnippetEnd
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	74
661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	75
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	76 class LexBase(RegexLexer):
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	77
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	78 """A base that defines some common lexer states.
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	79
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	80 Default flags are not important.
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	81
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	82 """
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	83
62 7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts Franz Glasner <fzglas.hg@dom66.de> parents: 56 diff changeset	84 def op_ignore(lexer, match, ctx=None):
71 2ea86269e84e Doc string Franz Glasner <fzglas.hg@dom66.de> parents: 65 diff changeset	85 """Unconditionally ignore the match."""
62 7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts Franz Glasner <fzglas.hg@dom66.de> parents: 56 diff changeset	86 if False:
7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts Franz Glasner <fzglas.hg@dom66.de> parents: 56 diff changeset	87 yield match.start(), Other, ""
83 cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely. Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	88 if ctx:
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely. Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	89 ctx.pos = match.end()
62 7153e945a3d6 Implement ignoring of \ENDxxx including its optional name parts Franz Glasner <fzglas.hg@dom66.de> parents: 56 diff changeset	90
73 a2a56d08b860 FIX: Python 2 needs a "staticmethod" on "op_fixed()" when called via the LexBase. Franz Glasner <fzglas.hg@dom66.de> parents: 72 diff changeset	91 @_staticmethod
72 206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	92 def op_fixed(toktype, value):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	93 """Unconditionally yield a given token type and value."""
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	94
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	95 def _op_fixed(lexer, match, ctx=None):
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	96 yield match.start(), toktype, value
83 cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely. Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	97 if ctx:
cd79d2c76347 If a Pygments callback gets a "Context" it must set the new position explicitely. Franz Glasner <fzglas.hg@dom66.de> parents: 73 diff changeset	98 ctx.pos = match.end()
72 206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	99
206017a08ed7 Refactor: Make the "op_space()" and related methods more flexible and allow a given fixed token type and value Franz Glasner <fzglas.hg@dom66.de> parents: 71 diff changeset	100 return _op_fixed
65 3f4223a79d2b Normalize whitespace handling for entity names Franz Glasner <fzglas.hg@dom66.de> parents: 62 diff changeset	101
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	102 tokens = {
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	103 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	104 # These states are borrowed from Pygment's Python lexer.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	105 # Their names have been prefixed with `py-'.
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	106 #
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	107 # SPDX-SnippetBegin
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	108 # SPDX-License-Identifier: BSD-2-Clause
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	109 # SPDX-SnippetCopyrightText: Copyright 2006-2023 by the Pygments team
43 bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed Franz Glasner <fzglas.hg@dom66.de> parents: 42 diff changeset	110 # SPDX-SnippetCopyrightText: Copyright 2026 by Franz Glasner
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	111 #
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	112 'py-numbers': [
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	113 (r'(\d(?:_?\d)\.(?:\d(?:_?\d))?\|(?:\d(?:_?\d))?\.\d(?:_?\d))'
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	114 r'([eE][+-]?\d(?:_?\d)*)?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	115 (r'\d(?:_?\d)[eE][+-]?\d(?:_?\d)j?', Number.Float),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	116 (r'0[oO](?:_?[0-7])+', Number.Oct),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	117 (r'0[bB](?:_?[01])+', Number.Bin),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	118 (r'0[xX](?:_?[a-fA-F0-9])+', Number.Hex),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	119 (r'\d(?:_?\d)*', Number.Integer),
db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	120 ],
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	121 'py-strings': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	122 # non-raw strings
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	123 ('([uU]?)(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	124 combined('py-stringescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	125 ("([uU]?)(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	126 combined('py-stringescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	127 ('([uU]?)(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	128 combined('py-stringescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	129 ("([uU]?)(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	130 combined('py-stringescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	131 # non-raw bytes
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	132 ('([bB])(""")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	133 combined('py-bytesescape', 'py-tdqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	134 ("([bB])(''')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	135 combined('py-bytesescape', 'py-tsqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	136 ('([bB])(")', bygroups(String.Affix, String.Double),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	137 combined('py-bytesescape', 'py-dqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	138 ("([bB])(')", bygroups(String.Affix, String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	139 combined('py-bytesescape', 'py-sqs')),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	140 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	141 'py-stringescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	142 (r'\\(N\{.*?\}\|u[a-fA-F0-9]{4}\|U[a-fA-F0-9]{8})', String.Escape),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	143 include('py-bytesescape')
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	144 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	145 'py-bytesescape': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	146 (r'\\([\\abfnrtv"\']\|\n\|x[a-fA-F0-9]{2}\|[0-7]{1,3})',
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	147 String.Escape)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	148 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	149 'py-dqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	150 (r'"', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	151 (r'\\\\\|\\"\|\\\n', String.Escape), # included here for raw strings
43 bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed Franz Glasner <fzglas.hg@dom66.de> parents: 42 diff changeset	152 include('py-strings-double'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed Franz Glasner <fzglas.hg@dom66.de> parents: 42 diff changeset	153 (r'\n', Error), # added by fag
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	154 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	155 'py-sqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	156 (r"'", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	157 (r"\\\\\|\\'\|\\\n", String.Escape), # included here for raw strings
43 bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed Franz Glasner <fzglas.hg@dom66.de> parents: 42 diff changeset	158 include('py-strings-single'),
bbef0ac6cfcf FIX: Proper string parsing: explicitely error out when single-line strings contain a linefeed Franz Glasner <fzglas.hg@dom66.de> parents: 42 diff changeset	159 (r'\n', Error), # added by fag
34 1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	160 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	161 'py-tdqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	162 (r'"""', String.Double, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	163 include('py-strings-double'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	164 (r'\n', String.Double)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	165 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	166 'py-tsqs': [
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	167 (r"'''", String.Single, '#pop'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	168 include('py-strings-single'),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	169 (r'\n', String.Single)
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	170 ],
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	171 'py-strings-single': py_innerstring_rules(String.Single),
1f741934205e Begin a new Pseudocode lexer using numbers and strings from Python Franz Glasner <fzglas.hg@dom66.de> parents: 33 diff changeset	172 'py-strings-double': py_innerstring_rules(String.Double),
56 661461fb4dfc Make the "py-name" rules parameterized: allow to provide the token type. Franz Glasner <fzglas.hg@dom66.de> parents: 52 diff changeset	173 'py-name': py_name_rules(Name.Entity),
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	174 # SPDX-SnippetEnd
42 193ee1eb5013 Document where "multiline-nested-comment" is taken from Franz Glasner <fzglas.hg@dom66.de> parents: 39 diff changeset	175 # This snippet is from the Pygments' documentation "Write your own lexer"
39 a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	176 'multiline-nested-comment': [
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	177 (r'[^*/]+', Comment.Multiline),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	178 (r'/\*', Comment.Multiline, '#push'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	179 (r'\*/', Comment.Multiline, '#pop'),
a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	180 (r'[*/]', Comment.Multiline),
115 e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	181 ],
e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	182 'multiline-nested-comment-alt': [
e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	183 (r'[^*()]+', Comment.Multiline),
e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	184 (r'\(\*', Comment.Multiline, '#push'),
e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	185 (r'\*\)', Comment.Multiline, '#pop'),
e1663ac707b0 An alternate multiline comment syntax using "(* ... )". Franz Glasner <fzglas.hg@dom66.de>* parents: 83 diff changeset	186 (r'[*()]', Comment.Multiline),
39 a3151d837258 Some basic keywords for programs, algorithms, procedures and functions; also comments (single and multiline) and "remarks" Franz Glasner <fzglas.hg@dom66.de> parents: 34 diff changeset	187 ]
33 db1bc740a201 FIX: ADD: Forgotten bases.py. Franz Glasner <fzglas.hg@dom66.de> parents: diff changeset	188 }

Mercurial > hgrepos > Python > libs > pygments-lexer-pseudocode2

annotate pygments_lexer_pseudocode2/bases.py @ 160:b4028838e0c8